[APM] API Snapshot Testing #77229

dgieselaar · 2020-09-10T18:25:14Z

Enables snapshot testing for APM API tests.

expectSnapshot(received).toMatch(): stores snapshot to file
expectSnapshot(received).toMatchInline(): stores snapshot inline

Titles from snapshots strive to be unique per-file. This allows non-inline snapshot tests to be shared between configurations to compare results.
Any remaining obsolete snapshots will fail the test.

Update all snapshots (and remove obsolete snapshots) by passing UPDATE_SNAPSHOTS=1 as an environment variable.

I've also taken the liberty of moving all literal value tests and JSON responses to (inline) snapshots.

Output from a failed test:

Inline snapshot:

dgieselaar · 2020-09-10T19:59:31Z

x-pack/test/apm_api_integration/basic/tests/index.ts

+    beforeEach(matchSnapshot.init);
+    afterEach(matchSnapshot.teardown);


Alternatively, we can try using require('../../common/match_snapshot'), and then calling beforeEach() and afterEach() in that file. We could also add an after() hook there that checks if there are unused snapshots (we'd need to track them in expectMatchSnapshot).

dgieselaar · 2020-09-11T18:55:41Z

x-pack/test/apm_api_integration/basic/tests/traces/top_traces.ts

-        );
-
-        expect(responseWithoutSamples).to.eql(expectedTracesWithoutSamples);
+        expectSnapshot(responseWithoutSamples).toMatchInline(`


I'll probably remove this later, this is just so we can see what it would mean to use inline snapshots for bigger responses.

Replaced it with a more realistic scenario.

…testing

sorenlouv · 2020-09-14T09:23:39Z

x-pack/test/apm_api_integration/basic/tests/traces/top_traces.ts

@@ -19,13 +19,19 @@ export default function ApiTest({ getService }: FtrProviderContext) {

  describe('Top traces', () => {
    describe('when data is not loaded ', () => {
-      it('handles empty state', async () => {
+      it('handles empty state', async function () {


Is this change necessary?

no longer, left over from initial exploration. I'll remove it.

sorenlouv · 2020-09-14T09:29:36Z

x-pack/test/apm_api_integration/basic/tests/transaction_groups/breakdown.ts

+        expectSnapshot({
+          title,
+          color,
+          type,
+          hideLegend,
+          legendValue,
+        }).toMatchInline(`
+          Object {
+            "color": "#54b399",
+            "hideLegend": false,
+            "legendValue": "100%",
+            "title": "app",
+            "type": "areaStacked",
+          }
+        `);


Any advantages to grouping the values over doing separate expectations?
asking mostly because you've been splitting objects into multiple expectations elsewhere.

Yeah, I prefer the latter approach. This was also just a first iteration that I neglected to update. I'll replace it.

sorenlouv · 2020-09-14T09:32:06Z

x-pack/test/apm_api_integration/basic/tests/transaction_groups/error_rate.ts

+        it('has the correct start date', () => {
+          expectSnapshot(first(errorRateResponse.erroneousTransactionsRate)?.x).toMatchInline(
+            `1598439600000`
+          );
        });


Would it make sense to convert the timestamp to a human readable date?

Suggested change

it('has the correct start date', () => {

expectSnapshot(first(errorRateResponse.erroneousTransactionsRate)?.x).toMatchInline(

`1598439600000`

);

});

it('has the correct start date', () => {

expectSnapshot(new Date(first(errorRateResponse.erroneousTransactionsRate)?.x).toISOString()).toMatchInline(

`2020-08-26T11:00:00.000Z`

);

});

great idea 👍🏻

sorenlouv

This is truly awesome!! Very excited to see how this works in practice.

Terminology: we cannot call the es archives "snapshots". It's just going to be too confusing 😂

es archives 👍
jest snapshots 👍
es snapshots 👎
jest archives 👎👎👎

kibanamachine · 2020-09-14T11:46:40Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c98af53

Build metrics

✅ unchanged

History

💚 Build #74017 succeeded 541ef2a
💚 Build #73975 succeeded b94fbbd
💚 Build #73967 succeeded 4be2387
💚 Build #73963 succeeded 129780e
💚 Build #73958 succeeded 5efece4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (26 commits) updating datatable type (elastic#77320) [ML] Fix custom URLs processing for security app (elastic#76957) [telemetry] add schema guideline + schema_check new check for --path config (elastic#75747) [ML] Transforms: API schemas and integration tests (elastic#75164) [Mappings editor] Add support for wildcard field type (elastic#76574) [Ingest Manager] Fix flyout instruction selection (elastic#77071) [Telemetry Tools] update lodash to 4.17 (elastic#77317) [APM] Service inventory redesign (elastic#76744) Hide management sections based on cluster/index privileges (elastic#67791) [Snapshot Restore] Disable steps when form is invalid (elastic#76540) [Mappings editor] Add support for positive_score_impact to rank_feature (elastic#76824) Update apm.ts (elastic#77310) [OBS] Remove beta badge, change news feed size and add external icon to news feed link (elastic#77164) [Discover] Convert legacy sort to be compatible with multi sort (elastic#76986) [APM] API Snapshot Testing (elastic#77229) [ML] Functional tests - increase wait time for DFA start (elastic#77307) [UiActions][Drilldowns] Fix actions sorting in context menu (elastic#77162) [Drilldowns] Wire up new links to new docs (elastic#77154) Fix APM issue template [Ingest Pipelines] Drop into an empty tree (elastic#76885) ...

* master: (65 commits) [Security Solution][Resolver] Analyzed event styling (elastic#77115) filter invalid SOs from the searc hresults in Task Manager (elastic#76891) [RUM Dashboard] Visitors by region map (elastic#77135) [Security Solution][Endpoint][Admin] Task/endpoint list actions (elastic#76555) [Ingest pipelines] Forms for processors T-U (elastic#76710) updating datatable type (elastic#77320) [ML] Fix custom URLs processing for security app (elastic#76957) [telemetry] add schema guideline + schema_check new check for --path config (elastic#75747) [ML] Transforms: API schemas and integration tests (elastic#75164) [Mappings editor] Add support for wildcard field type (elastic#76574) [Ingest Manager] Fix flyout instruction selection (elastic#77071) [Telemetry Tools] update lodash to 4.17 (elastic#77317) [APM] Service inventory redesign (elastic#76744) Hide management sections based on cluster/index privileges (elastic#67791) [Snapshot Restore] Disable steps when form is invalid (elastic#76540) [Mappings editor] Add support for positive_score_impact to rank_feature (elastic#76824) Update apm.ts (elastic#77310) [OBS] Remove beta badge, change news feed size and add external icon to news feed link (elastic#77164) [Discover] Convert legacy sort to be compatible with multi sort (elastic#76986) [APM] API Snapshot Testing (elastic#77229) ...

[APM] API Snapshot Testing

fefe6c0

dgieselaar added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 labels Sep 10, 2020

dgieselaar commented Sep 10, 2020

View reviewed changes

dgieselaar added 2 commits September 11, 2020 20:37

Support inline snapshots

818a96a

Make sure received is compared to actual

2c26899

dgieselaar commented Sep 11, 2020

View reviewed changes

dgieselaar added 2 commits September 11, 2020 21:14

Remove unused import

291cbb9

Add more realistic examples of inline snapshot testing

5efece4

dgieselaar requested a review from a team September 12, 2020 14:15

dgieselaar added 2 commits September 12, 2020 17:51

Merge branch 'master' of github.com:elastic/kibana into api-snapshot-…

129780e

…testing

Move literal value checks to (inline) snapshots

4be2387

dgieselaar force-pushed the api-snapshot-testing branch from c07f115 to 4be2387 Compare September 12, 2020 19:45

Fail on obsolete snapshots; Use file-specific unique snapshot names

b94fbbd

dgieselaar mentioned this pull request Sep 13, 2020

[APM] Service inventory redesign #76744

Merged

Merge branch 'master' of github.com:elastic/kibana into api-snapshot-…

541ef2a

…testing

sorenlouv reviewed Sep 14, 2020

View reviewed changes

Remove unnecessary regular function

ba8195d

sorenlouv reviewed Sep 14, 2020

View reviewed changes

sorenlouv approved these changes Sep 14, 2020

View reviewed changes

dgieselaar added 2 commits September 14, 2020 12:05

Make sure snapshots for skipped tests are marked as used

98524d8

Use human-readable timestamps

c98af53

dgieselaar merged commit 8a898fe into elastic:master Sep 14, 2020

dgieselaar deleted the api-snapshot-testing branch September 14, 2020 11:51

dgieselaar mentioned this pull request Sep 14, 2020

[7.x] [APM] API Snapshot Testing (#77229) #77322

Merged

dgieselaar added a commit that referenced this pull request Sep 14, 2020

[7.x] [APM] API Snapshot Testing (#77229) (#77322)

f50ab3f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] API Snapshot Testing #77229

[APM] API Snapshot Testing #77229

dgieselaar commented Sep 10, 2020 •

edited

dgieselaar Sep 10, 2020

dgieselaar Sep 11, 2020

dgieselaar Sep 12, 2020

sorenlouv Sep 14, 2020

dgieselaar Sep 14, 2020

sorenlouv Sep 14, 2020 •

edited

dgieselaar Sep 14, 2020

sorenlouv Sep 14, 2020 •

edited

dgieselaar Sep 14, 2020

sorenlouv left a comment •

edited

kibanamachine commented Sep 14, 2020

		beforeEach(matchSnapshot.init);
		afterEach(matchSnapshot.teardown);

[APM] API Snapshot Testing #77229

[APM] API Snapshot Testing #77229

Conversation

dgieselaar commented Sep 10, 2020 • edited

dgieselaar Sep 10, 2020

Choose a reason for hiding this comment

dgieselaar Sep 11, 2020

Choose a reason for hiding this comment

dgieselaar Sep 12, 2020

Choose a reason for hiding this comment

sorenlouv Sep 14, 2020

Choose a reason for hiding this comment

dgieselaar Sep 14, 2020

Choose a reason for hiding this comment

sorenlouv Sep 14, 2020 • edited

Choose a reason for hiding this comment

dgieselaar Sep 14, 2020

Choose a reason for hiding this comment

sorenlouv Sep 14, 2020 • edited

Choose a reason for hiding this comment

dgieselaar Sep 14, 2020

Choose a reason for hiding this comment

sorenlouv left a comment • edited

Choose a reason for hiding this comment

kibanamachine commented Sep 14, 2020

💚 Build Succeeded

Build metrics

History

dgieselaar commented Sep 10, 2020 •

edited

sorenlouv Sep 14, 2020 •

edited

sorenlouv Sep 14, 2020 •

edited

sorenlouv left a comment •

edited