feat: add support for validating OpenAPI specs against test runs #19955

sennyeya · 2023-09-14T20:05:34Z

Hey, I just made a Pull Request!

Optic has a command called optic capture that supports making requests through a proxy that can be validated against the written OpenAPI spec afterwards. It also supports a --update flag for automated updates. Wanting to add this into catalog-backend and search-backend as an initial test.

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

backstage-goalie · 2023-09-14T20:07:35Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-openapi-utils	packages/backend-openapi-utils	patch	`v0.0.5`
@backstage/repo-tools	packages/repo-tools	minor	`v0.3.5`
@backstage/plugin-catalog-backend	plugins/catalog-backend	patch	`v1.14.0`
@backstage/plugin-search-backend	plugins/search-backend	patch	`v1.4.6`

packages/repo-tools/src/commands/openapi/test.ts

github-actions · 2023-09-14T20:22:21Z

Uffizzi Preview deployment-36006 was deleted.

packages/repo-tools/src/commands/openapi/test.ts

sennyeya · 2023-09-18T23:18:40Z

An example of a failed run output, https://github.com/backstage/backstage/actions/runs/6226398906/job/16899164878

jhaals

Cool! Very nice to get this output in ci 👏

I took a peak and added some questions!

optic.yml

package.json

plugins/search-backend/package.json

packages/repo-tools/package.json

packages/repo-tools/src/commands/openapi/test.ts

jhaals · 2023-09-25T11:58:15Z

Looks like we run into some OOM errors in CI 😬

Run yarn backstage-repo-tools schema openapi test
## Processing packages/app
## Processing packages/app-defaults
## Processing packages/app-next

<--- Last few GCs --->

[25510:0x6603670]   262858 ms: Mark-sweep 4050.8 (4[13](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:14)7.8) -> 4039.9 (4[14](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:15)2.6) MB, 4531.0 / 0.0 ms  (average mu = 0.692, current mu = 0.189) allocation failure; scavenge might not succeed
[25510:0x6603670]   269931 ms: Mark-sweep 4056.1 (4143.1) -> 4045.1 (4148.1) MB, 6[17](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:18)7.8 / 0.0 ms  (average mu = 0.487, current mu = 0.127) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb83f50 node::Abort() [/opt/hostedtoolcache/node/[18](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:19).17.1/x64/bin/node]
 2: 0xa94834  [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 3: 0xd647c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 4: 0xd64b67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 5: 0xf42265  [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 6: 0xf5474d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 7: 0xf2ee4e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 8: 0xf30[21](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:22)7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
 9: 0xf113ea v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
10: 0x12d674f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]
11: 0x170[35](https://github.com/backstage/backstage/actions/runs/6267509682/job/17020797745?pr=19955#step:9:36)b9  [/opt/hostedtoolcache/node/18.17.1/x64/bin/node]

packages/repo-tools/src/commands/openapi/test.ts

sennyeya · 2023-09-25T12:22:48Z

@jhaals I think the GC error is due to running the test command in a sub shell? As is, I think we're running 25 node workers. I can adjust this down to 1 for CI as I wasn't hitting this locally.

sennyeya · 2023-10-02T18:19:26Z

@Rugvip This should be ready for a re-review when you get a chance!

freben

Alright, some rough review comments

plugins/search-backend/src/service/router.test.ts

freben · 2023-10-03T11:48:27Z

plugins/search-backend/src/schema/openapi.yaml

  schemas:
+    Error:


This mimics ErrorResponseBody right? If so, it needs some double checking I think. The request field is not required, and the error field has optional stack and code. Check the same in the catalog one of course too

This also raises an interesting question about sharing common schemas across packages, since this is such a central type

Ah yup, will update to match that type.

I've been thinking about the shared common schemas for a while, but kicking it down the road. Entities were the one I was primarily thinking about, but this should also be added to the stack. Interested to hear any ideas here, haven't been able to come up with a solution as yet.

freben · 2023-10-03T12:00:59Z

packages/repo-tools/src/lib/exec.ts

+  return execPromise(
+    [
+      command,
+      ...options.filter(e => e).map(e => (e.startsWith('-') ? e : `"${e}"`)),


Hm, manual quoting is a bit unfortunate. Can this be rewritten to use the spawn('a', ['b', 'c'], { windowsVerbatimArguments: true } form or so?

Thoughts on using the CLI's run command here? I think it would require extracting into cli-node or cli-common, but would come with other benefits like stdout forwarding.

packages/repo-tools/src/commands/openapi/runner.ts

docs/openapi/test-case-validation.md

packages/backend-openapi-utils/README.md

alde

LGTM for the files belonging to catalog

freben · 2023-10-09T14:52:13Z

@sennyeya Seems the builds got real confused here ... things have been improved a lot lately, you may want to rebase (maybe squash too? see if the changeset warning clears up) and force push to kick the tires a bit, until a review gets in

sennyeya · 2023-10-10T14:10:03Z

@freben It should be updated now, the yarn file conflicts are going to be the death of me...

backstage-goalie · 2023-10-22T02:46:15Z

Thanks for the contribution!
All commits need to be DCO signed before they are reviewed. Please refer to the the DCO section in CONTRIBUTING.md or the DCO status for more info.

sennyeya · 2023-10-22T03:00:03Z

packages/repo-tools/src/commands/openapi/test/index.ts

+    return;
+  }
+
+  const opticLocation = (


@freben When you get a chance, one thing that I've been thinking about on top of this one is where to install optic. I think repo-tools is the best place for it, so had to add this to get the binary location to then run in the context of other directories. Hopefully, there's a better way to do this...

one kind of weird thing with Optic is they derive the config file based on context and you can't override it with a passed in value, opticdev/optic#2256. Ideally, we could just run optic in repo-tools and point to each directory, but for now that's a bit of a pipe dream.

What we could do is leave optic as a peer dependency and require the target repo to install it through the root package.json instead. That'd skip the installation for anyone that doesn't use it as well.

Another option to the way it's run is if optic expose a JS API as well that allows us to run these equivalent command straight through JS instead. A bit like we do with Jest here

Rugvip

👍, more or less good to go imo, just need to get deps in order. Rest I think can be future improvements

Rugvip · 2023-10-23T17:09:41Z

packages/repo-tools/src/commands/openapi/test/index.ts

+    return;
+  }
+
+  const opticLocation = (


What we could do is leave optic as a peer dependency and require the target repo to install it through the root package.json instead. That'd skip the installation for anyone that doesn't use it as well.

Another option to the way it's run is if optic expose a JS API as well that allows us to run these equivalent command straight through JS instead. A bit like we do with Jest here

packages/repo-tools/src/commands/openapi/test/init.ts

yarn.lock

packages/repo-tools/package.json

… OpenAPI schema feedback. Signed-off-by: Aramis Sennyey <aramiss@spotify.com> Signed-off-by: Aramis <sennyeyaramis@gmail.com>

Signed-off-by: Aramis <sennyeyaramis@gmail.com> Signed-off-by: Aramis Sennyey <aramiss@spotify.com>

Signed-off-by: Aramis Sennyey <aramiss@spotify.com>

sennyeya · 2023-10-23T18:21:22Z

@Rugvip I can try and get Optic to update their set up upstream, but currently they only support using it through the CLI. Would we want it to be installed as a dependency per your comment on the size of the dependency tree?

Rugvip

Nice! 👍

Let's

github-actions · 2023-10-24T08:55:24Z

Thank you for contributing to Backstage! The changes in this pull request will be part of the 1.20.0 release, scheduled for Tue, 14 Nov 2023.

sennyeya force-pushed the optic-integration-oa branch from 4b658ca to 16c9945 Compare September 14, 2023 20:07

github-actions bot added area:catalog Related to the Catalog Project Area search Things related to Search area:discoverability Related to the Discoverability Project Area labels Sep 14, 2023

sennyeya changed the title ~~Optic integration oa~~ feat: add support for validating OpenAPI specs against test runs Sep 14, 2023

github-advanced-security bot found potential problems Sep 14, 2023

View reviewed changes

packages/repo-tools/src/commands/openapi/test.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Sep 18, 2023

View reviewed changes

packages/repo-tools/src/commands/openapi/test.ts Fixed Show fixed Hide fixed

github-actions bot added the documentation Improvements or additions to documentation label Sep 18, 2023

github-advanced-security bot found potential problems Sep 18, 2023

View reviewed changes

packages/repo-tools/src/commands/openapi/test.ts Fixed Show fixed Hide fixed

packages/repo-tools/src/commands/openapi/test.ts Fixed Show fixed Hide fixed

sennyeya mentioned this pull request Sep 18, 2023

feat(openapi): improve the internal typings and create real public utility types for OpenAPI #20007

Merged

5 tasks

sennyeya marked this pull request as ready for review September 18, 2023 23:24

sennyeya requested review from a team and backstage-service as code owners September 18, 2023 23:24

sennyeya requested review from jhaals, vinzscam and mikeyhc September 18, 2023 23:24

djamaile approved these changes Sep 19, 2023

View reviewed changes

jhaals requested changes Sep 20, 2023

View reviewed changes

optic.yml Outdated Show resolved Hide resolved

package.json Outdated Show resolved Hide resolved

plugins/search-backend/package.json Outdated Show resolved Hide resolved

packages/repo-tools/package.json Show resolved Hide resolved

sennyeya force-pushed the optic-integration-oa branch from a17b6ba to a7ee771 Compare September 21, 2023 21:48

github-advanced-security bot found potential problems Sep 21, 2023

View reviewed changes

packages/repo-tools/src/commands/openapi/test.ts Fixed Show fixed Hide fixed

jhaals reviewed Sep 25, 2023

View reviewed changes

packages/repo-tools/src/commands/openapi/test.ts Outdated Show resolved Hide resolved

sennyeya force-pushed the optic-integration-oa branch from b5691a1 to 73988c4 Compare September 25, 2023 19:33

sennyeya force-pushed the optic-integration-oa branch from 4c62856 to f1cd7bd Compare September 29, 2023 16:01

freben reviewed Oct 3, 2023

View reviewed changes

alde approved these changes Oct 3, 2023

View reviewed changes

sennyeya added the area:openapi-tooling Related to the OpenAPI Tooling Project Area label Oct 5, 2023

sennyeya force-pushed the optic-integration-oa branch from 618e70a to c1574fa Compare October 9, 2023 16:34

sennyeya requested a review from a team as a code owner October 9, 2023 16:34

github-actions bot added the stale label Oct 17, 2023

backstage deleted a comment from github-actions bot Oct 17, 2023

sennyeya removed the stale label Oct 17, 2023

sennyeya force-pushed the optic-integration-oa branch from 600d652 to 8ecf8d6 Compare October 22, 2023 02:39

sennyeya force-pushed the optic-integration-oa branch from bf981ff to dc82ef9 Compare October 22, 2023 02:57

sennyeya commented Oct 22, 2023

View reviewed changes

Rugvip requested changes Oct 23, 2023

View reviewed changes

sennyeya and others added 3 commits October 23, 2023 13:45

feat(repo-tools): add a new command that provides dynamic test-driven…

6694b36

… OpenAPI schema feedback. Signed-off-by: Aramis Sennyey <aramiss@spotify.com> Signed-off-by: Aramis <sennyeyaramis@gmail.com>

update to use optic in repo-tools and local binary dynamically

b7e7d01

Signed-off-by: Aramis <sennyeyaramis@gmail.com> Signed-off-by: Aramis Sennyey <aramiss@spotify.com>

updated optic to a peer dependency

2191bb3

Signed-off-by: Aramis Sennyey <aramiss@spotify.com>

sennyeya force-pushed the optic-integration-oa branch from 1497991 to 2191bb3 Compare October 23, 2023 17:45

sennyeya requested a review from Rugvip October 23, 2023 17:45

fix yarn file

0995903

Signed-off-by: Aramis Sennyey <aramiss@spotify.com>

Rugvip approved these changes Oct 24, 2023

View reviewed changes

Rugvip merged commit 46f92a6 into backstage:master Oct 24, 2023
36 of 37 checks passed

This was referenced Oct 25, 2023

[Meta Issue]: OpenAPI Support #17482

Open

OpenAPI spec for the REST API #2566

Closed

sennyeya mentioned this pull request Nov 27, 2023

fix: search-backend router tests should use standard URL query parameter format #21556

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for validating OpenAPI specs against test runs #19955

feat: add support for validating OpenAPI specs against test runs #19955

sennyeya commented Sep 14, 2023

backstage-goalie bot commented Sep 14, 2023 •

edited

github-actions bot commented Sep 14, 2023 •

edited

sennyeya commented Sep 18, 2023

jhaals left a comment

jhaals commented Sep 25, 2023

sennyeya commented Sep 25, 2023

sennyeya commented Oct 2, 2023

freben left a comment

freben Oct 3, 2023

freben Oct 3, 2023

sennyeya Oct 3, 2023

freben Oct 3, 2023

sennyeya Oct 3, 2023

alde left a comment

freben commented Oct 9, 2023

sennyeya commented Oct 10, 2023

backstage-goalie bot commented Oct 22, 2023

sennyeya Oct 22, 2023

Rugvip Oct 23, 2023

Rugvip left a comment

Rugvip Oct 23, 2023

sennyeya commented Oct 23, 2023

Rugvip left a comment

github-actions bot commented Oct 24, 2023

feat: add support for validating OpenAPI specs against test runs #19955

feat: add support for validating OpenAPI specs against test runs #19955

Conversation

sennyeya commented Sep 14, 2023

Hey, I just made a Pull Request!

✔️ Checklist

backstage-goalie bot commented Sep 14, 2023 • edited

Changed Packages

github-actions bot commented Sep 14, 2023 • edited

sennyeya commented Sep 18, 2023

jhaals left a comment

Choose a reason for hiding this comment

jhaals commented Sep 25, 2023

sennyeya commented Sep 25, 2023

sennyeya commented Oct 2, 2023

freben left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alde left a comment

Choose a reason for hiding this comment

freben commented Oct 9, 2023

sennyeya commented Oct 10, 2023

backstage-goalie bot commented Oct 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rugvip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sennyeya commented Oct 23, 2023

Rugvip left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 24, 2023

backstage-goalie bot commented Sep 14, 2023 •

edited

github-actions bot commented Sep 14, 2023 •

edited