-
Notifications
You must be signed in to change notification settings - Fork 283
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build: add OpenAPI spec bundling, stop using URL references #3288
build: add OpenAPI spec bundling, stop using URL references #3288
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. The local references are much better in my opinion too
12ef49d
to
7ef7302
Compare
@hyperledger/cacti-maintainers I've found a more efficient and less brittle way to do this type of vendoring automatically by way of this concept called bundling the open API specifications. It was originally a feature of the (now deprecated) swagger-cli tool, but the successor of it also implements it and it appears to work seamlessly based on my tests. I'll send an update to this pull request shortly with the refactored version which will be slightly less verbose but equivalent functinality. |
7ef7302
to
41b5fa4
Compare
@hyperledger/cacti-maintainers I've pushed the latest changes and now this is once again ready for your review. I made sure to update the pull request description with the new changes, please re-read carefully! |
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@petermetz Looks great, sorry for late review. I've left few comment but none of them are critical, you can look into them in future PR or here if there's no rush :)
41b5fa4
to
f91a8be
Compare
**IMPORTANT:** From now on, if you are changing the OpenAPI specification of any given package within Cacti, please make sure to edit the template file instead of editing the openapi.json specific file directly because changes in the openapi.json file will be overwritten by the codegen script the next time you run it. This slight alteration in the development flow is the least intrusive solution I could find to resolving our issues with the release automation. This change enables us to have our openapi.json files work without having remote and URL references in them (which was a blocker issue for release automation). 1. The openapi.json files that we used to have are now called openapi.tpl.json where the tpl stands for template. Their content is equivalent to what openapi.json files used to have prior to this commit. 2. These template specs are fed into the bundler tool which then spits out the files which then are saved as openapi.json files. The big change is that these bundled versions are no longer containing any remote nor URL references, only local ones. 3. This means that we still get project-wide re-use of schema types from packages such as cactus-core-api, but we no longer suffer from the additional complexities of having to deal with remote and URL references. 4. The scirpt that performs the bundling is callable separately by executing this command ```sh yarn tools:bundle-open-api-tpl-files ``` 5. The `yarn tools:bundle-open-api-tpl-files` is also embedded as a warmup step of the larger `codegen` script so there is no need usually to call the bundling script separately. 6. The heavylifting in terms of bundling is done by the tooling script that can be found here: `tools/bundle-open-api-tpl-files.ts`. On a high level what it does is loop through existing `openapi.tpl.json` files throughout the project and then renders their bundled version next to it as `openapi.json` which then can be used by all of our tools as a self contained version of the template file which *does* still have the remote and URL references in it. More information on what URL and remote references are can be read here on the official OpenAPI website: https://swagger.io/docs/specification/using-ref/ Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
f91a8be
to
870634b
Compare
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on #3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on #3288 Depends on #3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
IMPORTANT: From now on, if you are changing the OpenAPI specification of any given
package within Cacti, please make sure to edit the template file instead of editing the
openapi.json specific file directly because changes in the openapi.json file will be
overwritten by the codegen script the next time you run it.
This slight alteration in the development flow is the least intrusive solution I could find
to resolving our issues with the release automation.
This change enables us to have our openapi.json files work without having remote and URL
references in them (which was a blocker issue for release automation).
tpl stands for template. Their content is equivalent to what openapi.json files used to
have prior to this commit.
then are saved as openapi.json files. The big change is that these bundled versions are
no longer containing any remote nor URL references, only local ones.
cactus-core-api, but we no longer suffer from the additional complexities of having to deal
with remote and URL references.
yarn tools:bundle-open-api-tpl-files
is also embedded as a warmup step of thelarger
codegen
script so there is no need usually to call the bundling script separately.here:
tools/bundle-open-api-tpl-files.ts
. On a high level what it does is loop throughexisting
openapi.tpl.json
files throughout the project and then renders their bundledversion next to it as
openapi.json
which then can be used by all of our tools as a selfcontained version of the template file which does still have the remote and URL references
in it.
More information on what URL and remote references are can be read here on the official
OpenAPI website: https://swagger.io/docs/specification/using-ref/
Signed-off-by: Peter Somogyvari peter.somogyvari@accenture.com
Pull Request Requirements
upstream/main
branch and squashed into single commit to help maintainers review it more efficient and to avoid spaghetti git commit graphs that obfuscate which commit did exactly what change, when and, why.-s
flag when usinggit commit
command. You may refer to this link for more information.Character Limit
A Must Read for Beginners
For rebasing and squashing, here's a must read guide for beginners.