Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: add OpenAPI spec bundling, stop using URL references #3288

Conversation

petermetz
Copy link
Contributor

@petermetz petermetz commented Jun 1, 2024

IMPORTANT: From now on, if you are changing the OpenAPI specification of any given
package within Cacti, please make sure to edit the template file instead of editing the
openapi.json specific file directly because changes in the openapi.json file will be
overwritten by the codegen script the next time you run it.
This slight alteration in the development flow is the least intrusive solution I could find
to resolving our issues with the release automation.

This change enables us to have our openapi.json files work without having remote and URL
references in them (which was a blocker issue for release automation).

  1. The openapi.json files that we used to have are now called openapi.tpl.json where the
    tpl stands for template. Their content is equivalent to what openapi.json files used to
    have prior to this commit.
  2. These template specs are fed into the bundler tool which then spits out the files which
    then are saved as openapi.json files. The big change is that these bundled versions are
    no longer containing any remote nor URL references, only local ones.
  3. This means that we still get project-wide re-use of schema types from packages such as
    cactus-core-api, but we no longer suffer from the additional complexities of having to deal
    with remote and URL references.
  4. The scirpt that performs the bundling is callable separately by executing this command
yarn tools:bundle-open-api-tpl-files
  1. The yarn tools:bundle-open-api-tpl-files is also embedded as a warmup step of the
    larger codegen script so there is no need usually to call the bundling script separately.
  2. The heavylifting in terms of bundling is done by the tooling script that can be found
    here: tools/bundle-open-api-tpl-files.ts. On a high level what it does is loop through
    existing openapi.tpl.json files throughout the project and then renders their bundled
    version next to it as openapi.json which then can be used by all of our tools as a self
    contained version of the template file which does still have the remote and URL references
    in it.

More information on what URL and remote references are can be read here on the official
OpenAPI website: https://swagger.io/docs/specification/using-ref/

Signed-off-by: Peter Somogyvari peter.somogyvari@accenture.com

Pull Request Requirements

  • Rebased onto upstream/main branch and squashed into single commit to help maintainers review it more efficient and to avoid spaghetti git commit graphs that obfuscate which commit did exactly what change, when and, why.
  • Have git sign off at the end of commit message to avoid being marked red. You can add -s flag when using git commit command. You may refer to this link for more information.
  • Follow the Commit Linting specification. You may refer to this link for more information.

Character Limit

  • Pull Request Title and Commit Subject must not exceed 72 characters (including spaces and special characters).
  • Commit Message per line must not exceed 80 characters (including spaces and special characters).

A Must Read for Beginners
For rebasing and squashing, here's a must read guide for beginners.

Copy link
Contributor

@jagpreetsinghsasan jagpreetsinghsasan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The local references are much better in my opinion too

@petermetz petermetz force-pushed the chore-vendor-openapi-json-files-v200alpha2 branch from 12ef49d to 7ef7302 Compare June 11, 2024 13:22
@petermetz
Copy link
Contributor Author

@hyperledger/cacti-maintainers I've found a more efficient and less brittle way to do this type of vendoring automatically by way of this concept called bundling the open API specifications. It was originally a feature of the (now deprecated) swagger-cli tool, but the successor of it also implements it and it appears to work seamlessly based on my tests. I'll send an update to this pull request shortly with the refactored version which will be slightly less verbose but equivalent functinality.

@petermetz petermetz force-pushed the chore-vendor-openapi-json-files-v200alpha2 branch from 7ef7302 to 41b5fa4 Compare June 12, 2024 19:12
@petermetz
Copy link
Contributor Author

@hyperledger/cacti-maintainers I've pushed the latest changes and now this is once again ready for your review. I made sure to update the pull request description with the new changes, please re-read carefully!

@petermetz petermetz changed the title chore: vendor v2.0.0-alpha.2 openapi.json files project-wide build: add OpenAPI spec bundling, stop using URL references Jun 12, 2024
petermetz added a commit to petermetz/cacti that referenced this pull request Jun 12, 2024
On a high level this is a find & replace operation where the occurrences of the
first bullet point were replaced with the second bullet point:
* `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2`
* `"$ref": "../../../../..`

The firs bullet point above is called a URL reference while the second one is
called a REMOTE references (remote as in a different spec file on the file-system).

1. With this change, we unlock the release process being able to issue code that
is working on the latest OpenAPI specifications that we are cross-referencing
from one package to another.
2. Previously you had to manually update the references in about a hundred
and fifty locations to make sure that the versions are bumped but after this
change this happens automatically as the newly introduced bundling process
and the usage of the REMOTE references instead of URL references.
3. The problem so far with the release process was that with the URL references
we dependended on the existence of a pushed git tag for a successful release build.
But we cannot git push the tag before having performed a successful release build,
so this was a chicken-egg problem that had to be somehow untangled from its
circular dependency hell and this change is what makes it happen by no longer
depending on the git tags having been pushed to the upstream repository.

Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175

Depends on hyperledger-cacti#3288

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this pull request Jun 12, 2024
1. After this change the steps within the release management documentation should
work without issues.
2. Currently the process is (was) broken due to our reliance on URL references
within the OpenAPI specifications which created a chicken-egg problem with the
release tag issuance and the building of the source code to be released.

This change depends on the other pull requests that are refactoring the cross-package
OpenAPI specification references:

Depends on hyperledger-cacti#3288
Depends on hyperledger-cacti#3315

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Copy link
Contributor

@outSH outSH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@petermetz Looks great, sorry for late review. I've left few comment but none of them are critical, you can look into them in future PR or here if there's no rush :)

tools/bundle-open-api-tpl-files.ts Outdated Show resolved Hide resolved
tools/bundle-open-api-tpl-files.ts Outdated Show resolved Hide resolved
@petermetz petermetz force-pushed the chore-vendor-openapi-json-files-v200alpha2 branch from 41b5fa4 to f91a8be Compare June 13, 2024 20:23
**IMPORTANT:** From now on, if you are changing the OpenAPI specification of any given
package within Cacti, please make sure to edit the template file instead of editing the
openapi.json specific file directly because changes in the openapi.json file will be
overwritten by the codegen script the next time you run it.
This slight alteration in the development flow is the least intrusive solution I could find
to resolving our issues with the release automation.

This change enables us to have our openapi.json files work without having remote and URL
references in them (which was a blocker issue for release automation).

1. The openapi.json files that we used to have are now called openapi.tpl.json where the
tpl stands for template. Their content is equivalent to what openapi.json files used to
have prior to this commit.
2. These template specs are fed into the bundler tool which then spits out the files which
then are saved as openapi.json files. The big change is that these bundled versions are
no longer containing any remote nor URL references, only local ones.
3. This means that we still get project-wide re-use of schema types from packages such as
cactus-core-api, but we no longer suffer from the additional complexities of having to deal
with remote and URL references.
4. The scirpt that performs the bundling is callable separately by executing this command
```sh
yarn tools:bundle-open-api-tpl-files
```
5. The `yarn tools:bundle-open-api-tpl-files` is also embedded as a warmup step of the
larger `codegen` script so there is no need usually to call the bundling script separately.
6. The heavylifting in terms of bundling is done by the tooling script that can be found
here: `tools/bundle-open-api-tpl-files.ts`. On a high level what it does is loop through
existing `openapi.tpl.json` files throughout the project and then renders their bundled
version next to it as `openapi.json` which then can be used by all of our tools as a self
contained version of the template file which *does* still have the remote and URL references
in it.

More information on what URL and remote references are can be read here on the official
OpenAPI website: https://swagger.io/docs/specification/using-ref/

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz force-pushed the chore-vendor-openapi-json-files-v200alpha2 branch from f91a8be to 870634b Compare June 13, 2024 21:24
@petermetz petermetz merged commit 4a9ed0a into hyperledger-cacti:main Jun 13, 2024
139 of 149 checks passed
@petermetz petermetz deleted the chore-vendor-openapi-json-files-v200alpha2 branch June 13, 2024 21:35
petermetz added a commit to petermetz/cacti that referenced this pull request Jun 13, 2024
On a high level this is a find & replace operation where the occurrences of the
first bullet point were replaced with the second bullet point:
* `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2`
* `"$ref": "../../../../..`

The firs bullet point above is called a URL reference while the second one is
called a REMOTE references (remote as in a different spec file on the file-system).

1. With this change, we unlock the release process being able to issue code that
is working on the latest OpenAPI specifications that we are cross-referencing
from one package to another.
2. Previously you had to manually update the references in about a hundred
and fifty locations to make sure that the versions are bumped but after this
change this happens automatically as the newly introduced bundling process
and the usage of the REMOTE references instead of URL references.
3. The problem so far with the release process was that with the URL references
we dependended on the existence of a pushed git tag for a successful release build.
But we cannot git push the tag before having performed a successful release build,
so this was a chicken-egg problem that had to be somehow untangled from its
circular dependency hell and this change is what makes it happen by no longer
depending on the git tags having been pushed to the upstream repository.

Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175

Depends on hyperledger-cacti#3288

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit that referenced this pull request Jun 14, 2024
On a high level this is a find & replace operation where the occurrences of the
first bullet point were replaced with the second bullet point:
* `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2`
* `"$ref": "../../../../..`

The firs bullet point above is called a URL reference while the second one is
called a REMOTE references (remote as in a different spec file on the file-system).

1. With this change, we unlock the release process being able to issue code that
is working on the latest OpenAPI specifications that we are cross-referencing
from one package to another.
2. Previously you had to manually update the references in about a hundred
and fifty locations to make sure that the versions are bumped but after this
change this happens automatically as the newly introduced bundling process
and the usage of the REMOTE references instead of URL references.
3. The problem so far with the release process was that with the URL references
we dependended on the existence of a pushed git tag for a successful release build.
But we cannot git push the tag before having performed a successful release build,
so this was a chicken-egg problem that had to be somehow untangled from its
circular dependency hell and this change is what makes it happen by no longer
depending on the git tags having been pushed to the upstream repository.

Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175

Depends on #3288

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this pull request Jun 14, 2024
1. After this change the steps within the release management documentation should
work without issues.
2. Currently the process is (was) broken due to our reliance on URL references
within the OpenAPI specifications which created a chicken-egg problem with the
release tag issuance and the building of the source code to be released.

This change depends on the other pull requests that are refactoring the cross-package
OpenAPI specification references:

Depends on hyperledger-cacti#3288
Depends on hyperledger-cacti#3315

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit that referenced this pull request Jun 14, 2024
1. After this change the steps within the release management documentation should
work without issues.
2. Currently the process is (was) broken due to our reliance on URL references
within the OpenAPI specifications which created a chicken-egg problem with the
release tag issuance and the building of the source code to be released.

This change depends on the other pull requests that are refactoring the cross-package
OpenAPI specification references:

Depends on #3288
Depends on #3315

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
fazzatti pushed a commit to fazzatti/cacti that referenced this pull request Jun 24, 2024
On a high level this is a find & replace operation where the occurrences of the
first bullet point were replaced with the second bullet point:
* `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2`
* `"$ref": "../../../../..`

The firs bullet point above is called a URL reference while the second one is
called a REMOTE references (remote as in a different spec file on the file-system).

1. With this change, we unlock the release process being able to issue code that
is working on the latest OpenAPI specifications that we are cross-referencing
from one package to another.
2. Previously you had to manually update the references in about a hundred
and fifty locations to make sure that the versions are bumped but after this
change this happens automatically as the newly introduced bundling process
and the usage of the REMOTE references instead of URL references.
3. The problem so far with the release process was that with the URL references
we dependended on the existence of a pushed git tag for a successful release build.
But we cannot git push the tag before having performed a successful release build,
so this was a chicken-egg problem that had to be somehow untangled from its
circular dependency hell and this change is what makes it happen by no longer
depending on the git tags having been pushed to the upstream repository.

Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175

Depends on hyperledger-cacti#3288

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
fazzatti pushed a commit to fazzatti/cacti that referenced this pull request Jun 24, 2024
1. After this change the steps within the release management documentation should
work without issues.
2. Currently the process is (was) broken due to our reliance on URL references
within the OpenAPI specifications which created a chicken-egg problem with the
release tag issuance and the building of the source code to be released.

This change depends on the other pull requests that are refactoring the cross-package
OpenAPI specification references:

Depends on hyperledger-cacti#3288
Depends on hyperledger-cacti#3315

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES pushed a commit to sandeepnRES/cacti that referenced this pull request Jul 30, 2024
On a high level this is a find & replace operation where the occurrences of the
first bullet point were replaced with the second bullet point:
* `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2`
* `"$ref": "../../../../..`

The firs bullet point above is called a URL reference while the second one is
called a REMOTE references (remote as in a different spec file on the file-system).

1. With this change, we unlock the release process being able to issue code that
is working on the latest OpenAPI specifications that we are cross-referencing
from one package to another.
2. Previously you had to manually update the references in about a hundred
and fifty locations to make sure that the versions are bumped but after this
change this happens automatically as the newly introduced bundling process
and the usage of the REMOTE references instead of URL references.
3. The problem so far with the release process was that with the URL references
we dependended on the existence of a pushed git tag for a successful release build.
But we cannot git push the tag before having performed a successful release build,
so this was a chicken-egg problem that had to be somehow untangled from its
circular dependency hell and this change is what makes it happen by no longer
depending on the git tags having been pushed to the upstream repository.

Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175

Depends on hyperledger-cacti#3288

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES pushed a commit to sandeepnRES/cacti that referenced this pull request Jul 30, 2024
1. After this change the steps within the release management documentation should
work without issues.
2. Currently the process is (was) broken due to our reliance on URL references
within the OpenAPI specifications which created a chicken-egg problem with the
release tag issuance and the building of the source code to be released.

This change depends on the other pull requests that are refactoring the cross-package
OpenAPI specification references:

Depends on hyperledger-cacti#3288
Depends on hyperledger-cacti#3315

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants