-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GitHub Actions documentation-deployment pipeline #10610
Conversation
This brings a documentation-deployment pipeline into the Qiskit/Terra repository, allowing it to fully deploy its documentation to `qiskit.org`, a task previously only the metapackage could perform. This does not fully unify the documentation with files from the metapackage, it just adds a pipeline to do the final deployment. This includes a revitalised translatable-strings pipeline, which was previously broken on the metapackage for the last month or two. It also previously included a fair amount of legacy weight that was no longer relevant.
One or more of the the following people are requested to review this:
|
/stable/** | ||
/locale/** | ||
/dev/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that this file is completely different to the one currently on the metapackage, since we're no longer pushing various applications modules / elements to the root of qiskit.org/documentation
any more. A deployment with this drastically stripped-down docs_exclude.txt
will cause some things (like https://qiskit.org/documentation/retworkx) that have currently just about survived to be deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to wait until we have all the redirects in place before deploy then. Right now there are some pages on documentation/*
that have survived as you point out and we don't want links to those pages to 404.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Luciano's on top of this, I believe. Fwiw, the only things affected are retworkx
, metal
, neko
and the index page of partners
.
We also shouldn't add the secrets or actually enable deployment from this PR until the docs build is actually fully complete on Terra, which is still at least a PR away.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The secrets are actually already populated. I added them 3 weeks ago in anticipation of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you'll need to check the translatable-strings ones have the same names I gave them - I think I might have renamed them by accident.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But if you've added the secrets, then we definitely shouldn't allow (at the very least) the push events from this workflow to fire, and most likely we should only merge this PR after the documentation is actually ready to deploy.
tbh, that was probably the more sensible ordering from the start, I just thought this PR would be way easier to write than it was.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in fc27c4a. edit: whoops, wrong comment thread.
Pull Request Test Coverage Report for Build 5837131142
💛 - Coveralls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall this LGTM, the deployment scripts seem fine. My only real concern is about the deployment strategy on stable branches.
/stable/** | ||
/locale/** | ||
/dev/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to wait until we have all the redirects in place before deploy then. Right now there are some pages on documentation/*
that have survived as you point out and we don't want links to those pages to 404.
This changes the logic for the deployments so that pushes to 'stable/*' no longer trigger any deployment to qiskit.org. Instead, tag events trigger a deployment to the relevant stable branch, and a tag event of the _latest_ tag triggers a deployment to the documentation root. The translatables logic is modified to push only the latest full-release tag.
Ok, fc27c4a improves the logic for deployments significantly and fixes a bug that would have caused tag-push events to fail, because GitHub calls receiving a tag event a "push" (totally validly), but I'd accidentally referred to it as "tag". The new logic is:
Example runs, on a repo whose highest version tag before any of these triggers was
In all those runs, I cut down the actual builds to not take forever, and I cut out all the logic that would actually push and just turned it into echoes, etc. |
This is ready to review, I've just marked it on hold because the secrets are in place on Terra, but we're not actually ready for the auto-deployment to start triggering yet. There's still some old documentation redirects that need configuring on qiskit.org, and we need to bring in the last components of the full documentation build from the metapackage - at the moment, it'd deploy an ugly old version of the docs without the correct index pages. |
The new deployment logic causes this PR to fix Qiskit/qiskit-metapackage#1717. |
### Summary This cross-repository merge unifies the documentation, benchmarks and code of conduct from the metapackage into Qiskit/Terra's build. There are very non-trivial merge conflicts that have been resolved by this commit. The summary is: - `CODE_OF_CONDUCT.md`: taken directly from the metapackage's version. - `docs/conf.py`: strongly unified, albeit without the translations components that are added in a separate commit. - `docs/index.rst`: taken almost verbatim from the metapackage. All the API documentation RST files on Terra are moved to `docs/apidoc` (without the trailing 's') to match the metapackage expectation, so the URLs of built documentation will not change. - `docs/release_notes.rst`: The metapackage's version is renamed to `docs/legacy_release_notes.rst`, given a small introductory header, and made an orphan linked only from a _new_ `docs/release_notes.rst` that uses `reno`. - `docs/tutorials.rst`: Mostly these were the same already. Updated to include Qiskit/Terra's correction that it's not an orphan, and contain the metapackage's extra intro tutorial. - `docs/apidoc/terra.rst`: renamed to `docs/apidoc/index.rst` and retitled to be correctly just "API Documentation". - `requirements-dev.txt`: the version of the Sphinx theme is bumped to 1.14 to match the metapackage expectation. Following merge commit ec5c9ca, there is a commit 66a5d9f that fixes the ASV build for use in Terra, which is the rollup of the post-merge commits of #10546, which this PR supersedes. ### Details and comments This should probably be merged ASAP before `main` moves on. Resolutions on the metapackage: - Fix Qiskit/qiskit-metapackage#1723 - Fix Qiskit/qiskit-metapackage#1722 - Fix Qiskit/qiskit-metapackage#1746 After this has merged, #10610 should merge which will close the remaining migration-related issues from the metapackage. The metapackage was prepared for the migration using [`git-filter-repo`](https://github.com/newren/git-filter-repo), with the scripts and configuration files contained within [metapackage_migration.zip](https://github.com/Qiskit/qiskit-terra/files/12324298/metapackage_migration.zip). If you extract that zip, you need to activate a Python 3.11 `venv` then run `metapackage_rewrite.bash` which will prepare the repo in the exact state I merged to create this PR. *edit*: In retrospect writing this, you might need to modify my script so that it pulls only starting from commit Qiskit/qiskit-metapackage@f131daf, which was the tip of `master` at the time I ran this. I tested the docs build locally and it looks as correct as I can tell. There's still big cards pointing to experiments, dynamics etc on the landing page, but I figured that enough's enough, and we can just fix those last two things in Terra. This PR does not include Qiskit/qiskit-metapackage#1791, which should be now cherry-picked onto Terra.
Ok, merge conflicts with #10611 are now resolved and that is merged, so this repository now builds a complete version of the documentation. With the new deployment logic meaning that only a new tag event or manual deployment to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM, I had a couple of inline comments but more just my confirmation of how things worked rather than things to change.
- main | ||
tags: | ||
# Only match non-prerelease tags. | ||
- '[0-9]+.[0-9]+.[0-9]' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it was a regex I'd want to end this in $
to assert there is no rc
or other post notation. I looked at the docs for this: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet and it's not a regex and this should work. This also precludes more than 10 patch versions, but I don't think we've ever needed that. If we do, we can revisit the docs publishing behavior at that time.
echo "Push to unhandled branch '$GITHUB_REF_NAME'" >&2 | ||
exit 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I initially did a double take on this thinking it would cause failures on stable/*
branches. But then I remembered the job is only set to trigger on pushes to main
.
This cross-repository merge unifies the documentation, benchmarks and code of conduct from the metapackage into Qiskit/Terra's build. There are very non-trivial merge conflicts that have been resolved by this commit. The summary is: - `CODE_OF_CONDUCT.md`: taken directly from the metapackage's version. - `docs/conf.py`: strongly unified, albeit without the translations components that are added in a separate commit. - `docs/index.rst`: taken almost verbatim from the metapackage. All the API documentation RST files on Terra are moved to `docs/apidoc` (without the trailing 's') to match the metapackage expectation, so the URLs of built documentation will not change. - `docs/release_notes.rst`: The metapackage's version is renamed to `docs/legacy_release_notes.rst`, given a small introductory header, and made an orphan linked only from a _new_ `docs/release_notes.rst` that uses `reno`. - `docs/tutorials.rst`: Mostly these were the same already. Updated to include Qiskit/Terra's correction that it's not an orphan, and contain the metapackage's extra intro tutorial. - `docs/apidoc/terra.rst`: renamed to `docs/apidoc/index.rst` and retitled to be correctly just "API Documentation". - `requirements-dev.txt`: the version of the Sphinx theme is bumped to 1.14 to match the metapackage expectation. Following merge commit ec5c9ca, there is a commit 66a5d9f that fixes the ASV build for use in Terra, which is the rollup of the post-merge commits of Qiskit#10546, which this PR supersedes. This should probably be merged ASAP before `main` moves on. Resolutions on the metapackage: - Fix Qiskit/qiskit-metapackage#1723 - Fix Qiskit/qiskit-metapackage#1722 - Fix Qiskit/qiskit-metapackage#1746 After this has merged, Qiskit#10610 should merge which will close the remaining migration-related issues from the metapackage. The metapackage was prepared for the migration using [`git-filter-repo`](https://github.com/newren/git-filter-repo), with the scripts and configuration files contained within [metapackage_migration.zip](https://github.com/Qiskit/qiskit-terra/files/12324298/metapackage_migration.zip). If you extract that zip, you need to activate a Python 3.11 `venv` then run `metapackage_rewrite.bash` which will prepare the repo in the exact state I merged to create this PR. *edit*: In retrospect writing this, you might need to modify my script so that it pulls only starting from commit Qiskit/qiskit-metapackage@f131daf, which was the tip of `master` at the time I ran this. I tested the docs build locally and it looks as correct as I can tell. There's still big cards pointing to experiments, dynamics etc on the landing page, but I figured that enough's enough, and we can just fix those last two things in Terra. This PR does not include Qiskit/qiskit-metapackage#1791, which should be now cherry-picked onto Terra. (cherry picked from commit 965fd23)
This commit fixes an issue in the job definition for the new docs deployment job added in Qiskit#10610. The new jobs are conditioned to only execute from the Qiskit/qiskit-terra repo. However that line is being flagged as a syntax error in github. This seems to be due to the double quotes used for `"Qiskit"`. A similar condition exists on the pre-existing github actions jobs but they use a single quote instead of a double and they function correctly. This commit updates the condition to used single quotes to match the working syntax in other jobs.
* Add GitHub Actions documentation-deployment pipeline This brings a documentation-deployment pipeline into the Qiskit/Terra repository, allowing it to fully deploy its documentation to `qiskit.org`, a task previously only the metapackage could perform. This does not fully unify the documentation with files from the metapackage, it just adds a pipeline to do the final deployment. This includes a revitalised translatable-strings pipeline, which was previously broken on the metapackage for the last month or two. It also previously included a fair amount of legacy weight that was no longer relevant. * Add missing secret insertions * Improve logic for deployments This changes the logic for the deployments so that pushes to 'stable/*' no longer trigger any deployment to qiskit.org. Instead, tag events trigger a deployment to the relevant stable branch, and a tag event of the _latest_ tag triggers a deployment to the documentation root. The translatables logic is modified to push only the latest full-release tag. (cherry picked from commit 80e95d1) # Conflicts: # docs/conf.py
This commit fixes an issue in the job definition for the new docs deployment job added in #10610. The new jobs are conditioned to only execute from the Qiskit/qiskit-terra repo. However that line is being flagged as a syntax error in github. This seems to be due to the double quotes used for `"Qiskit"`. A similar condition exists on the pre-existing github actions jobs but they use a single quote instead of a double and they function correctly. This commit updates the condition to used single quotes to match the working syntax in other jobs.
This cross-repository merge unifies the documentation, benchmarks and code of conduct from the metapackage into Qiskit/Terra's build. There are very non-trivial merge conflicts that have been resolved by this commit. The summary is: - `CODE_OF_CONDUCT.md`: taken directly from the metapackage's version. - `docs/conf.py`: strongly unified, albeit without the translations components that are added in a separate commit. - `docs/index.rst`: taken almost verbatim from the metapackage. All the API documentation RST files on Terra are moved to `docs/apidoc` (without the trailing 's') to match the metapackage expectation, so the URLs of built documentation will not change. - `docs/release_notes.rst`: The metapackage's version is renamed to `docs/legacy_release_notes.rst`, given a small introductory header, and made an orphan linked only from a _new_ `docs/release_notes.rst` that uses `reno`. - `docs/tutorials.rst`: Mostly these were the same already. Updated to include Qiskit/Terra's correction that it's not an orphan, and contain the metapackage's extra intro tutorial. - `docs/apidoc/terra.rst`: renamed to `docs/apidoc/index.rst` and retitled to be correctly just "API Documentation". - `requirements-dev.txt`: the version of the Sphinx theme is bumped to 1.14 to match the metapackage expectation. Following merge commit ec5c9ca, there is a commit 66a5d9f that fixes the ASV build for use in Terra, which is the rollup of the post-merge commits of #10546, which this PR supersedes. This should probably be merged ASAP before `main` moves on. Resolutions on the metapackage: - Fix Qiskit/qiskit-metapackage#1723 - Fix Qiskit/qiskit-metapackage#1722 - Fix Qiskit/qiskit-metapackage#1746 After this has merged, #10610 should merge which will close the remaining migration-related issues from the metapackage. The metapackage was prepared for the migration using [`git-filter-repo`](https://github.com/newren/git-filter-repo), with the scripts and configuration files contained within [metapackage_migration.zip](https://github.com/Qiskit/qiskit-terra/files/12324298/metapackage_migration.zip). If you extract that zip, you need to activate a Python 3.11 `venv` then run `metapackage_rewrite.bash` which will prepare the repo in the exact state I merged to create this PR. *edit*: In retrospect writing this, you might need to modify my script so that it pulls only starting from commit Qiskit/qiskit-metapackage@f131daf, which was the tip of `master` at the time I ran this. I tested the docs build locally and it looks as correct as I can tell. There's still big cards pointing to experiments, dynamics etc on the landing page, but I figured that enough's enough, and we can just fix those last two things in Terra. This PR does not include Qiskit/qiskit-metapackage#1791, which should be now cherry-picked onto Terra. (cherry picked from commit 965fd23)
#10638) * Add GitHub Actions documentation-deployment pipeline (#10610) * Add GitHub Actions documentation-deployment pipeline This brings a documentation-deployment pipeline into the Qiskit/Terra repository, allowing it to fully deploy its documentation to `qiskit.org`, a task previously only the metapackage could perform. This does not fully unify the documentation with files from the metapackage, it just adds a pipeline to do the final deployment. This includes a revitalised translatable-strings pipeline, which was previously broken on the metapackage for the last month or two. It also previously included a fair amount of legacy weight that was no longer relevant. * Add missing secret insertions * Improve logic for deployments This changes the logic for the deployments so that pushes to 'stable/*' no longer trigger any deployment to qiskit.org. Instead, tag events trigger a deployment to the relevant stable branch, and a tag event of the _latest_ tag triggers a deployment to the documentation root. The translatables logic is modified to push only the latest full-release tag. (cherry picked from commit 80e95d1) # Conflicts: # docs/conf.py * Fix bad merge conflict * Hardcode publish path from tags For any tag we publish from the `stable/0.25` branch the publish path should be both `stable/0.44` (which matches the qiskit package version) and the root as it's the current release. When we stop publishing from the stable/0.25 branch after the release of 0.45.0rc1 the existing logic will work and we'll no longer have any 0.25.x releases. But for the 0.25.x release series only the version numbers are not unified so we need to manually ensure we're publishing to the right paths. --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Luciano Bello <bel@zurich.ibm.com> Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
This commit fixes an issue in the job definition for the new docs deployment job added in Qiskit#10610. The new jobs are conditioned to only execute from the Qiskit/qiskit-terra repo. However that line is being flagged as a syntax error in github. This seems to be due to the double quotes used for `"Qiskit"`. A similar condition exists on the pre-existing github actions jobs but they use a single quote instead of a double and they function correctly. This commit updates the condition to used single quotes to match the working syntax in other jobs. (cherry picked from commit d25f905)
* Prepare 0.25.1 release This commit prepares the 0.25.1 release, this consists of bumping the version number to 0.25.` in all the appropriate files. It also adds a prelude release note to provide a brief introduction to the release. When this commit merges it should be tagged as the 0.25.1 release. Of particular note for this release, this is the first time we'll be releasing the qiskit package from the qiskit-terra repo and also the first time we're publishing the documentation from this repo too. * Fix repository_owner condition docs_deploy gha job (#10635) This commit fixes an issue in the job definition for the new docs deployment job added in #10610. The new jobs are conditioned to only execute from the Qiskit/qiskit-terra repo. However that line is being flagged as a syntax error in github. This seems to be due to the double quotes used for `"Qiskit"`. A similar condition exists on the pre-existing github actions jobs but they use a single quote instead of a double and they function correctly. This commit updates the condition to used single quotes to match the working syntax in other jobs. (cherry picked from commit d25f905)
* Add GitHub Actions documentation-deployment pipeline This brings a documentation-deployment pipeline into the Qiskit/Terra repository, allowing it to fully deploy its documentation to `qiskit.org`, a task previously only the metapackage could perform. This does not fully unify the documentation with files from the metapackage, it just adds a pipeline to do the final deployment. This includes a revitalised translatable-strings pipeline, which was previously broken on the metapackage for the last month or two. It also previously included a fair amount of legacy weight that was no longer relevant. * Add missing secret insertions * Improve logic for deployments This changes the logic for the deployments so that pushes to 'stable/*' no longer trigger any deployment to qiskit.org. Instead, tag events trigger a deployment to the relevant stable branch, and a tag event of the _latest_ tag triggers a deployment to the documentation root. The translatables logic is modified to push only the latest full-release tag.
This commit fixes an issue in the job definition for the new docs deployment job added in Qiskit#10610. The new jobs are conditioned to only execute from the Qiskit/qiskit-terra repo. However that line is being flagged as a syntax error in github. This seems to be due to the double quotes used for `"Qiskit"`. A similar condition exists on the pre-existing github actions jobs but they use a single quote instead of a double and they function correctly. This commit updates the condition to used single quotes to match the working syntax in other jobs.
…10611) ### Summary This cross-repository merge unifies the documentation, benchmarks and code of conduct from the metapackage into Qiskit/Terra's build. There are very non-trivial merge conflicts that have been resolved by this commit. The summary is: - `CODE_OF_CONDUCT.md`: taken directly from the metapackage's version. - `docs/conf.py`: strongly unified, albeit without the translations components that are added in a separate commit. - `docs/index.rst`: taken almost verbatim from the metapackage. All the API documentation RST files on Terra are moved to `docs/apidoc` (without the trailing 's') to match the metapackage expectation, so the URLs of built documentation will not change. - `docs/release_notes.rst`: The metapackage's version is renamed to `docs/legacy_release_notes.rst`, given a small introductory header, and made an orphan linked only from a _new_ `docs/release_notes.rst` that uses `reno`. - `docs/tutorials.rst`: Mostly these were the same already. Updated to include Qiskit/Terra's correction that it's not an orphan, and contain the metapackage's extra intro tutorial. - `docs/apidoc/terra.rst`: renamed to `docs/apidoc/index.rst` and retitled to be correctly just "API Documentation". - `requirements-dev.txt`: the version of the Sphinx theme is bumped to 1.14 to match the metapackage expectation. Following merge commit 22a406c, there is a commit 66a5d9fe0 that fixes the ASV build for use in Terra, which is the rollup of the post-merge commits of Qiskit/qiskit#10546, which this PR supersedes. ### Details and comments This should probably be merged ASAP before `main` moves on. Resolutions on the metapackage: - Fix Qiskit/qiskit-metapackage#1723 - Fix Qiskit/qiskit-metapackage#1722 - Fix Qiskit/qiskit-metapackage#1746 After this has merged, Qiskit/qiskit#10610 should merge which will close the remaining migration-related issues from the metapackage. The metapackage was prepared for the migration using [`git-filter-repo`](https://github.com/newren/git-filter-repo), with the scripts and configuration files contained within [metapackage_migration.zip](https://github.com/Qiskit/qiskit-terra/files/12324298/metapackage_migration.zip). If you extract that zip, you need to activate a Python 3.11 `venv` then run `metapackage_rewrite.bash` which will prepare the repo in the exact state I merged to create this PR. *edit*: In retrospect writing this, you might need to modify my script so that it pulls only starting from commit Qiskit/qiskit-metapackage@f131daf, which was the tip of `master` at the time I ran this. I tested the docs build locally and it looks as correct as I can tell. There's still big cards pointing to experiments, dynamics etc on the landing page, but I figured that enough's enough, and we can just fix those last two things in Terra. This PR does not include Qiskit/qiskit-metapackage#1791, which should be now cherry-picked onto Terra.
Summary
This brings a documentation-deployment pipeline into the Qiskit/Terra repository, allowing it to fully deploy its documentation to
qiskit.org
, a task previously only the metapackage could perform. This does not fully unify the documentation with files from the metapackage, it just adds a pipeline to do the final deployment.This includes a revitalised translatable-strings pipeline, which was previously broken on the metapackage for the last month or two. It also previously included a fair amount of legacy weight that was no longer relevant.
Details and comments
/
and/stable/0.x
together)... although spiritually they're not totally fixed until we also do Qiskit/qiskit-metapackage#1722, which is most likely coming after this PR, since the deployment of the documentation would still be missing a bunch of top-level files (like the landing page) right now.
This needs the secrets for decrypting the rclone remote config and the SSH key with write access on qiskit-community/qiskit-translations, although in the new form, it may be cleaner to add the SSH key as a direct secret on Terra, since then we can avoid the manual step to mark it as secret (masked) text when moving the decryption into the git credentials.
You can see an example run here: https://github.com/jakelishman/qiskit-terra/actions/runs/5825390347. (Obviously the actual pushes fail due to missing secrets / permissions.)
edit: The following is out-of-date and has been superseded by #10610 (comment).
The strategy for pushing docs to
qiskit.org/documentation
here is:push
event tomain
, deploy to/dev
push
event to anystable/*
, deploy to/
- this needs revisiting when we start supporting more than one branchtag
event, deploy to the relevant/stable/<major>.<minor>
- can be changed if we decide to only push a single form for each majorqiskit.org/documentation/<prefix>
, though in practice anything other than/
,/stable
or/dev
would get removed on the next push to/
.We push the translatable strings on any push to
stable/*
, which has the same caveat as the docs push above.