ci: Add nightly OpenAPI spec sync and make model codegen reproducible - #983
Draft
vdusek wants to merge 2 commits into
Draft
ci: Add nightly OpenAPI spec sync and make model codegen reproducible#983vdusek wants to merge 2 commits into
vdusek wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #983 +/- ##
==========================================
- Coverage 94.62% 94.62% -0.01%
==========================================
Files 58 58
Lines 5248 5247 -1
==========================================
- Hits 4966 4965 -1
Misses 282 282
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the docs-PR-triggered model regeneration with a nightly job, and makes the generated files reproducible from committed state so drift can no longer slip in unnoticed.
What changes
spec/openapi.json- a committed snapshot of the published spec, now the only input codegen reads..gitattributesmarks itlinguist-generatedso it collapses in diffs (one click away) instead of burying the model diff, and doesn't skew the repository language stats.scripts/update_openapi_spec.py+poe update-spec- downloads the published spec, refuses anything that isn't a plausible OpenAPI document (size, required top-level members) so an error page can't overwrite the snapshot, and normalizes the formatting. Key order is deliberately preserved:keep_model_orderties generated model order to the spec.poe generate-modelsnow generates from the snapshot instead of the live URL.generate-models-from-fileis redundant with that and is removed - to try a candidate spec, overwrite the snapshot and regenerate.poe check-models(new, part ofcheck-codeand the newModels checkCI job) regenerates, compares, and restores the working tree. It never leaves the generated files modified and doesn't care whether they're committed, so it's safe to run mid-change.on_schedule_regenerate_models.yaml- nightly at 02:00 UTC plus manual dispatch, modelled on Crawlee'supdate_playwright_version.yaml. Refreshes the snapshot, regenerates, and opens a PR when the models change. Scheduled failures go to Slack, following Crawlee'son_schedule_tests.yaml.manual_regenerate_models.yamlis deleted._models.pyloses thePlan.available_proxy_groupsdocstring - see below.Why
Two independent problems, both visible in #979:
The generated files weren't a function of committed state. The spec was fetched live and the codegen toolchain floated, so nobody could tell whether
_models.pywas current. Renovate moveddatamodel-code-generator0.68.1 -> 0.69.0 (#968) -> 0.71.0 (#975) andruffto 0.16 (#978); master's generated output no longer matched what that toolchain produces.check-modelscloses this: a codegen or formatter upgrade that changes output now turns its own dependency-bump PR red, so that PR carries the regenerated files instead of the drift contaminating an unrelated spec change later. The accumulated drift is thePlan.available_proxy_groupsdocstring removal in this PR - the description is still in the spec, onAvailableProxyGroups.additionalProperties, so this is purely codegen catching up.The regeneration branch rotted. The manual branch was permanent and never rebased, and the workflow checked it out before regenerating - so it generated with the codegen tooling from that stale branch. In #979 that produced
_literals.pyin the pre-#941 closedLiteral[...]form; merging it would have silently reverted the enum relaxation. The workflow's own comment stated the intent ("regenerating on master keeps the codegen tooling current") but that only held on the branch-creation path.The new workflow fixes both structurally:
ci/regenerate-modelsbranch from master rather than appending, so the PR diff is always "current spec vs current master" and can't resurrect a stale generated file. The previous PR is closed and its branch deleted first, which also avoids the window where the branch tip equals master and GitHub auto-closes the PR as empty.info.versioncarries a build timestamp that apify-docs bumps on every spec change, so a spec diff alone says nothing about client impact.chore:title instead of aTODOthat blockspr-title-check. A design that requires human action to become mergeable is what let fix: Mark generated model fields that the API can omit as optional #979 rot for 20 days, while the same spec changes were landed by hand (docs: fix dataset schema link in generated Dataset model #923, fix: Relax constraints on generated models based on updated specification #936, fix: relax required stats fields (compute units, counts) to optional #947, fix: Add missing cannot-monetize-without-payout-billing-info error code #960, docs: Update description ofrequest_urlinWebhooks#974). Reviewers retitle tofix:/feat:when the diff is user-facing, so it reaches the changelog and triggers a release.Notes
manual_regenerate_models.yamldeleted here.SLACK_WEBHOOK_URLrepository secret for the failure notification. Without it the alert step fails, but only on a run that has already failed.Plan.available_proxy_groupsloses its description becausecollapse_root_modelsdiscards the$ref-ed schema's docs, and postprocess doesn't carry them over. That predates this PR (it came with the generator bump) and applies to every$ref-ed root model, so it's worth fixing separately inpostprocess_generated_models.py.The second commit applies review follow-ups: the "already up for review" gate now also requires an open PR (a leftover branch would otherwise silently suppress every future regeneration), the sync script validates
info.versionbefore overwriting the snapshot, retries transient download failures, writes LF endings, resolves its path from the repo root, and reports errors on stderr.✍️ Drafted by Claude Code