Skip to content

Automate stable REST API permission reference doc generation#67606

Open
durgaprasadml wants to merge 6 commits into
apache:mainfrom
durgaprasadml:feat/automate-api-permission-doc
Open

Automate stable REST API permission reference doc generation#67606
durgaprasadml wants to merge 6 commits into
apache:mainfrom
durgaprasadml:feat/automate-api-permission-doc

Conversation

@durgaprasadml
Copy link
Copy Markdown

Automate stable REST API permission reference doc generation.

This PR introduces an automated extractor for stable REST API permissions and generates a reference RST document directly from the source routes and permission declarations.

Current implementation:

  • statically analyzes /api/v2 route definitions
  • extracts requires_access_* permission metadata
  • generates api_permissions_ref.rst
  • includes regression and invariant tests
  • integrates with existing prek-based generation workflows

This PR is intentionally opened as a draft to gather maintainer feedback on:

  • AST-based extraction vs runtime introspection
  • final file placement
  • generation workflow integration
  • committed generated artifacts

Closes: #10937

@boring-cyborg boring-cyborg Bot added area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch kind:documentation labels May 27, 2026
@boring-cyborg
Copy link
Copy Markdown

boring-cyborg Bot commented May 27, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@durgaprasadml durgaprasadml marked this pull request as ready for review May 29, 2026 06:30
Comment thread airflow-core/docs/security/api_permissions_ref.rst
Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is missing to add it to .pre-commit-config.yaml

@durgaprasadml
Copy link
Copy Markdown
Author

Implemented the requested changes:

  • Added deterministic sorting for generated REST API permission entries (grouped by endpoint path, then HTTP method).
  • Registered the generator in .pre-commit-config.yaml following existing Airflow prek hook conventions.
  • Regenerated api_permissions_ref.rst with the stabilized ordering.

This ensures the generated permission reference stays automatically synchronized with route changes and prevents stale generated docs from passing CI.

@durgaprasadml durgaprasadml requested a review from jscheffl May 29, 2026 12:57
Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for me, any other reviewer opinion? (besides CI needs to turn green...)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR automates generation of stable REST API permission reference documentation by statically extracting requires_access_* dependencies from /api/v2 route definitions and rendering a committed RST reference.

Changes:

  • Adds a stdlib-only AST extractor and CLI for generating/checking API permission docs.
  • Adds regression/invariant tests for route parsing, resource mapping, rendering, and check mode.
  • Registers a prek hook and commits the generated api_permissions_ref.rst artifact.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
scripts/ci/prek/extract_permissions.py New extractor and RST generator for stable REST API permissions.
scripts/tests/ci/prek/test_extract_permissions.py Tests for extractor helpers, real route integration, rendering, and check mode.
airflow-core/docs/security/api_permissions_ref.rst Generated permission reference documentation.
.pre-commit-config.yaml Adds prek hook to regenerate the permission reference.

Comment thread scripts/ci/prek/extract_permissions.py
Comment thread .pre-commit-config.yaml
durgaprasadml and others added 2 commits May 29, 2026 21:49
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@durgaprasadml
Copy link
Copy Markdown
Author

Thanks for the review and the detailed feedback.

Addressed the positional access_entity case for requires_access_dag by extending the AST extraction logic to support both:

  • keyword usage (access_entity=...)
  • second positional argument usage

Also added a regression test covering the positional form (DagAccessEntity.TASK_LOGS) and regenerated the permission reference docs.

Additionally, the generator hook is now registered in .pre-commit-config.yaml to ensure generated docs remain synchronized with route/security changes during CI and local prek runs.

@durgaprasadml durgaprasadml requested a review from jscheffl May 29, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automate reference doc creation for stable API permissions

3 participants