Skip to content

Conversation

@Fury0508
Copy link

@Fury0508 Fury0508 commented Jan 17, 2026

What change does this PR introduce?

Strips dev dependencies before running provider YAML validation checks by running uv sync --no-dev --all-packages.

Why is this change needed?

Currently, the provider YAML check runs with all dependencies installed, including dev dependencies. This can mask issues where provider code has optional cross-provider dependencies that aren't handled properly (missing try/except blocks for optional imports). By stripping dev dependencies first, we create an environment closer to production and can detect these issues during CI.

Related issue(s)

issue #60662

Changes

  • Added sync_dependencies_without_dev() function that runs uv sync --no-dev --all-packages before provider checks
  • Function is called at the start of the main execution before any validation runs

Testing

  • Ran uv sync --no-dev --all-packages in breeze container to strip dev dependencies
  • Verified providers and their dependencies (including jsonpath-ng from amazon provider) remain installed
  • Executed python scripts/in_container/run_provider_yaml_files_check.py
  • All provider checks passed with 0 errors

Gen-AI Assisted Contribution

Claude AI was consulted for guidance on Airflow's codebase structure and assistance with resolving CI failures. Core implementation and testing were done independently.

@Fury0508 Fury0508 requested a review from jscheffl January 18, 2026 01:09
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK for me.

Just a bit unsure - so would request another pair of eyes - I saw it also today (before this PR) that my venv is "wiped" after running soe prek commands... alwas need to re-sync my UV venv... is it intended by pre-commit checks to change / alter venvs? Or shall a temporary secondary be created for this check then?

Otherwise LGTM

Copy link
Contributor

@bugraoz93 bugraoz93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Fury0508 , we have recently added an AI Agent section in PRs and saw this PR is 16hours old means while you created it should be there. Could you please fill out that part whether it is used or how degree?

@bugraoz93
Copy link
Contributor

- Add sync_dependencies_without_dev() to strip dev dependencies
- Move jsonpath_ng import inside function to avoid ImportError
- Helps detect unhandled optional cross-provider dependencies

Related: apache#60662
- Exit immediately if uv sync --no-dev fails instead of continuing with warning
- Add stdout display for better visibility of sync operation
- Ensures validation never runs with dev dependencies present
@Fury0508 Fury0508 force-pushed the feature/uv-sync-provider-yaml-check branch from fbbef26 to 292796f Compare January 18, 2026 11:46
@Fury0508 Fury0508 requested a review from jscheffl January 18, 2026 11:58
@Fury0508
Copy link
Author

@bugraoz93 Thanks for pointing that out! I've updated the PR description with the Gen-AI disclosure section.

@Fury0508
Copy link
Author

Looks OK for me.

Just a bit unsure - so would request another pair of eyes - I saw it also today (before this PR) that my venv is "wiped" after running soe prek commands... alwas need to re-sync my UV venv... is it intended by pre-commit checks to change / alter venvs? Or shall a temporary secondary be created for this check then?

Otherwise LGTM

Good point! I didn't think about how this would affect local development - only focused on catching the dependency issues in CI.

So yeah, wiping the venv every time someone runs pre-commit hooks is definitely annoying. Should this maybe only run in CI and not locally? Or would creating a temp venv for just this check work better?

Not sure what's the best fix here - happy to adjust based on what you think makes sense!

@bugraoz93
Copy link
Contributor

@bugraoz93 Thanks for pointing that out! I've updated the PR description with the Gen-AI disclosure section.

Thanks @Fury0508!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants