[v3-2-test] remove redundant quote in integration-system-tests.yml (#65381)#65385
Draft
github-actions[bot] wants to merge 102 commits intov3-2-testfrom
Draft
[v3-2-test] remove redundant quote in integration-system-tests.yml (#65381)#65385github-actions[bot] wants to merge 102 commits intov3-2-testfrom
github-actions[bot] wants to merge 102 commits intov3-2-testfrom
Conversation
…#63826) (#64723) * Load hook metadata from YAML without importing Hook class * Add hook-name to all provider.yaml connection-types * Add hook-name to connection types and regenerate get_provider_info.py * Fix ruff import order in connections.py * fix: import ProvidersManager at top level per review * Fix provider connection hook display names * Add iter_connection_type_hook_ui_metadata for connection UI hook metadata (cherry picked from commit c4a209b) Co-authored-by: Yuseok Jo <yuseok89@gmail.com>
…zedDagModel (#64322) (#64738) * fix(scheduler): skip asset-triggered dags without SerializedDagModel in dags_needing_dagruns Remove those dag_ids from the in-memory candidate set until serialization exists; retain AssetDagRunQueue rows and emit DEBUG logs. Add unit tests and a bugfix newsfragment. * fix(scheduler): prevent premature asset-triggered DagRuns when SerializedDagModel is unavailable * test(dag): persist DagModel before AssetDagRunQueue in unit tests Split DagModel and AssetDagRunQueue inserts and flush after DagModel so foreign-key order matches production DB constraints in TestDagModel. * Apply suggestions from code review * refactor(dag): clarify ADRQ skip log and condense serialized-DAG guard Combine the missing-from-serialized set check with a walrus assignment and improve the debug message when DagRun creation is skipped for DAGs with queued asset events but no SerializedDagModel row. * test(models): align caplog assertions with updated serialized dag warnings * test(dag): align ADRQ missing-serialized log assertion with message text * Apply suggestion from @jscheffl * chore(newsfragments): remove 64322.bugfix.rst --------- (cherry picked from commit b91394a) Co-authored-by: Leonardo Soares <leoss33@outlook.com.br> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
…IH (#61631) (#64693) * Return 410 Gone for heartbeat when TI was cleared and moved to TIH When a running task instance is cleared, its previous try is archived to the Task Instance History table and the TI receives a new UUID. Subsequent heartbeats from the old process get a 404 because the old UUID no longer exists in the TI table. This change improves the error handling by checking the TIH table when a heartbeat TI is not found. If the UUID exists in TIH, return 410 Gone instead of 404 Not Found, giving the client a more specific signal that the task was cleared rather than never existing. - Server: check TIH on heartbeat NoResultFound, return 410 if found - Supervisor: handle 410 Gone same as 404/409 (terminate process) - Keep 404 for TIs that genuinely never existed closes: #53140 * Update task_instances.py * Update task_instances.py * Update test_task_instances.py * fix(api): use task_instance_id in heartbeat 410 path and align detail message - Replace undefined ti_id_str with task_instance_id in TIH query and log - Use task_instance_id (UUID) for TIH.task_instance_id comparison - Set 410 Gone detail message to match test expectation --------- (cherry picked from commit ce1270b) Co-authored-by: André Ahlert <andre@aex.partners> Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
… requirement (#64901) (#64903) The `labeled`/`unlabeled` event types triggered this workflow when boring-cyborg bot added labels, which required manual approval to run. Since this check only needs to run on code changes (opened, reopened, synchronize), removing the label event types avoids unnecessary workflow runs that need approval. (cherry picked from commit 40902de) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…e PR workflow (#64836) (#64915) Move upgrade checks from basic-tests.yml (where they failed the build) to a scheduled workflow that runs Mon/Wed/Fri for main and Tue/Thu for v3-2-test. The workflow uses `breeze ci upgrade` to run all upgrade steps and create a draft PR. A Slack notification is sent to #internal-airflow-ci-cd when a PR needs review. Changes: - Remove upgrade-check job from basic-tests.yml - Add reusable upgrade-check.yml workflow (workflow_call) - Add scheduled-upgrade-check.yml caller (cron + workflow_dispatch) - Enhance breeze ci upgrade: --draft flag, stable branch naming, existing PR detection, no -w flag for CI compatibility (cherry picked from commit dd0ff5a)
… (#64916) * Skip pre-commit checks on automated CI tool upgrades * Skip pre-commit checks on automated CI tool upgrades * Review feedback, attempt with checks once, collect auto-fixes and commit again * Shorten code (cherry picked from commit 93552c2) Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
…evant dirs (#64927) (#64930) Replace two rglob calls with a single os.walk that prunes node_modules and hidden directories (e.g. .git, .venv) in-place, avoiding unnecessary traversal of large directory trees that never contain relevant .pyc files. (cherry picked from commit 27258d5) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…arrow toggl… (#64284) (#64919) * open task group when navigating via group name; preserve arrow toggle behavior * Remove redundant comment. * Update airflow-core/src/airflow/ui/src/layouts/Details/Grid/TaskNames.tsx * Update airflow-core/src/airflow/ui/src/layouts/Details/Grid/TaskNames.tsx * Fixing static checks * Update TaskNames.tsx --------- (cherry picked from commit 11a0aa8) Co-authored-by: fat-catTW <124506982+fat-catTW@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
…worker recycling (#65036) (#65037) * Add FAQ entry for API server memory growth with gunicorn fix Users experiencing API server memory growth from accumulated serialized DAG cache entries can use gunicorn with rolling worker restarts as the recommended solution. This FAQ entry documents the configuration and cross-references the dag version inflation FAQ. * Add version availability note for gunicorn support (cherry picked from commit 74283fb) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…5045) * Sync default image Python version references * Fix mypy type error in _read_global_constants_assignment return type Since ast.literal_eval() returns Any, annotating the private helper as returning str | list[str] caused mypy errors in both callers: assigning the result to `list[str]` and returning it as `str`. Changing the return type to Any (which matches ast.literal_eval) resolves both without requiring casts. * Update scripts/ci/prek/common_prek_utils.py --------- (cherry picked from commit d212f62) Co-authored-by: DaveT1991 <129008657+DaveT1991@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…4999) (#65094) Add shared/*.py pattern to ALL_SOURCE_FILES so changes that only touch the shared libraries (e.g., shared/timezones, shared/logging) trigger the unit test suite. Previously such changes were missed by selective checks because the shared/ folder was not part of the source files patterns, leading to skipped tests for changes that affect symlinked code in airflow-core, task-sdk, and providers. (cherry picked from commit f13b713) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
… multi_team=False (#65275) (#65282) get_team_name_dep() declared AsyncSessionDep as a FastAPI dependency parameter, which meant FastAPI checked out an async database connection on every Execution API request -- even when multi_team=False (the default), where the function immediately returned None without using the session. Remove the AsyncSessionDep parameter and create the session inside the function only when multi_team=True. This avoids a connection checkout, no-op commit round-trip, and connection return on every connection and variable lookup from workers. (cherry picked from commit 1e64f41) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…aints (#65244) (#65255) Pass exact version specifiers when installing apache-airflow, apache-airflow-core, and apache-airflow-task-sdk for the highest resolution constraint build so uv resolves against the versions currently declared in-tree rather than the latest published on PyPI. (cherry picked from commit d31f6ac) Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
…5284) When _configure_async_session() was extracted from configure_orm() in PR #51920, the corresponding cleanup in dispose_orm() was not updated. This left async_engine connections abandoned on process exit, gunicorn worker restarts, and atexit -- gradually exhausting PostgreSQL's max_connections. Dispose async_engine.sync_engine (the synchronous path, matching the existing clean_in_fork pattern) and clear both async_engine and AsyncSession references. (cherry picked from commit 69a88bf) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…) (#65293) * Add overridable metadata engine creation hooks in settings.py * retrigger CI * Add tests and docs for metadata engine creation hooks Address review feedback: add unit tests proving configure_orm() and _configure_async_session() delegate to the overridable hooks, verify default implementations forward args correctly, and confirm overrides via airflow_local_settings.py take effect. Document the new hooks in cluster-policies.rst with function signatures and a JWT example. * Fix ruff formatting and restore settings attrs in teardown Ruff format required single-line function signatures. More importantly, teardown_method now restores the original create_metadata_engine and create_async_metadata_engine on the settings module to prevent the override from test_override_via_local_settings from leaking into subsequent test classes (e.g. test_sqlalchemy_config.py). * Update PR with Lee's comments (cherry picked from commit 85ccf1a) Co-authored-by: Arthur <arthur.volant@datadoghq.com>
* fix(connection): check pre-normalized conn_type when warning about presence of underscores in conn_type * feat(connection): added test_get_uri_conn_type_warning method to TestConnection * fix(connection): use conn_type attr as default value when checking conn_type for presence of underscores in get_uri * fix(connection): generalized test cases used by test_get_uri_conn_type_warning test to include connections instantiated using a URI and connections instantiated not using a URI * fix(connection): coalesce conn_type value to an empty string before checking for presence of underscores in conn_type in get_uri * refactor(connection): updated assertion messages in test_get_uri_conn_type_warning method * style(connection): formatting changes according to prek static checks (cherry picked from commit 06c75db) Co-authored-by: Michael Trossbach <mptrossbach@gmail.com>
* fix: use Dag form when materializing asset * fixup! fix: use Dag form when materializing asset * fixup! fixup! fix: use Dag form when materializing asset * fixup! fixup! fixup! fix: use Dag form when materializing asset * fixup! fixup! fixup! fixup! fix: use Dag form when materializing asset * test: add test case test_should_respond_200_with_trigger_fields_without_dag_run_id * fix: restore useTrigger in TriggerDAGModal after TriggerDAGForm refactor TriggerDAGForm no longer owns the trigger mutation after the refactor (onSubmitTrigger was made a prop). TriggerDAGModal must now supply useTrigger and pass error/isPending/onSubmitTrigger down to the form, otherwise clicking Trigger does nothing and dagRunId stays null. (cherry picked from commit b312314) Co-authored-by: Wei Lee <weilee.rx@gmail.com>
…#65312) The zip import error fix (#63617) changed the public signature of `DagFileProcessorManager.deactivate_deleted_dags` from `(bundle_name, present: set[DagFileInfo])` to `(bundle_name, observed_filelocs: set[str])`, breaking subclass overrides. Restore the original signature and compute observed filelocs internally. Also widen `DagModel.deactivate_deleted_dags` `rel_filelocs` type from `set[str]` to `Collection[str]` to accept both list and set callers. (cherry picked from commit 9ad468c) Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
* Add dag runs filters (Consuming Asset) * Fix: correct consuming asset filter setup using association_table * Trigger CI rebuild * Rename consuming_asset filter to consuming_asset_pattern with database icon * Rename consuming_asset filter to consuming_asset_pattern with database icon * Trigger CI rebuild * Fix consuming_asset_pattern naming * Fix: rename consuming_asset to consuming_asset_pattern * Fix: rename consuming_asset to consuming_asset_pattern * Fix: Resolve PostgreSQL JSON comparison error in _ConsumingAssetFilter * Rebase and fix _ConsumingAssetFilter * Trigger CI * add consumingAsset and filters.searchAsset to en/common.json --------- (cherry picked from commit 5245419) Co-authored-by: fat-catTW <124506982+fat-catTW@users.noreply.github.com> Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…tance (#63923) (#65304) After clearing a task instance, the TaskInstances list page was not refreshing to show the updated state. This was because `useClearTaskInstances` was missing `[useTaskInstanceServiceGetTaskInstancesKey]` in the list of query keys to invalidate on success. Both `useClearRun` and `usePatchTaskInstance` correctly invalidate this query — this change brings `useClearTaskInstances` in line with them. Fixes: #60703 (cherry picked from commit f47038e) Co-authored-by: nagasrisai <59650078+nagasrisai@users.noreply.github.com>
The sphinx_airflow_theme default navbar_links includes Registry, but the docs override navbar_links in get_html_theme_options(), so the theme default never applies. (cherry picked from commit d988f75) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…e_fileloc + bundle (#65329) (#65343) The public Import Errors API used to match ParseImportError.filename against DagModel.fileloc. In real deployments ``fileloc`` is an absolute path while ``filename`` is relative, so the file-to-DAG resolution often came back empty and the single endpoint fell through to returning the raw error. The list endpoint had a related gap: its CTE was pre-filtered by the caller-visible subset of DAGs, so the per-file authorization check only ever saw the DAGs the caller could already read -- a file containing a mix of readable and unreadable DAGs passed the check on the readable subset alone. * The single endpoint now matches ParseImportError.filename against DagModel.relative_fileloc + DagModel.bundle_name, which is the same key the list endpoint already uses for its join. When the resolved DAG set is empty (parse failed before any DAG was defined, or the name keys did not resolve), the stacktrace is now redacted rather than returned verbatim. * The list endpoint splits the previous ``visible_files_cte`` into two CTEs: ``readable_files_cte`` enumerates the ``(relative_fileloc, bundle_name)`` pairs where the caller can read at least one DAG, and ``file_dags_cte`` enumerates the full ``(relative_fileloc, dag_id, bundle_name)`` set for those files. The per-file authorization check in the groupby loop now receives the complete DAG set for each file and can correctly detect co-located DAGs outside the caller's scope. * The same fall-through in the list endpoint (file has no matching DAGs in DagModel) now redacts the stacktrace before appending. Add a test class that exercises the fix with distinct ``fileloc`` (absolute) and ``relative_fileloc`` (relative) string values, closing the test-fixture gap where both columns previously held the same relative string and the absolute-vs-relative mismatch could not manifest. One existing single-endpoint test documenting the previous fall-through behaviour is updated to assert the new redaction. (cherry picked from commit eba9b65) Generated-by: Claude Opus 4.6 (1M context) following the guidelines at https: //github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…se log folder (#65325) (#65345) * Refuse to follow log symlinks that resolve outside the base log folder FileTaskHandler._read_from_local used to open every file that matched the task's log glob pattern, including symlinks whose real path was outside the configured base_log_folder. On deployments where worker logs are accessible from the api-server, that meant the log viewer could end up streaming content from files outside the configured log tree whenever a symlink in the task log directory happened to match the glob pattern. Canonicalise self.local_base once via os.path.realpath and, for every glob hit, resolve the path with os.path.realpath and skip it if the resolved form is not contained in the canonicalised base log folder (using os.path.commonpath, with a ValueError fallback for the different-drive case on Windows). Open the resolved path rather than the original glob hit so the file we open is the one we just validated. Append to sources only after a successful open so sources and log_streams stay aligned. Drop the @staticmethod decorator so the method can read self.local_base; existing call sites already invoke it via self. Add a test class covering: regular-file-inside-base is still streamed; a symlink whose real path is outside base_log_folder is skipped; a symlink that stays inside base_log_folder is followed (legitimate rotation case); and base_log_folder itself being a symlink works. Generated-by: Claude Opus 4.6 (1M context) following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions * Fix test__read_from_local to use valid base_log_folder The existing test passed an empty string as base_log_folder, which after the containment check resolves to CWD via os.path.realpath(""), causing all files under tmp_path to be rejected. Use tmp_path instead. (cherry picked from commit 3eda845) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…65348) (#65363) JWTRefreshMiddleware derived the cookie Secure flag from the local api.ssl_cert config only. Deployments with TLS terminated at a reverse proxy (no local SSL cert on the Airflow process) therefore received the JWT refresh cookie without the Secure flag. Match the pattern already used by every other cookie-setting location in the codebase (auth.py, simple/routes/login.py, FAB and Keycloak login routes): treat secure as True when either the request came in over HTTPS or a local ssl_cert is configured. (cherry picked from commit 60db83f) Generated-by: Claude Opus 4.6 (1M context) following the guidelines at https: //github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…_id_pattern (#65309) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * Fix: PATCH /dags pagination bug and document wildcard dag_id_pattern (#63665) * fixed pagination bug and updated docstring to clarify dag_id_pattern wildcard usage * removed batch loop to update all dags in one shot and added additional test case * Fixed MySQL subquery issue (cherry picked from commit 9504886) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com>
…nection. (#65231) (#65368) This was discovered by running with a custom External DB manager that had some gnarly queries that ended up being locked behind this transaction. `_single_connection_pool` replaces `settings.engine` with a SingletonThreadPool engine. But `work_session` was created before that — it still holds an internal reference to the old engine object. _single_connection_pool has no way to rebind work_session. So when _get_current_revision(session=work_session) runs on line 1203 — inside the _single_connection_pool() block — it calls session.connection() which goes through the old engine, not the SingletonThreadPool. The old engine's pool was disposed and recreated empty by engine.dispose(), so it creates a brand new connection. That connection is completely outside _single_connection_pool's control. _single_connection_pool guarantees one connection on the new engine. It can't prevent work_session from creating connections on the old one. The name is a bit of a lie — it's really "single connection pool for new code that uses settings.engine", not "single connection total." (cherry picked from commit e3fea3a) (cherry picked from commit f8e0876) Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
…5167) (#65321) * [v3-2-test] Bump actions/github-script in the github-actions-updates group (#65150) (#65160) Bumps the github-actions-updates group with 1 update: [actions/github-script](https://github.com/actions/github-script). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) (cherry picked from commit e5a047c) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-updates ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [v3-2-test] Added breeze generate issue content for airflow-ctl (#65042) (#65241) * Add breeze generate issue content for airflow-ctl * add new command to doc (cherry picked from commit b24538b) Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> * [v3-2-test] Run release calendar verification on its own schedule (#65118) (#65242) * Move release calendar verification to its own scheduled workflow Run dev/verify_release_calendar.py from a dedicated daily scheduled workflow instead of as a canary job in the main CI pipeline, and notify the #release-management Slack channel when the check fails so the issue is surfaced to release managers directly. * Include wiki and calendar links in release calendar Slack alert (cherry picked from commit 048e9a1) * [v3-2-test] fix(ui): register trigger and sensor graph node types (#65167) * fix(ui): register trigger and sensor graph node types Adds missing Graph node type mappings for trigger/sensor and includes a focused unit test to prevent regressions where dependency graph rendering breaks for those node kinds. * docs(ui): add graph screenshot showing sensor and trigger nodes * chore(ui): keep PR scoped to graphTypes.ts only --------- (cherry picked from commit e0ed795) Co-authored-by: Windro.xd <88357206+windro-xdd@users.noreply.github.com> Co-authored-by: Kripa Dev <dev@kripa-car-care.local> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Justin Pakzad <114518232+justinpakzad@users.noreply.github.com> Co-authored-by: Windro.xd <88357206+windro-xdd@users.noreply.github.com> Co-authored-by: Kripa Dev <dev@kripa-car-care.local>
1 task
boring-cyborg
bot
added
area:dev-tools
backport-to-v3-2-test
vatsrahul1001
force-pushed
the
v3-2-test
branch
from
f7c41f9 to
5560e7c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(cherry picked from commit 6f0af81)
Co-authored-by: Henry Chen henryhenry0512@gmail.com