Sync v2-3-stable with v2-3-test to release 2.3.1#23820
Merged
ephraimbuddy merged 136 commits intov2-3-stablefrom May 21, 2022
Merged
Sync v2-3-stable with v2-3-test to release 2.3.1#23820ephraimbuddy merged 136 commits intov2-3-stablefrom
v2-3-stable with v2-3-test to release 2.3.1#23820ephraimbuddy merged 136 commits intov2-3-stablefrom
Conversation
Empty image preparation failed in CI because it was impossible to build an empty image without buildkit. This change sets DOCKER_BUILDKIT variable for empty image build which make it always use the buildkit. (cherry picked from commit 4f6fe72)
…3334) The "upgrade-to-newer-dependencies" in the image can take "false" "true" and <RANDOM> value (the RANDOM value is used to always invalidate docker cache). This has been carried to Breeze command line, but this was a source of major confusion as the name of the parameter suggest bool value. The change modifies the parameter to be flag, and when the flag is set the parameter is set to random during image building. That should help with recent bug when image was always rebuilt without checking if it should be rebuilt. (cherry picked from commit b6db0e9)
(cherry picked from commit 94c3203)
There was a bit of noise printed when Breeze started: * information about branch/python/image/backend used * information about actions performed (like fixing permissions) * information that docke image build is not needed * warnings about missing variables This PR marks all the messages as "info" and only prints them when --verbose flag is used and it adds default values for the variables that generated warnings. (cherry picked from commit dd7002d)
(cherry picked from commit c1528f7)
The new Breeze did not use conditionally debian version to run MsSQL docker compose. This PR fixes it (cherry picked from commit 9e11d20)
When you run exec and breeze is not running, there was a stack trace printed rather than straightforward error message. This fixes it - stacktrace is only printed now when verbose is used. If not just error message is printed. (cherry picked from commit e5ac9d8)
The regexp expression for deriving right branch from version was missing . for version numbers. (cherry picked from commit 46da268)
This PR improves handling of both entry and exit to common Breeze commands: * at entry all common commands check if rebuild of image is needed * when you exit and there is an error from shell commands, rather than printing stack trace an error message is printed (cherry picked from commit 9a0080c)
(cherry picked from commit 190e911)
(cherry picked from commit 2d10940)
When using "hybrid" executors (`CeleryKubernetesExecutor` or `LocalKubernetesExecutor`), then the `clear_not_launched_queued_tasks` mechnism in the `KubernetesExecutor` can reset the queued tasks, that were given to the other executor. `KuberneterExecutor` should limit itself to the configured queue when working in the "hybrid" mode. (cherry picked from commit ae19eab)
(cherry picked from commit 451c7cb)
(cherry picked from commit 3970ea3)
There is a bug in CPython (fixed in March 2022 but not yet released) that makes async.io handle SIGTERM improperly by using async unsafe functions and hanging the triggerer receive SIGPIPE while handling SIGTERN/SIGINT and deadlocking itself. Until the bug is handled we should rather rely on standard handling of the signals rather than adding our own signal handlers. Seems that even if our signal handler just run exit(0) - it caused a race condition that led to the hanging. More details: * https://bugs.python.org/issue39622 * python/cpython#83803 Fixes: #19260 (cherry picked from commit 6bdbed6)
* Add url params for dag_run_id and task_id * Persist other search params * simplify useSelection * delete extra params * remove API change (cherry picked from commit 4c1fcee)
(cherry picked from commit 45aadd2)
The start/end dates for a mapped instance weren't updating when a user changed their timezone. Using <Time /> fixes that (cherry picked from commit 98ec8c6)
(cherry picked from commit 1c886ea)
(cherry picked from commit 16e1170)
* Add shared test wrapper & treeData placeholder * remove console log * move testUtils to /utils * change unnamed export (cherry picked from commit 8f6b855)
(cherry picked from commit a5f9df5)
(cherry picked from commit f5f9c58)
(cherry picked from commit 59e9310)
github-actions
bot
added
the
full tests needed
(cherry picked from commit 8494fc7)
(cherry picked from commit 6f5749c)
(cherry picked from commit ce8ea66)
It turned out that deprecation of config values did not work as
intended. While deprecation worked fine when the value was specified
in configuration value it did not work when `run_as_user` was used.
In those cases the "as_dict" option was used to generate temporary
configuratin and this temporary configuration contained default value
for the new configuration value - for example it caused that
the generated temporary value contained:
```
[database]
sql_alchemy_conn=sqlite:///{AIRFLOW_HOME}/airflow.db
```
Even if the deprecated `core/sql_alchemy_conn` was set (and no
new `database/sql_alchemy_conn` was set at the same time.
This effectively rendered the old installation that did not convert
to the new "database" configuration not working for run_as_user, because
the tasks run with "run_as_user" used wrong, empty sqlite database
instaead of the one configured for Airflow.
Also during adding tests, it turned out that the mechanism was also
not working as intended before - in case `_CMD` or `_SECRET` were used
as environment variables rather than configuration. In those cases
both _CMD and _SECRET should be evaluated during as_dict() evaluation,
because the "run_as_user" might have not enough permission to run the
command or retrieve secret. The _cmd and _secret variables were only
evaluated during as_dict() when they were in the config file (note
that this only happens when include_cmd, include_env, include_secret
are set to True).
The changes implemented in this PR fix both problems:
* the _CMD and _SECRET env vars are evaluated during as_dict when the
respective include_* is set
* the defaults are only set for the values that have deprecations
in case the deprecations have no values set in either of the ways:
* in config file
* in env variable
* in _cmd (via config file or env variable)
* in _secret (via config file or env variable)
Fixes: #23679
(cherry picked from commit 888bc2e)
* Add expand/collapse all groups button to Grid * add tests * add comments * Switch to 2 icon buttons Disable buttons if all groups are expanded or collapsed * Update localStorage key (cherry picked from commit 83784d9)
* Move tree filtering inside react and add some filters * Move filters from context to utils * Fix tests for useTreeData * Fix last tests. * Add tests for useFilters * Refact to use existing SimpleStatus component * Additional fix after rebase. * Update following bbovenzi code review * Update following code review * Fix tests. * Fix page flickering issues from react-query * Fix side panel and small changes. * Use default_dag_run_display_number in the filter options * Handle timezone * Fix flaky test Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com> (cherry picked from commit 46c1c00)
* communicate via customevents * Handle open group logic in wrapper * fix tests * Make grid action buttons sticky * Add default toggle fn * fix splitting task id by '.' * fix missing dagrun ids (cherry picked from commit afdfece)
* Add UI tests for /utils and /components * add test for Table * Address PR feedback * Fix window prompt var * Fix TaskName test from rebase * fix lint errors (cherry picked from commit 694e380)
…23690) Celery can lose tasks on worker shutdown, causing airflow to just wait on them indefinitely (may be related to celery/celery#7266). This PR expands the "stalled tasks" functionality which is already in place for adopted tasks, and adds the ability to apply it to all tasks such that these lost/hung tasks can be automatically recovered and queued up again. (cherry picked from commit baae70c)
ephraimbuddy
force-pushed
the
v2-3-test
branch
2 times, most recently
from
84f03e2 to
1e3e4dc
Compare
In certain upgrade paths, Airflow isn't given an opportunity to track the old `log_id_template`, so document the fix for folks who run into trouble. (cherry picked from commit 6f738d9)
jedcunningham
approved these changes
May 20, 2022
potiuk
approved these changes
May 21, 2022
(cherry picked from commit 8f3ce33)
The build commands were missing "airflow-constraints-reference" parameter and it always defaulted to constraints-main (cherry picked from commit cc3ab43)
* only get necessary task instances * add comment * encode_ti -> get_task_summary (cherry picked from commit 7ab5ea7)
The previous fix (#23844) broke main on package verification as the package verification used the same parameter that was set to empty. This change rmeoves some remnant from the "bash" version where we had to check if variable was empty and also making the "constraint" parameters accepting default values from the current branch to be used also for build commands. (cherry picked from commit 10a0d8e)
The JPype1 limit has to be introduced because otherwise the 1.4.0 JPype1 breaks our ARM builds. The 1.4.0 did not release the sdist version of the package. This made our cache refresh job to fail as 1.4.0 version cannot be installed on ARM image. The issue is captured in jpype-project/jpype#1069 (cherry picked from commit 3699be4)
jedcunningham
approved these changes
May 21, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Time for
2.3.1rc1!