Enable task run setting to be able reinitialise #19845
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When task is executed using "StandardTaskRunner._start_by_exec()" (when fork is not available) it's using old DONOT_MODIFY_HANDLERS config variable loaded during settings.py load and not the one from externally supplied --cfg-path param
TL;DR
When "tasks run" command is executed it then internally re-executed as below command using external process (non fork when it's not available).
[2021-11-16 15:28:16,818] {base_task_runner.py:144} INFO - Running: ['airflow', 'tasks', 'run', 'test_logging_dag', 'test_task', 'test_run', '--job-id', '102', '--raw', '--subdir', 'DAGS_FOLDER', '--cfg-path', '/tmp/tmpfv4cj9py', '--error-file', '/tmp/tmphjtaj5gi']
Here
'--cfg-path', '/tmp/tmpfv4cj9py'
is a copy of active config settings of parent process. When command is executed settings.py file is loaded as one of the first things but external config (/tmp/tmpfv4cj9py) is loaded later within task_command.py->def task_run.And when it does load it does not re-initialize this setting.
This causes DONOT_MODIFY_HANDLERS to be initialized to a default value rather than one that is defined in the /tmp/tmpfv4cj9py
The PR fixes it by moving initialization into settings.py/configure_vars function where it can be called when needed.
PS:
I think this issue is not limited to this variable but all other similar GLOBAL variables that is initialized during settings.py load.
But I think we need to check each of them separately before making similar changes.
#19755 linked to this which I closed in favor this.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.