Parse recently modified files even if just parsed #16075
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaxil
force-pushed
the
mtime-optimization
branch
2 times, most recently
from
4edcd5b
to
d727034
Compare
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of.
jedcunningham
approved these changes
May 25, 2021
ephraimbuddy
approved these changes
May 26, 2021
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jun 2, 2021
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of. (cherry picked from commit add7490)
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jun 2, 2021
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of. (cherry picked from commit add7490) (cherry picked from commit 19b3f1b)
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jun 2, 2021
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of. (cherry picked from commit add7490) (cherry picked from commit 19b3f1b) (cherry picked from commit cb21b0a)
jhtimmins
pushed a commit
to astronomer/airflow
that referenced
this pull request
Jun 3, 2021
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of. (cherry picked from commit add7490)
ashb
pushed a commit
that referenced
this pull request
Jun 22, 2021
This commit adds an optimization where the recently modified files (detected by mtime) will be parsed even though it has not reached `min_file_process_interval`. This way you can increase `[scheduler] min_file_process_interval` to a higher value like `600` or so when you have large number of files to avoid unnecessary reparsing if files haven't changed, while still making sure that modified files are taken care of. (cherry picked from commit add7490)
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 10, 2021
This feature was added in apache#16075. This PR adds it to docs to avoid situations like apache#17437 closes apache#17437
kaxil
added a commit
that referenced
this pull request
Aug 10, 2021
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Mar 10, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jun 4, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jul 10, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Aug 27, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 4, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
aglipska
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 7, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Dec 7, 2022
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
leahecole
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jan 27, 2023
This feature was added in apache/airflow#16075. This PR adds it to docs to avoid situations like apache/airflow#17437 closes apache/airflow#17437 GitOrigin-RevId: 7dfc52068c75b01a309bf07be3696ad1f7f9b9e2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds an optimization where the recently modified files
(detected by mtime) will be parsed even though it has not reached
min_file_process_interval.This way you can increase
[scheduler] min_file_process_intervaltoa higher value like
600or so when you have large number of files toavoid unnecessary reparsing if files haven't changed, while still making
sure that modified files are taken care of.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.