Backport TI updated_at column from 2.5.0#42596
Closed
luisglft wants to merge 566 commits intoapache:mainfrom
Closed
Backport TI updated_at column from 2.5.0#42596luisglft wants to merge 566 commits intoapache:mainfrom
luisglft wants to merge 566 commits intoapache:mainfrom
Conversation
The rest of the page has consistently alphabetized tables. This commit fixes three `extras` that were not alphabetized. (cherry picked from commit ef3f530)
This PR fixes inverted levels in the sections added to the "Best Practices" document in apache#21879. (cherry picked from commit 8e7b76d)
The sql_alchemy_conn option is in the database section, not the core section. Simple typo fix. (cherry picked from commit f294a26)
Fix minor typos (cherry picked from commit 31f113e)
following to this change: apache#22114 (cherry picked from commit 2113d72)
The recent changes in selective checks introduced a few problems
in non-main branches:
* constraints branch was not used by CI build in non-main branch
this was not a proble, (until it started to fail) because
other branches always work in "upgradeDepencies" case
* default branch calculation did not account for the selective-check
comment that was run before default branch was retrieved. Therefore
we did not run build as from 2.3 branch
* Some precomits should be skiped in non-main branch
* Integration was also excluded from Provider Check
* Two more commit outputs added:
* docs-filter - depends whether you are with more packages
* skip-pre-commit - allows to dynamically choose which pre-commits
should be skipped
(cherry picked from commit c8d3850)
…pache#24780) When pushing cache we cannot use default builder because it has no capability to push cache. We need to use airflow_cache for it, which has additionally the capacity of forwarding the builds to a remote ARM instance in case cache is built for ARM images too. Recently we added a new --builder flag to allow to choose the builder, but our scripts and CI have not been modified to account for that (previously builder was chosen automatically but that has proven to be limiting in some manual operations and it also allows to choose different names for builders in case someone wants to build their own pipeline of builds. (cherry picked from commit a1679be)
(cherry picked from commit 056b8cf)
…nns (apache#24735) (cherry picked from commit 54d5972)
(cherry picked from commit ca361ee)
(cherry picked from commit 46ac083)
There were errors with retieving constraints branch caused by using different convention for output names (sometimes dash, sometimes camelCase as suggested by most GitHub documents). The "dash-name" looks much better and is far more readable so we shoud unify all internal outputs to follow it. During that rename some old, unused outputs were removed, also it turned out that the new selective-check can replace previous "dynamic outputs" written in Bash as well. Additionally, the "defaults" are now retrieved via Python script, not bash script which will make it much more readable - both build_images and ci.yaml use it in the right place - before replacing the scripts and dev with the version coming in from PR in case of build_images.yaml. (cherry picked from commit 017507be1e1dbf39abcc94a44fab8869037893ea)
There were errors with retieving constraints branch caused by using different convention for output names (sometimes dash, sometimes camelCase as suggested by most GitHub documents). The "dash-name" looks much better and is far more readable so we shoud unify all internal outputs to follow it. During that rename some old, unused outputs were removed, also it turned out that the new selective-check can replace previous "dynamic outputs" written in Bash as well. Additionally, the "defaults" are now retrieved via Python script, not bash script which will make it much more readable - both build_images and ci.yaml use it in the right place - before replacing the scripts and dev with the version coming in from PR in case of build_images.yaml. (cherry picked from commit 017507be1e1dbf39abcc94a44fab8869037893ea)
Since we are now building ARM images in parallel, We need more powerful machines and we implemented in-memory docker, similarly as in our AMD instances. The m6g.2xlarge are quite a bit better than c6g.xlarge for our case: 1) They have 8 vCPUs 2) They have 32 GB memory (should be enough to build 4 ARM CI images 3) Thye are Memory-optimised, and since docker is build in memory the memory speed is the most important factor This also allows to switch building all images (including cache) in parallel - so that we can have 1 job instead of 4 - similarly as we have in case of regular AMD builds. Another advantage of it is that we loose far less time of the AMD instance which "triggers" the docker build, because this instance will control 4 parallel builds at a time effectively, which will decrease a lot of overhead connected with running the instance mostly idle during the build (and since the builds will be generally faster, the overhead will be even smaller). (cherry picked from commit 6621f68)
…pache#24818) (cherry picked from commit 3157858)
(cherry picked from commit e520ef5)
There was a problem with custom classes for SQLAlchemy that prevented it to work on MySQL. This PR removes the SQLAlchemy upper binding. This has been added as an issue in: sqlalchemy/sqlalchemy#7660 But apparently it's been fixed in one of the more recent SQLAlchemy releases. (cherry picked from commit 0853d60)
(cherry picked from commit d969473)
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Tzu-ping Chung <tp@astronomer.io> (cherry picked from commit 1507ca4)
…ailed_state (apache#23647) * Respecting soft_fail in ExternalTaskSensor when the upstream tasks are in the failed state (apache#19754) - Changed behaviour of sensor to as above to respect soft_fail - Added tests of new soft_fail behaviour (apache#19754) - Added newsfragment and improved sensor docstring (cherry picked from commit 1b34598)
(cherry picked from commit 2303750)
Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com> (cherry picked from commit 5877f45)
(cherry picked from commit 1dccaad)
Fixes: apache#23639 (cherry picked from commit d86ae09)
(cherry picked from commit 603c555)
(cherry picked from commit 7e56bf6)
* Fix bugs in URI constructor for MySQL connection * Update unit tests (cherry picked from commit ea54faf)
The DebugExecutor previously executed tasks by calling the "private" ti._run_raw_task(...) method instead of ti.run(...). But the latter contains the logic to increase task instance try_numbers when running, thus tasks executed with the DebugExecutor were never getting their try_numbers increased and for rescheduled tasks this led to off-by-one errors (as the logic to reduce the try_number for the reschedule was still working while the increase was not). (cherry picked from commit da7b22b)
catch all exceptions for inserting into cross_dag_comm table
Currently even when there is an error when doing the request to` /get_logs_with_metadata` endpoint, **the UI end does not catch it**. As a result, the spinning wheel remains, leaving the user waiting in the hope that the logs will load eventually. <img width="572" alt="Screenshot 2024-09-27 at 10 55 29 a m" src="https://github.com/user-attachments/assets/f6eb5ae2-f6f5-4944-a831-bd09a19f17a6"> This PR catches that error and shows the user the alternative to use kibana to see the logs together with the link pointing to that specific log. Before (spinning forever...): <img width="920" alt="Screenshot 2024-09-27 at 10 34 10 a m" src="https://github.com/user-attachments/assets/9c42466d-30a0-485b-b89d-e3a12ce9b9a4"> After (once the endpoint failed): <img width="923" alt="Screenshot 2024-09-27 at 10 29 50 a m" src="https://github.com/user-attachments/assets/62fe1997-0aca-4115-b0e5-595fd55f02a4">
luisglft
requested review from
XD-DENG,
ashb,
bbovenzi,
bolkedebruin,
dstandish,
eladkal,
ephraimbuddy,
hussein-awala,
jedcunningham,
jscheffl,
kaxil,
o-nikolas,
pierrejeambrun,
potiuk,
ryanahamilton,
uranusjr and
vincbeck
as code owners
boring-cyborg
bot
added
area:dev-tools
area:production-image
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
Contributor
|
Uuups, something went totally wrong in re-basing your PR? |
jscheffl
requested changes
Sep 30, 2024
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Something is terribly wrong in this PR
Member
|
Pretty sure this wasn't meant for the Apache repo @luisglft, but an (internal) fork? I'm going to close this. But please open another PR if I've interpreted this incorrectly. |
Author
yes you're right, sorry about that |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New column name will be useful to query tasks stuck in scheduled state.
#26252