Skip to content

Fix/awsbatchoperator#55657

Closed
yash1thsa wants to merge 829 commits intoapache:mainfrom
yash1thsa:fix/awsbatchoperator
Closed

Fix/awsbatchoperator#55657
yash1thsa wants to merge 829 commits intoapache:mainfrom
yash1thsa:fix/awsbatchoperator

Conversation

@yash1thsa
Copy link
Contributor


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

kaxil and others added 30 commits July 4, 2025 18:01
…pache#52875) (apache#52878)

When we "hard-coded" the link we had to hard-code the version and
it did not work for staging (because we had link to prod site)

With this PR:

* no more hard-coding
* works on staging
(cherry picked from commit 1d1b1d4)
) (apache#52896)

The task-sdk package did not have full integration with airflow
docs:

* did not have inventory mapping
* did not have a way to refer to it via doc:`task-sdk:` directive
8 did not have a development link from the devel index

This PR fixes all that.

(cherry picked from commit 244af72)
…dagrun is fetched. (apache#52918) (apache#52919)

(cherry picked from commit 671d561)

Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
…build deps (apache#52909) (apache#52924)

I noticed that we did not have `pre-commit-uv` in the image
that was building the distribution packages. Adding it should
slightly speed up building such an image for isolated package
preparation.

The `uv` has been upgraded along the way, also the comment that
we should automate updating the versions in
release_management_command.py has been removed - because we
already have automation in place. The missing pieces were
adding "hatch", "pyyaml", "gitpython" and "rich".
(cherry picked from commit bbc8f74)
…distributions (apache#52908) (apache#52910)

The file might be a left-over from earlier api generation and it is
already .gitignored, so it does not appear in the source tarball,
but sdist and wheel generation does not look at .gitignored files
(for a good reason - we might want to include some generated files
in those distributions) - so we should exclude it manually similarly
as we already do with node_modules for example.
(cherry picked from commit 93f2bcc)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…pache#52937)

(cherry picked from commit 601a06d)

Co-authored-by: GPK <gopidesupavan@gmail.com>
…y imports (apache#52956) (apache#52957)

(cherry picked from commit aecfc53)

Co-authored-by: GPK <gopidesupavan@gmail.com>
(cherry picked from commit 64d87e5)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
…apache#52988) (apache#53007)

(cherry picked from commit a30bd70)

Co-authored-by: Jun Jeong <121210928+givemechocopy@users.noreply.github.com>
…3005)

* fix rollback from airflow3

* use fab for test

* amend

* fix test

* revert condition fix and add shortcut if revisions are same

* updating revision to 2.10.3 in condition

* remove fab from tests
(cherry picked from commit d458554)

Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
…apache#52297) (apache#53011)

* docs: Update public interface documentation for Airflow 3.0+ for metadata direct access change

* docs: replace direct metadata model imports in how-to examples with airflow.sdk

* Make Public interface for Airflow 3 and add link for Airflow 2.11

* Fix PR comments

* Update airflow-core/docs/public-airflow-interface.rst



* Fix PR comments

* Remove duplicates and remove RuntimeTaskInstanceProtocol as it is not public

* Fix PR comments

---------


(cherry picked from commit e8767b6)

Co-authored-by: Ankit Chaurasia <8670962+sunank200@users.noreply.github.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
* [v3-0-test] Fix log for skipped taks (apache#53024)
(cherry picked from commit 0d6e417)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* Fix CI

---------

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
…pache#53081) (apache#53086)

(cherry picked from commit 3eaf4e9)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
apache#53092)

* [v3-0-test] Fix no_status and duration for grid summaries apache#53087

* Fix CI
…e#53101)

(cherry picked from commit b052f84)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
…all` (apache#53020) (apache#53102)

(cherry picked from commit bdc9cd1)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
…3193)

(cherry picked from commit 6eca233)

Co-authored-by: GPK <gopidesupavan@gmail.com>
…pache#53186) (apache#53194)

If you tried to send a large XCom value, it would fail in the task/child
process side with this error:

> RuntimeError: unable to read full response in child. (We read 36476, but expected 1310046)

(The exact number that was able to read dependent on any different factors,
like the OS, the current state of the socket and other things. Sometimes it
would read up to 256kb fine, othertimes only 35kb as here)

This is because the kernel level read-side socket buffer is full, so that was
as much as the Supervisor could send. The fix is to read in a loop until we
get it all.
(cherry picked from commit b9620bf)

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
…ache#50371) (apache#52698)

* Add back dag parsing pre-import optimization (apache#50371)

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

* Apply suggestions from code review

---------

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…pache#52902) (apache#52925)

(cherry picked from commit 1996171)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
…mmit's (apache#52931) (apache#52942)

We are still waiting for release of the Lucas's pre-commit after
the Lucas-C/pre-commit-hooks#103 has
been merged - and for now we need to use `bleeding-edge` for
the repo.

Also upgrades zizmor and solves using "env." in shell commands
that might lead to security issues.
(cherry picked from commit 535d71d)
…lient to re matches (apache#52960) (apache#52961)

(cherry picked from commit 8e5c284)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
Follow up after apache#52967 -> from later discussions it turned out that
it's not really the ~= that is wrong and ambiguous, but that just
upper-binding of Python version is generally considered as a bad
idea - and it's not Astral's view but it's general consensus that
upper-binding of "python-requires" is bad. Since ~= implies
upper-binding, simply replacing it with >= is likely the best option
we can choose.

(cherry picked from commit e9eb481)
) (apache#52987)

Follow up after apache#52980 - there are still few more places where
the ~= was used in requires-python.
(cherry picked from commit 3f6f1db)
@boring-cyborg
Copy link

boring-cyborg bot commented Sep 15, 2025

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@yash1thsa
Copy link
Contributor Author

Wrong commit. Please ignore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.