FF #1
Merged
FF #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement GCP Cloud Tasks' Hook and Operators
) Change SubDagOperator to use Airflow scheduler to schedule tasks in subdags instead of backfill. In the past, SubDagOperator relies on backfill scheduler to schedule tasks in the subdags. Tasks in parent DAG are scheduled via Airflow scheduler while tasks in a subdag are scheduled via backfill, which complicates the scheduling logic and adds difficulties to maintain the two scheduling code path. This PR simplifies how tasks in subdags are scheduled. SubDagOperator is reponsible for creating a DagRun for subdag and wait until all the tasks in the subdag finish. Airflow scheduler picks up the DagRun created by SubDagOperator, create andschedule the tasks accordingly.
…tom image from a different project (#5752)
It implements the method proposed in AIP-24 to serialize DAG. It will be used in DAG persistency in DB to solve webserver scalability issue.
* AIRFLOW-5139 Allow custom ES configs While attempting to create a self-signed TLS connection between airflow and ES, we discovered that airflow does now allow users to modify the SSL state of the elasticsearchtaskhandler. This commit will allow users to define ES settings in the airflow.cfg
[AIRFLOW-XXX] Add iS2.co to list of airflow users
- Refactors `BaseOperator.render_template()` and removes `render_template_from_field()`. The functionality could be greatly simplified into a single `render_template()` function. - Removes six usage. - Improves performance by removing two `hasattr` calls and avoiding recreating Jinja environments. - Removes the argument `attr` to `render_template()` which wasn't used. - Squashes multiple similar tests into two parameterized tests. - Adheres to 110 line length. - Adds support for templating sets. - Adds Pydoc. - Adds typing.
* [AIRLFOW-4851] refactor Airflow kubernetes * [AIRFLOW-4851] refactor Airflow k8s models * [AIRFLOW-4851] Fix linting and tests * Refactor and add some tests * [AIRLFOW-4851] Add assertions to PodOperator tests
The fuzzy licence matching implemented by Jarek Potiuk was accepted and merged by Lucas-C in his pre-commit hooks implementation (released today ver. 1.1.7) so we can switch back to it.
- change webserver link to RST syntax - link resulted in a `about:blank#blocked` .
…6001) * [AIRFLOW-XXX] Make Breeze The default integration test environment Co-Authored-By: Chao-Han Tsai <milton0825@gmail.com>
* [AIRFLOW-5358] Improved mocking of the project ID
Without **kwargs push_by_returning was giving error, so added that parameter.
* [AIRFLOW-5319] Add system tests for Dataproc
This commit adds additional test related to problem mentioned in initial issue https://issues.apache.org/jira/browse/AIRFLOW-3804\#
When a user supplied a filename the expected behaviour is that airflow downloads the file and does not return it's content as a string.
…5671) * AIRFLOW-5049 Add validation for src_fmt_configs in bigquery hook Adds validation for the src_fmt_configs arguments in the bigquery hook. Otherwise wrong src_fmt_configs would be silently ignored which is non-desireable. * [AIRFLOW-5049] Update - Add validation for src_fmt_configs in bigquery hook Adds a common method for validating the src_ftm_configs
SQLalchemy supports connection check while returning it from the pool. There is a need to allow this parameter (`pool_pre_ping`) while creating the connection to database. More info here: https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
* [AIRFLOW-5412] Add get_conn/get_client to hooks tests
[AIRFLOW-XXX] Fix incorrect GCP integration sections
* [AIRFLOW-4758] Add GcsToGDriveOperator operator
* [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.