…#5738)

…oudBaseHook (#5741)

Implement GCP Cloud Tasks' Hook and Operators

#5714)

…5737)

…parser (#5617)

…ry (#5744)

)

Change SubDagOperator to use Airflow scheduler to schedule
tasks in subdags instead of backfill.

In the past, SubDagOperator relies on backfill scheduler
to schedule tasks in the subdags. Tasks in parent DAG
are scheduled via Airflow scheduler while tasks in
a subdag are scheduled via backfill, which complicates
the scheduling logic and adds difficulties to maintain
the two scheduling code path.

This PR simplifies how tasks in subdags are scheduled.
SubDagOperator is reponsible for creating a DagRun for subdag
and wait until all the tasks in the subdag finish. Airflow
scheduler picks up the DagRun created by SubDagOperator,
create andschedule the tasks accordingly.

…tom image from a different project (#5752)

#5763)"

This reverts commit 502ed74.

It implements the method proposed in AIP-24 to serialize DAG. It will be used in DAG persistency in DB to solve webserver scalability issue.

* AIRFLOW-5139 Allow custom ES configs

While attempting to create a self-signed TLS connection between airflow
and ES, we discovered that airflow does now allow users to modify the
SSL state of the elasticsearchtaskhandler. This commit will allow users
to define ES settings in the airflow.cfg

[AIRFLOW-XXX] Add iS2.co to list of airflow users

- Refactors `BaseOperator.render_template()` and removes `render_template_from_field()`. The functionality could be greatly simplified into a single `render_template()` function.
- Removes six usage.
- Improves performance by removing two `hasattr` calls and avoiding recreating Jinja environments.
- Removes the argument `attr` to `render_template()` which wasn't used.
- Squashes multiple similar tests into two parameterized tests.
- Adheres to 110 line length.
- Adds support for templating sets.
- Adds Pydoc.
- Adds typing.

…AG files (#5757)

The scheduler calls `list_py_file_paths` to find DAGs to schedule. It does so
without passing any parameters other than the directory. This means that
it *won't* discover DAGs that are missing the words "airflow" and "DAG" even
if DAG_DISCOVERY_SAFE_MODE is disabled.

Since `list_py_file_paths` will refer to the configuration if
`include_examples` is not provided, it makes sense to have the same behaviour
for `safe_mode`.

This gives the option to delete the dataset as well as its tables.

…Hook (#5783)

Use from google.api_core.gapic_v1.client_info import ClientInfo
because this object inherits from
from google.api_core.client_info import ClientInfo
and implements one additional method used by Python SDKs.

…5736)

The GCS Transfer Service REST API requires that a schedule be set, even for
one-time immediate runs. This adds code to
`S3ToGoogleCloudStorageTransferOperator` and
`GoogleCloudStorageToGoogleCloudStorageTransferOperator` to set a default
one-time immediate run schedule when no `schedule` argument is passed.

…rguments (#5727)

`GCPTransferServiceHook.wait_for_transfer_job` defeaults its `timeout`
parameter to 60 and assumes it is an integer or at least comparable to
one. This is a problem as some of the built-in operators that use it
like `S3ToGoogleCloudStorageTransferOperator` and
`GoogleCloudStorageToGoogleCloudStorageTransferOperator` default their
`timeout` param to `None`, and when they call this method with their
default value, it causes an error. Fix this by allowing
`wait_for_transfer_job` to accept a timeout of `None` and fill in
appropriate defaults. This also adds functionality to allow it to take
a `timedelta` instead of an integer, allows seconds to be any real, as
there is really no need for them to actually be an integer, and fixes
the counting of time for determining timeout to be a bit more accurate.

…5489)

* [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator)

Add support for running Docker containers via Docker Swarm
which allows the task to run on any machine (node) which
is a part of your Swarm cluster

More details: https://issues.apache.org/jira/browse/AIRFLOW-4843

Built with <3 at Agoda!

…ery (#5717)

… patch_dataset and get_dataset (#5546)

Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset

…#5747)

…ckOperator (#5816)

For large DAGs, iterating over template fields to find template files can be time intensive.
Save this time for tasks that do not specify a template file extension.

The recent commit 3724c2a to master introduced a __init__.py file in
the project root folder, which basically breaks all imports in local
development (`pip install -e .`) as it turns the project root into a
package.

[ci skip]

#5777)

…5079)

)

…roblems (#5835)

* [AIRFLOW-5233] Fixed consistency in whitespace (tabs/eols) + common problems

…5838)

…e in GoogleCloudStorageToBigQueryOperator. (#5771)

Set autodetect default value from false to be true to avoid breaking downstream
services using GoogleCloudStorageToBigQueryOperator but not aware of the newly
added autodetect field.

This is to fix the current regression introduced by #3880

…perators (#5672)

- changes the order of arguments for `has_mail_attachment`, `retrieve_mail_attachments` and `download_mail_attachments`
- add `get_conn` function
- refactor code
- fix pylint issues
- add imap_mail_filter arg to ImapAttachmentToS3Operator
- add mail_filter arg to ImapAttachmentSensor
- remove superfluous tests
- changes the order of arguments in the sensors + operators __init__

List the two separate pylint scripts for use inside the Docker
containers in CONTRIBUTING.md.

This commit moves GCP Container from contrib to core.
For more information check AIP-21.

This commit moves GCP Video Intelligence from contrib to core.
For more information check AIP-21.

This commit moves GCP Translate from contrib to core.
For more information check AIP-21.

This commit moves GCP PubSub from contrib to core.
For more information check AIP-21.

This commit moves GCP Spanner from contrib to core.
For more information check AIP-21.

This commit moves GCP DLP from contrib to core.
For more information check AIP-21.

This commit moves GCP Dataflow from contrib to core.
For more information check AIP-21.

This commit moves GCP Vision from contrib to core.
For more information check AIP-21.

This commit moves GCP Functions from contrib to core.
For more information check AIP-21.

)

This commit intends to remove files under contrib/plugins out
of the Pylint todo list.

…t credentials (#5475)

…%I) in _clean_execution_date instead of %H and %M (#5864)

This commit moves GCP Transfer from contrib to core.
For more information check AIP-21.

This was done in the top level README.me in abbb1ea in 2016.

… shell files (#5807)

* [AIRFLOW-5204] Shellcheck + common licence in shell files

…se logs (#5863)