sync fork from main #1

manesioz · 2020-07-16T17:50:31Z

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Target Github ISSUE in description if exists
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

) So far tests were not triggered when only requirements changed, but this is quite needed in fact.

closes #9640

The previously-listed person is no longer at the company

* generate go client from openapi spec * move openapi codegen to seperate workflow

…9714)

Co-authored-by: Dylan Joss <dylanjoss@gmail.com>

Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com> Co-authored-by: Vinay <vinay@synctactic.ai> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>

* Error in description after deployment Co-authored-by: Daniel Debny <daniel.debny@polidea.com>

Skip one version of Python for each test.

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com>

…`yarn kill` (#9044) do a kinit before yarn kill if keytab and principal is provided

One condition was bad and warns when the plugin is for admin and FAB flask.

Removing the suicide joke.

* Tests are working for newly added backport providers

Otherwise at large scale this can end up with some tasks failing as they try to create the result table at the same time. This was always possible before, just exceedingly rare, but in large scale performance testing where I create a lot of tasks quickly (especially in my HA testing) I hit this a few times. This is also only a problem for fresh installs/clean DBs, as once these tables exist the possible race goes away. This is the same fix from #8909, just for runtime, not test time.

Currently there is "unbound" variable error printed in CI logs because of that.

* Allow `replace` flag in gcs_to_gcs operator. If we are not replacing, list all files in the Destination GCS bucket and only keep those files which are present in Source GCS bucket and not in Destination GCS bucket

Co-authored-by: yongheng.liu <yongheng.liu@kyligence.io>

Co-authored-by: Ephraim Anierobi <4122866+ephraimbuddy@users.noreply.github.com>

Co-authored-by: Refael Y <refael@seadata.co.il>

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>

) Until Airflow 1.10.10 the default run_as_user config (https://airflow.readthedocs.io/en/1.10.10/configurations-ref.html#run-as-user) which defaulted it to root user `0` (https://github.com/apache/airflow/blob/96697180d79bfc90f6964a8e99f9dd441789177c/airflow/contrib/executors/kubernetes_executor.py#L295-L301) In Airflow 1.10.11 we changed it to `50000`

Co-authored-by: Ashwin Shankar <ashankar@slack-corp.com>

…ors (#9798)

* Change DAG.clear to take dag_run_state * fix lint * fix tests * assign var * extend original clause

…8936) Documentation for S3FileTransformOperator states that users can skip transformation script if S3 Select experession is specified, but in this case the created file is always zero bytes long. This fix changes the behaviour, so in case of no transformation given, the source file (a result of S3Select) is uploaded.

fixes #9852

…9816) Rather than only allowing specific pre-determined config settings, this change allows the user to place _any_ config setting they like in the generated airflow.cfg, including overwriting the "generated defaults". This providers a nicer interface for the users of the chart (even if the could already set these via the env vars).

Co-authored-by: michalslowikowski00 <michal.slowikowski@polidea.com>

So that it can pull the specified image from a private registry.

Reviewed the scripts and removed some of the old unused ones.

potiuk and others added 30 commits July 6, 2020 20:41

Tests should also be triggered when there is just setup.py change (#9690

72abf82

) So far tests were not triggered when only requirements changed, but this is quite needed in fact.

Update FlaskAppBuilder to v3 (#9648)

e764ea5

Some Pylint fixes in airflow/models/taskinstance.py (#9674)

631ac48

Update migrations to ensure compatibility with Airflow 1.10.* (#9660)

eb403be

closes #9640

Fix _process_executor_events method to use in-memory try_number (#9692)

4ad3bb5

use the correct claim name in the webserver (#9688)

502b6ea

Update Thumbtack points of contact in Airflow Users list (#9701)

397b33f

The previously-listed person is no longer at the company

generate go client from openapi spec (#9502)

6c15885

* generate go client from openapi spec * move openapi codegen to seperate workflow

[AIRFLOW-XXXX] Remove unnecessary docstring in AWSAthenaOperator

ecce1ac

Add health API endpoint (#8144) (#9277)

c713d92

Add AWS StepFunctions integrations to the aws provider (#8749)

564192c

Move gcs & wasb task handlers to their respective provider packages (#…

23f80f3

…9714)

Allow AWSAthenaHook to get more than 1000/first page of results (#6075)

07b8102

Co-authored-by: Dylan Joss <dylanjoss@gmail.com>

Add Dag Runs CRUD endpoints (#9473)

7a4988a

Make airflow/migrations/env.py Pylint Compatible (#9670)

c353fed

Get Airflow configs with sensitive data from Secret Backends (#9645)

2f31b30

YAML file supports extra json parameters (#9549)

dfe8337

Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com> Co-authored-by: Vinay <vinay@synctactic.ai> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>

fix grammar in prereq tasks gcp operator docs (#9728)

38a0b1d

Add The Climate Corporation to user list (#9726)

b26017d

Add Qingping Hou to committers list (#9725)

576100b

Add new fantastic team member of Polidea. (#9724)

47c9f75

Error in description after deployment (#9723)

9db1fa3

* Error in description after deployment Co-authored-by: Daniel Debny <daniel.debny@polidea.com>

Skip one version of Python for each test.

0aea648

Skip one version of Python for each test.

Add read-only endpoints for DAG Model (#9045)

8b94ace

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com>

Ensure Kerberos token is valid in SparkSubmitOperator before running …

13a827d

…`yarn kill` (#9044) do a kinit before yarn kill if keytab and principal is provided

Update example DAG for AI Platform operators (#9727)

b230566

Fix warning about incompatible plugins (#9704)

8517af6

One condition was bad and warns when the plugin is for admin and FAB flask.

Update local_task_job.py (#9746)

66c8af1

Removing the suicide joke.

Tests are working for newly added backport providers (#9739)

070e3c3

* Tests are working for newly added backport providers

potiuk and others added 29 commits July 14, 2020 16:35

For now cloud tools are not needed in CI (#9818)

69f82e6

Currently there is "unbound" variable error printed in CI logs because of that.

Python 3.8.4 release breaks our builds (#9820)

4636fc6

Allow replace flag in gcs_to_gcs operator. (#9667)

ed5004c

* Allow `replace` flag in gcs_to_gcs operator. If we are not replacing, list all files in the Destination GCS bucket and only keep those files which are present in Source GCS bucket and not in Destination GCS bucket

Add kylin operator (#9149)

a2c5389

Co-authored-by: yongheng.liu <yongheng.liu@kyligence.io>

Fix SqlAlchemy-Flask failure with python 3.8.4 (#9821)

5805a36

Add API Reference docs (redoc) to sphinx (#9806)

4a547ee

Add Google Deployment Manager Hook (#9159)

9f01795

Co-authored-by: Ephraim Anierobi <4122866+ephraimbuddy@users.noreply.github.com>

Remove HTTP guide index in docs (#9796)

a0493bd

Improve type hinting to provider cloudant (#9825)

35fe972

Co-authored-by: Refael Y <refael@seadata.co.il>

Add option to delete by prefix to S3DeleteObjectsOperator (#9350)

52b6efe

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>

Add CloudVisionDeleteReferenceImageOperator (#9698)

2d8dbac

Improve typing in airflow/models/pool.py (#9835)

cccc76a

Remove global variable with API auth backend (#9833)

7b23d4d

Fix Writing Serialized Dags to DB (#9836)

2d12441

Update gcp to google in docs (#9839)

4aca72e

Co-authored-by: Ashwin Shankar <ashankar@slack-corp.com>

BigQueryTableExistenceSensor needs to specify keyword arguments (#9832)

770de53

Add guide for AI Platform (previously Machine Learning Engine) Operat…

6d65c15

…ors (#9798)

Change DAG.clear to take dag_run_state (#9824)

b01d95e

* Change DAG.clear to take dag_run_state * fix lint * fix tests * assign var * extend original clause

Rename DagBag.store_serialized_dags to Dagbag.read_dags_from_db (#9838)

d008ff6

Update more occurrences of gcp to google (#9842)

1e24051

Add Dynata to the Airflow users list (#9846)

892960d

Fix DagRun.conf when using trigger_dag API (#9853)

31cab8f

fixes #9852

Fix typo in datafusion operator (#9859)

eb6f1d1

Co-authored-by: michalslowikowski00 <michal.slowikowski@polidea.com>

Fix Experimental API Client (#9849)

f4067b6

Add imagePullSecrets to the create user job (#9802)

473f506

So that it can pull the specified image from a private registry.

Group CI scripts in subdirectories (#9653)

faec41e

Reviewed the scripts and removed some of the old unused ones.

manesioz merged commit 3641615 into manesioz:master Jul 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sync fork from main #1

sync fork from main #1

manesioz commented Jul 16, 2020

sync fork from main #1

sync fork from main #1

Conversation

manesioz commented Jul 16, 2020