Add typing to amazon provider EMR by Swalloow · Pull Request #9969 · apache/airflow

Swalloow · 2020-07-24T12:07:47Z

related: #9708
Adding type annotations to amazon/emr.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

airflow/providers/amazon/aws/hooks/emr.py

airflow/providers/amazon/aws/operators/emr_add_steps.py

airflow/providers/amazon/aws/operators/emr_create_job_flow.py

airflow/providers/amazon/aws/operators/emr_modify_cluster.py

airflow/providers/amazon/aws/operators/emr_terminate_job_flow.py

`action_subcommnads` -> `action_subcommands` `group_subcommnands` -> `group_subcommands`

airflow/providers/amazon/aws/operators/emr_create_job_flow.py

v2 was released on 21 March

Current available version is 1.9.0

The latest available version of jinja2 is 2.11.2 Fixes in 2.11.2: https://github.com/pallets/jinja/blob/master/CHANGES.rst#version-2112

mysqlclient was pinned as the functionality of Connection object of MySQL had changed and that it dropped python2. mysqlclient has added context manager interface to Connection which closes the connection on `__exit__` in 2.0.0 and Airflow Master no longer uses Python 2

This reverts commit c438812.

`startd_ate_lte` -> `start_date_lte`

apache#9993) Sharing session with RTIF leads to idle-in-transaction timeout error when DAG serialization is enabled and task running duration exceeds the idle-in-transaction timeout setting of the database.

Unfortunately cancelling workflows does not work from the workflows executed by forks because their tokens do not allow doing that (they are read only). So we have to run a separate cron-triggered action (in the context of apache/airflow repository to cancel all duplicate workflows. This action stops all the workflows running from the same fork/branch except the last one.

…gger_rule (apache#8867) Fixes the BaseSensorOperator to make respect the trigger_rule in downstream tasks, when setting soft_fail="True".

…0163)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

Signed-off-by: Raymond Etornam <retornam@users.noreply.github.com>

* add correct signature to operators in providers package * add keyword only to operators and sensors outside provider package * remove unused type ignore

Swalloow · 2020-08-07T07:26:36Z

@mik-laj Any updates on reviews?

…he#10209) We are using newly added feature of GitHub to add manually triggered workflow to enable manually-triggered force-syncing of your fork with apache/airflow.

* Pylint checks should be way faster now Instead of running separate pylint checks for tests and main source we are running a single check now. This is possible thanks to a nice hack - we have pylint plugin that injects the right "# pylint: disable=" comment for all test files while reading the file content by astroid (just before tokenization) Thanks to that we can also separate out pylint checks to a separate job in CI - this way all pylint checks will be run in parallel to all other checks effectively halfing the time needed to get the static check feedback and potentially cancelling other jobs much faster. * fixup! Pylint checks should be way faster now

Replace fixed namespace "airflow" with variable {{ .Release.Namespace }}

) * Improves stability of reported coverage and makes it nicer With this change we only upload coverage report in the case when all tests were successuful and actually executed. This means that coverage report will not be run when the job gets canceled. Currently a lof of coverage reports gathered contain far less coverage because when static check fails or docs some test jobs could already submit their coverage - resulting in partial coverage reports. With this change we also remove comment from coverage report and replace it with (for now) informational status message published to github. If we see that it works, we can change it to a PR-failing status if coverage drops for a given PR. This way we might get our coverage monotonously increasing :).

While doing a trigger_dag from UI, DagRun gets created first and then WebServer starts creating TIs. Meanwhile, Scheduler also picks up the DagRun and starts creating the TIs, which results in IntegrityError as the Primary key constraint gets violated. This happens when a DAG has a good number of tasks. Also, changing the TIs array with a set for faster lookups for Dags with too many tasks.

mik-laj · 2020-08-07T15:35:27Z

LGTM. Can you do a rebase?

…into typing-aws-emr

Swalloow · 2020-08-08T13:00:20Z

@mik-laj During the rebase process, all previous history has been pushed to this PR.
Would it be okay or How can I fix it?

potiuk · 2020-08-08T16:27:17Z

You need to just get your commit from the top and rebase it on top of master. See the CONTRIBUTING.rst - there is a section about it in the docs and some useful links explaining what to do.

Swalloow · 2020-09-13T07:12:41Z

Replace to #10910

Add typing to amazon provider EMR

8f522c7

boring-cyborg bot added the provider:amazon AWS/Amazon - related issues label Jul 24, 2020

jayrumi and others added 2 commits July 24, 2020 14:39

Add get_blobs_list method to WasbHook (apache#9950)

0bf330b

Fix static check error

2403c3a

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/hooks/emr.py Outdated Show resolved Hide resolved

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/operators/emr_add_steps.py Outdated Show resolved Hide resolved

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/operators/emr_create_job_flow.py Outdated Show resolved Hide resolved

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/operators/emr_modify_cluster.py Outdated Show resolved Hide resolved

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/operators/emr_terminate_job_flow.py Outdated Show resolved Hide resolved

kaxil and others added 4 commits July 24, 2020 21:25

Fix OpenShift Guidelines link in IMAGES.rst (apache#9978)

6d7739b

Add Badges for Airflow Docker (apache#9979)

42543c5

Add missing types

f7d29bc

Fix typo in airflow/cli/cli_parser.py (apache#9980)

c839705

`action_subcommnads` -> `action_subcommands` `group_subcommnands` -> `group_subcommands`

mik-laj reviewed Jul 24, 2020

View reviewed changes

airflow/providers/amazon/aws/operators/emr_create_job_flow.py Outdated Show resolved Hide resolved

Swalloow and others added 16 commits July 25, 2020 06:59

Remove ignore comments

328ec01

Fix various typos in airflow/cli/commands (apache#9983)

fc03304

Remove ignore comments

1386cc5

Stop using start_date in default_args in example_dags (apache#9982)

8b10a4b

Fix testcase error

b32b8a3

Bump actions/setup-python version to 2 (apache#9984)

2719fcf

v2 was released on 21 March

Bump tableauserverclient to 0.12 (apache#9988)

47cab5f

Relax requirement to allow latest version of flask-caching (apache#9989)

7cafa59

Current available version is 1.9.0

Relax requirement to allow latest version of jinja2 (apache#9991)

76c60af

The latest available version of jinja2 is 2.11.2 Fixes in 2.11.2: https://github.com/pallets/jinja/blob/master/CHANGES.rst#version-2112

Relax requirement to allow latest version of tenacity (apache#9992)

926aa1b

Revert "Bump mysqlclient to 2.0.1 (apache#9987)" (apache#9997)

2f73974

This reverts commit c438812.

Remove unnecessary environment variable from CI workflow (apache#9998)

81b65be

Fix typo in api_connexion/openapi/v1.yaml (apache#9986)

d3d918c

`startd_ate_lte` -> `start_date_lte`

Avoid sharing session with RenderedTaskInstanceFields write and delete (

ffcd060

apache#9993) Sharing session with RTIF leads to idle-in-transaction timeout error when DAG serialization is enabled and task running duration exceeds the idle-in-transaction timeout setting of the database.

j-y-matsubara and others added 8 commits August 6, 2020 13:08

Fix BaseSensorOperator soft_fail mode to respect downstream tasks tri…

73ad5a4

…gger_rule (apache#8867) Fixes the BaseSensorOperator to make respect the trigger_rule in downstream tasks, when setting soft_fail="True".

Changes to all the constructors to remove the args argument (apache#1…

24c8e4c

…0163)

Add thredup to list of Airflow users (apache#10198)

817e1ac

Add type annotations to S3 hook module (apache#10164)

0c77ea8

Update guide for Google Cloud Secret Manager Backend (apache#10172)

eff0f03

Type annotation for Docker operator (apache#9733)

d79e722

Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

Update JS packages to latest versions (apache#9811) (apache#9921)

c920b1b

Signed-off-by: Raymond Etornam <retornam@users.noreply.github.com>

Add correct signature to all operators and sensors (apache#10205)

cdec301

* add correct signature to operators in providers package * add keyword only to operators and sensors outside provider package * remove unused type ignore

potiuk and others added 7 commits August 7, 2020 10:29

You can sync your fork master with apache/airflow master via UI (apac…

1dc8b78

…he#10209) We are using newly added feature of GitHub to add manually triggered workflow to enable manually-triggered force-syncing of your fork with apache/airflow.

Fix chart: parameterize namespace (apache#10213)

4146aba

Replace fixed namespace "airflow" with variable {{ .Release.Namespace }}

Docs: Separate page for each Secrets backend (apache#10211)

9d4b914

Add airflow connections export command (apache#9856) (apache#10081)

d2540e6

Swalloow added 8 commits August 8, 2020 01:46

Add typing to amazon provider EMR

d8dfb79

Fix static check error

60f7bc8

Add missing types

f498a41

Remove ignore comments

e1f49cf

Remove ignore comments

febb5ed

Fix testcase error

6c3f783

Fix cluster_states type to list only

31e0df1

Merge branch 'typing-aws-emr' of https://github.com/Swalloow/airflow …

a61b2fb

…into typing-aws-emr

Swalloow mentioned this pull request Sep 13, 2020

Add typing to amazon provider EMR #10910

Merged

Swalloow closed this Sep 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add typing to amazon provider EMR#9969

Add typing to amazon provider EMR#9969
Swalloow wants to merge 133 commits intoapache:masterfrom
Swalloow:typing-aws-emr

Swalloow commented Jul 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Swalloow commented Aug 7, 2020

Uh oh!

mik-laj commented Aug 7, 2020

Uh oh!

Swalloow commented Aug 8, 2020

Uh oh!

potiuk commented Aug 8, 2020

Uh oh!

Swalloow commented Sep 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

Conversation

Swalloow commented Jul 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Swalloow commented Aug 7, 2020

Uh oh!

mik-laj commented Aug 7, 2020

Uh oh!

Swalloow commented Aug 8, 2020

Uh oh!

potiuk commented Aug 8, 2020

Uh oh!

Swalloow commented Sep 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants