Fetching databricks host from connection if not supplied in extras. by joshi95 · Pull Request #10749 · apache/airflow

joshi95 · 2020-09-05T09:25:38Z

Fix for the issue #10726 . Have supported the previous functionality of fetching the host from extras if specified else it is getting fetched from the databricks connection which makes more sense.

mik-laj

Can you add tests to avoid regression?

joshi95 · 2020-09-05T09:38:20Z

Can you add tests to avoid regression?

Sure will add that. :)

joshi95 · 2020-09-05T10:26:29Z

Can you add tests to avoid regression?

Sure will add that. :)

Hi I have added the tests. Not sure why CI/CD test were cancelled.

potiuk · 2020-09-05T10:30:29Z

Caused by the transient issue when building image in the "BuildImage" run: https://github.com/apache/airflow/runs/1075245858?check_suite_focus=true

potiuk · 2020-09-05T10:31:09Z

Re-runed both.

joshi95 · 2020-09-05T11:49:17Z

Re-runed both.

Thank you :)

potiuk · 2020-09-05T20:36:12Z

Ah no - you nee to rebase to latest master. The build of yours is based on rather old version of the Docker file that uses removed requirements, so you need rebase and it should work fine.

Google Cloud APIs introduced breaking changes in 2.0.0 (https://github.com/googleapis/python-container/blob/master/UPGRADING.md) and they already caused a number of changes. We should (for now - before we migrate to 2.0+ ) limit all our google-cloud deps to <2.0.0 Fixes #10316

The pytest bug in 6.0.0 has been fixed in 6.0.1. See changelog for details: https://docs.pytest.org/en/stable/changelog.html#pytest-6-0-1-2020-07-30

We run this on Webserver Startup and when DAG Serialization is enabled we expect that no files are required but because of this bug the files were still looked for.

… Manager. (#10326)

Co-authored-by: Jacob Ferriero <jferriero@google.com>

CI is failing because of incorrect spelling "everytime", it should be "every time"

Co-authored-by: Alikhan <alikhan.tagybergen@tomtom.com> Co-authored-by: alikhtag <43503284+alikhtag@users.noreply.github.com>

In case Apache Airflow directory is added as subrepo, a new .gitrepo file is created in the Airflow sources. When you try to run pre-commit checks, the RAT check fails in this case. Adding it to .rat-excludes fixes the problem

* More fancy environment checking * fixup! More fancy environment checking

Add Bigtable Update Instance Hook/Operator

Add type annotations, including a few changes to ensure the right types are passed through. Specifically, if region is not given, it must be provided in the DAG's default_args.

This change fixes error: open(quickstart.sh): Permission denied that was rised during git add.

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

The `@provide_session` wrapper will already commit the transaction when returned, unless an explicit session is passed in -- removing this parameter changes the behaviour to be: - If session explicitly passed in: don't commit (caller's responsibility) - If no session passed in, `@provide_session` will commit for us already.

If we run this test (TestTriggerRuleDep::test_get_states_count_upstream_ti specifically) more than once without clearing the DB in between it would fail due to a unique constraint violation.

It could have wedged, (but the process still be alive) and we would never notice. In this I use `time.monotonic` rather than a `datetime` object for two reasons: 1. We don't need the expense of a "full" date time object since all we care about is the second diff between two points in time. 2. It is "more correct" as `datetime.now()` would be inaccurate if the system clock changes (NTP etc.)

* Unify command names in CLI * fixup! Unify command names in CLI

@mik-laj

…ploying pods (#10677) * Add generate_template command for kubernetes_executor * move import * fix test failure * Address @mik-laj comments * Address @mik-laj comments * Use current dir * add docs * fix test

The current latest available version is 0.5.0 (https://pypi.org/project/Flask-Login/0.5.0/) Version 0.5.0 drops support for Python 2.6, 3.3, 3.4 and we don't use those versions in Airflow Master

* Make grace_period_seconds option on K8sPodOperator This PR allows users to choose whether they want to gracefully kill pods when they delete tasks in the UI or if they would like to immediately kill them. * Update airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

BATS has additional libraries of asserts that are much more straightforward and nicer to write tests for bash scripts There is no dockerfile from BATS that contains those, so we had to build our own (but it follows the same structure as #9652 - where we keep our dev docker image sources inside our repository and the generated docker images in "apache/airflow:<tool>-CALVER-TOOLVER format. We have more BATS unit test to add - following #10576 and this change will be of great help.

Co-authored-by: antonio-davide-cali <antonio.davide.cali@ikea.com>

openapi-generator relies on this component to generate auth code in some of the clients.

Enable D106, D207 and D208 D106 Missing docstring in public nested class D207 Docstring is under-indented D208 Docstring is over-indented

Adds test that an error is raised with specific message when unkown object type is passed

Adds test to verify that string can be passed to conf and ConfObject._deserialize works.

Add black to the table for Static Code Checkers

If the pod restarts before the sleep time is over, the trim command will not run. I think it's better if we reorder the commands to execute the delete and then go to sleep. At the moment sleep is every 15 mins but people will just increase the EVERY line if they want longer sleep time and can encounter this bug.

Part of #10576

joshi95 · 2020-09-06T04:11:56Z

I will start a new PR closing this.

joshi95 changed the title ~~Fetching databricks host from connection if not supplied in extras. #10726~~ Fetching databricks host from connection if not supplied in extras. (#10726) Sep 5, 2020

joshi95 changed the title ~~Fetching databricks host from connection if not supplied in extras. (#10726)~~ Fetching databricks host from connection if not supplied in extras. #10726 Sep 5, 2020

joshi95 changed the title ~~Fetching databricks host from connection if not supplied in extras. #10726~~ Fetching databricks host from connection if not supplied in extras. Sep 5, 2020

mik-laj reviewed Sep 5, 2020

View reviewed changes

ryanahamilton and others added 20 commits September 6, 2020 09:38

Use more human readable table heading labels on DAG details (#10305)

a96833f

Restrict google-cloud-dataproc to <2.0.0 (#10307)

2440eeb

Implement Google BigQuery Table Partition Sensor (#10218)

ca099c2

Unpin pytest (#10314)

1030862

The pytest bug in 6.0.0 has been fixed in 6.0.1. See changelog for details: https://docs.pytest.org/en/stable/changelog.html#pytest-6-0-1-2020-07-30

Added "sharded" word to spellchecker (#10320)

a4778b4

Respect DAG Serialization setting when running sync_perm (#10321)

f2f55ec

We run this on Webserver Startup and when DAG Serialization is enabled we expect that no files are required but because of this bug the files were still looked for.

Catch Permission Denied exception when getting secret from GCP Secret…

78f8544

… Manager. (#10326)

Fix clear future recursive when ExternalTaskMarker is used (#9515)

0ab8f0a

Webserver: Sanitize values passed to origin param (#10334)

4ce0a57

Improve idempotency of BigQueryInsertJobOperator (#9590)

d11df06

Co-authored-by: Jacob Ferriero <jferriero@google.com>

CI: Fix failing docs-build (#10342)

b4bf0ba

CI is failing because of incorrect spelling "everytime", it should be "every time"

Add ingress to the helm chart (#10064)

bf0027b

Co-authored-by: Alikhan <alikhan.tagybergen@tomtom.com> Co-authored-by: alikhtag <43503284+alikhtag@users.noreply.github.com>

Breeze was slightly too chatty when there was no dirs created (#10346)

a2d7029

More informative description of Breeze's --verbose flag. (#10348)

1c5845d

Breeze: More fancy environment checking (#10329)

9dd1cf3

* More fancy environment checking * fixup! More fancy environment checking

Improve language of a BaseSensorOperator in UPDATING.md (#10332)

c557521

Add Bigtable Update Instance Hook/Operator (#10340)

d1b17fc

Add Bigtable Update Instance Hook/Operator

Add type annotations for mlengine_operator_utils (#10297)

c866c8e

Add type annotations, including a few changes to ensure the right types are passed through. Specifically, if region is not given, it must be provided in the DAG's default_args.

turbaszek and others added 28 commits September 6, 2020 09:39

Make Cloud Build system tests setup runnable (#10692)

4328538

This change fixes error: open(quickstart.sh): Permission denied that was rised during git add.

[AIRFLOW-5948] Replace SimpleDag with SerializedDag (#7694)

73e047e

Update DAG Serialization docs (#10711)

6e2e9db

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

Migrate speccy to spectral in OpenAPI linting. (#10351)

dff0f5d

Make test_trigger_rule_dep tests re-runnable (#10712)

2e86325

If we run this test (TestTriggerRuleDep::test_get_states_count_upstream_ti specifically) more than once without clearing the DB in between it would fail due to a unique constraint violation.

Add Stacktrace when DagFileProcessorManager gets killed (#10681)

fad9ef4

Add Indeed to INTHEWILD.md (#10716)

e210b93

Unify command names in CLI (#10720)

e5a2dcf

* Unify command names in CLI * fixup! Unify command names in CLI

Add generate_yaml command to easily test KubernetesExecutor before de…

dda12cc

…ploying pods (#10677) * Add generate_template command for kubernetes_executor * move import * fix test failure * Address @mik-laj comments * Address @mik-laj comments * Use current dir * add docs * fix test

Relax requirement to allow latest version of flask-login (#9990)

f7ef1cd

The current latest available version is 0.5.0 (https://pypi.org/project/Flask-Login/0.5.0/) Version 0.5.0 drops support for Python 2.6, 3.3, 3.4 and we don't use those versions in Airflow Master

Add masterConfig parameter to MLEngineStartTrainingJobOperator (#10578)

7ccb80e

Co-authored-by: antonio-davide-cali <antonio.davide.cali@ikea.com>

Add securitySchemes in openapi spec (#10652)

5757344

openapi-generator relies on this component to generate auth code in some of the clients.

Fix docs for generate-dag-yaml cli command (#10735)

be578ec

Enable more checks for pydocstyle (#10741)

f81a06d

Enable D106, D207 and D208 D106 Missing docstring in public nested class D207 Docstring is under-indented D208 Docstring is over-indented

Update node installation cmd (#10744)

0ed1500

Improve test coverage for test_common_schema.py (#10740)

aef72be

Adds test that an error is raised with specific message when unkown object type is passed

Fix typo in test_dag_run_schema.py (#10739)

de7b5a2

Improve test coverage for ConfObject in dag_run_schema (#10738)

c81aaf4

Adds test to verify that string can be passed to conf and ConfObject._deserialize works.

Add black to STATIC_CODE_CHECKS.rst (#10737)

f4cca23

Add black to the table for Static Code Checkers

Remove duplicate entries from .mailmap (#10736)

d73f7bd

Add docs for how airflow manages packages and imports (#10303)

16df83d

Asynchronous execution of Dataproc jobs with a Sensor (#10673)

d389432

Make scripts/ci/openapi Google Shell Guide compatible (#10747)

09cd291

Part of #10576

joshi95 closed this Sep 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetching databricks host from connection if not supplied in extras.#10749

Fetching databricks host from connection if not supplied in extras.#10749
joshi95 wants to merge 620 commits intoapache:masterfrom
joshi95:databricks-hook-host-fix

joshi95 commented Sep 5, 2020 •

edited

Loading

Uh oh!

mik-laj left a comment

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

joshi95 commented Sep 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mik-laj left a comment

Choose a reason for hiding this comment

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 5, 2020

Uh oh!

potiuk commented Sep 5, 2020

Uh oh!

joshi95 commented Sep 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

joshi95 commented Sep 5, 2020 •

edited

Loading