WIP [AIRFLOW-3601] add location support to BigQuery operators #4409
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #4409 +/- ##
=========================================
- Coverage 78.25% 78.2% -0.05%
=========================================
Files 204 204
Lines 16433 16445 +12
=========================================
+ Hits 12859 12861 +2
- Misses 3574 3584 +10
Continue to review full report at Codecov.
|
This reverts commit eb76c30.
|
Thanks, I was planning to do this. @yohei1126 Can you please add tests for all the corresponding operators. |
|
yes working on it |
yohei1126
changed the title
|
@kaxil Is there any instruction to run specific test locally? I want to debug the following error. and it is weird the test failed but the job did not fail. |
|
@ron819 Thanks! |
got this error even after |
Add missing type of delegate_to
To help move away from Minikube, we need to remove the dependency on a local docker registry and move towards a solution that can be used in any kubernetes cluster. Custom image names allow users to use systems like docker, artifactory and gcr
* Fix Type Error for BigQueryOperator and support the unicode object. * Add tests
* Refactor Kubernetes operator with git-sync Currently the implementation of git-sync is broken because: - git-sync clones the repository in /tmp and not in airflow-dags volume - git-sync add a link to point to the revision required but it is not taken into account in AIRFLOW__CORE__DAGS_FOLDER Dags/logs hostPath volume has been added (needed if airflow run in kubernetes in local environment) To avoid false positive in CI `load_examples` is set to `False` otherwise DAGs from `airflow/example_dags` are always loaded. In this way is possible to test `import` in DAGs Remove `worker_dags_folder` config: `worker_dags_folder` is redundant and can lead to confusion. In WorkerConfiguration `self.kube_config.dags_folder` defines the path of the dags and can be set in the worker using airflow_configmap Refactor worker_configuration.py Use a docker container to run setup.py Compile web assets Fix codecov application path * Fix kube_config.dags_in_image
* Remove dagbag from trigger call * Adding fix to rbac * empty commit * Added create_dagrun to DagModel * Adding testing to /trigger calls * Make session a class var
…ce Flake8 test was broken (apache#4415) The flake8 test in the Travis CI was broken since apache#4361 (apache@7a6acbf ) And some Flake8 errors (code style/quality issues. found in 10 files) were introduce since flake8 test was broken.
Add missing type of delegate_to
To help move away from Minikube, we need to remove the dependency on a local docker registry and move towards a solution that can be used in any kubernetes cluster. Custom image names allow users to use systems like docker, artifactory and gcr
* Fix Type Error for BigQueryOperator and support the unicode object. * Add tests
fix flake8 errors
* Refactor Kubernetes operator with git-sync Currently the implementation of git-sync is broken because: - git-sync clones the repository in /tmp and not in airflow-dags volume - git-sync add a link to point to the revision required but it is not taken into account in AIRFLOW__CORE__DAGS_FOLDER Dags/logs hostPath volume has been added (needed if airflow run in kubernetes in local environment) To avoid false positive in CI `load_examples` is set to `False` otherwise DAGs from `airflow/example_dags` are always loaded. In this way is possible to test `import` in DAGs Remove `worker_dags_folder` config: `worker_dags_folder` is redundant and can lead to confusion. In WorkerConfiguration `self.kube_config.dags_folder` defines the path of the dags and can be set in the worker using airflow_configmap Refactor worker_configuration.py Use a docker container to run setup.py Compile web assets Fix codecov application path * Fix kube_config.dags_in_image
* Remove dagbag from trigger call * Adding fix to rbac * empty commit * Added create_dagrun to DagModel * Adding testing to /trigger calls * Make session a class var
…ce Flake8 test was broken (apache#4415) The flake8 test in the Travis CI was broken since apache#4361 (apache@7a6acbf ) And some Flake8 errors (code style/quality issues. found in 10 files) were introduce since flake8 test was broken.
Add missing type of delegate_to
To help move away from Minikube, we need to remove the dependency on a local docker registry and move towards a solution that can be used in any kubernetes cluster. Custom image names allow users to use systems like docker, artifactory and gcr
fix flake8 errors fix error
|
sorry recreated a PR #4448 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira
Description
bigquery_check_operator.pybigquery_get_data.pybigquery_operator.pybigquery_table_delete_operator.pybigquery_to_bigquery.pyBigQueryOperatorBigQueryCreateEmptyTableOperatorBigQueryCreateExternalTableOperatorBigQueryDeleteDatasetOperatorBigQueryCreateEmptyDatasetOperatorbigquery_to_gcs.pygcs_to_bq.pybigquery_sensor.pyTests
bigquery_to_bigquery.pyBigQueryOperatortest_bigquery_hook.py: [AIRFLOW-3327] Add support for location in BigQueryHook #4324 added test for location support of the hook.test_bigquery_operator.py:BigQueryOperatorTest test_bigquery_operator_locationBigQueryCreateEmptyTableOperatortest_bigquery_hook.py:TestBigQueryBaseCursor test_create_empty_table_with_location. A table insert request can have location in its request body (doc).test_bigquery_operator.py:BigQueryCreateEmptyTableOperatorTest test_executeBigQueryCreateExternalTableOperatortest_bigquery_hook.py:TestBigQueryExternalTableSourceFormat test_table_locationA table insert request can have location in its request body (doc).test_bigquery_operator.py:BigQueryCreateExternalTableOperatorTest test_executeBigQueryDeleteDatasetOperatortest_bigquery_hook.py: No test since a delete request does not have lcoation in the request body (doc).test_bigquery_operator.py:BigQueryDeleteDatasetOperatorTest test_executeBigQueryCreateEmptyDatasetOperatorCommits
Documentation
Code Quality
flake8