Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update SDKs for google provider package #30067

Merged
merged 31 commits into from
May 17, 2023
Merged

Update SDKs for google provider package #30067

merged 31 commits into from
May 17, 2023

Commits on May 17, 2023

  1. Update SDK versions for Google provider

    @ahidalgob
    Lukasz Wyszomirski authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    0829a7a View commit details
    Browse the repository at this point in the history

  2. Adjust google ads operators to v12

    @ahidalgob
    Lukasz Wyszomirski authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    73d2aa7 View commit details
    Browse the repository at this point in the history

  3. Fix BigQuery system tests 

    Changes:
- fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py
- fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries_async.py
    @moiseenkov @ahidalgob
    moiseenkov authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    86d2beb View commit details
    Browse the repository at this point in the history

  4. Fix GCS system tests

    @moiseenkov @ahidalgob
    moiseenkov authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    5af5781 View commit details
    Browse the repository at this point in the history

  5. Fix CloudBuild unit test

    @moiseenkov @ahidalgob
    moiseenkov authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    aeec47e View commit details
    Browse the repository at this point in the history

  6. Update BigTable operators to accomodate for new dependencies.

    @kosteev @ahidalgob
    kosteev authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    9c0a6bc View commit details
    Browse the repository at this point in the history

  7. Fix Cloud Tasks System tests 

    Tasks dag was quite flaky without the retry option in the run_task step,
but it's consistently green with the option set.

We also add a GCP_APP_ENGINE_LOCATION env variable since this depends on
the used GCP Project App Engine's location
    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    61ed6ae View commit details
    Browse the repository at this point in the history

  8. Add setup docstring to Tasks system tests.

    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    3947709 View commit details
    Browse the repository at this point in the history

  9. Update Vision operators to accommodate new dependencies. 

    Changes:
- fix methods for CloudVisionHook
- fix Vision Operators
- fix tests/providers/google/cloud/hooks/test_vision.py
- fix tests/providers/google/cloud/operators/test_vision.py
- fix tests/system/providers/google/cloud/vision/example_vision_annotate_image.py
- fix tests/system/providers/google/cloud/vision/example_vision_autogenerated.py
- fix tests/system/providers/google/cloud/vision/example_vision_explicit.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    ce791d2 View commit details
    Browse the repository at this point in the history

  10. Update SpeechToText operators to accommodate new dependencies. 

    Changes:
- fix synthesize_speech method for CloudTextToSpeechHook
- fix CloudSpeechToTextRecognizeSpeechOperator
- fix tests/providers/google/cloud/operators/test_speech_to_text.py
- fix tests/providers/google/cloud/hooks/test_text_to_speech.py
- fix tests/providers/google/cloud/hooks/test_speech_to_text.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    213db85 View commit details
    Browse the repository at this point in the history

  11. Update Translate Speech operators to accommodate new dependencies. 

    Changes:
- fix synthesize_speech method for CloudTextToSpeechHook
- fix CloudTranslateSpeechOperator
- tests/providers/google/cloud/operators/test_translate_speech.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    a951e2f View commit details
    Browse the repository at this point in the history

  12. Update VideoIntelligence operators to accommodate new dependencies. 

    Changes:
- fix annotate_video method for CloudVideoIntelligenceHook
- fix VideoIntelligence Operators
- fix tests/providers/google/cloud/hooks/test_video_intelligence.py
- fix tests/providers/google/cloud/operators/test_video_intelligence.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    4d957ba View commit details
    Browse the repository at this point in the history

  13. Update Compute Engine operators to accomodate for new dependencies. 

    Changes:
- added wait_for_operation_complete() method to check the execution flow
- added new attribute cmd_timeout for ComputeEngineSSHHook
    @VladaZakharova @ahidalgob
    VladaZakharova authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    deb017c View commit details
    Browse the repository at this point in the history

  14. Fix Stackdriver system test 

    This test has not worked because of slack channel and credentials not
being setup. We now test the same operators by creating notification
channels and policy alerts against pubsub topics, which don't need to
exist before the test is ran, making the test self-contained.
    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    75ae7cd View commit details
    Browse the repository at this point in the history

  15. Update Natural Language operators to accommodate new dependencies. 

    Changes:
- fix airflow/providers/google/cloud/operators/natural_language.py
- fix airflow/providers/google/cloud/hooks/natural_language.py
- fix tests/providers/google/cloud/hooks/test_natural_language.py
- fix tests/providers/google/cloud/operators/test_natural_language.py
- fix tests/system/providers/google/cloud/natural_language/example_natural_language.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    081e340 View commit details
    Browse the repository at this point in the history

  16. Update Composer system tests. 

    Fix environment id to contain underscores.
    @kosteev @ahidalgob
    kosteev authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    d15e09b View commit details
    Browse the repository at this point in the history

  17. Update AutoML operators to accommodate new dependencies. 

    Changes:
- add timeout parameter to all long-running operations for operators
- fix tests/system/providers/google/cloud/automl/example_automl_dataset.py
- fix tests/system/providers/google/cloud/automl/example_automl_model.py
- fix tests/system/providers/google/cloud/automl/example_automl_nl_text_extraction.py
- fix tests/system/providers/google/cloud/automl/example_automl_vision_classification.py
    @VladaZakharova @ahidalgob
    VladaZakharova authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    8945d37 View commit details
    Browse the repository at this point in the history

  18. Fix Cloud SQL delete operator 

    For some delete instance operations, the operation stops being available ~9 seconds after completion, so we need a shorter sleep time to make sure we don'tmiss the DONE status.
    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    89a6cca View commit details
    Browse the repository at this point in the history

  19. Update VertexAI operators to accommodate new dependencies.

    @VladaZakharova @ahidalgob
    VladaZakharova authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    42e8dc0 View commit details
    Browse the repository at this point in the history

  20. Add SQL to Sheets Test instructions

    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    781b3f6 View commit details
    Browse the repository at this point in the history

  21. Update Dataproc Metastore operators to accommodate new dependencies.

    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    734b0b0 View commit details
    Browse the repository at this point in the history

  22. Update Dataproc operators to accommodate new dependencies.

    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    d34fbf4 View commit details
    Browse the repository at this point in the history

  23. Update Dataflow sys tests to new sdk

    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    5562618 View commit details
    Browse the repository at this point in the history

  24. Update Dataproc on gke operators to accommodate new dependencies.

    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    1b7aa2e View commit details
    Browse the repository at this point in the history

  25. Update MLEngine operators to accomodate new dependencies. 

    Changes:
- update train model that is used for prediction
- update version and runner for ApacheBeam in utils for MLEngine
- update connection inside async hook
    @VladaZakharova @ahidalgob
    VladaZakharova authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7420103 View commit details
    Browse the repository at this point in the history

  26. Update Dataprep operators to accommodate new dependencies. 

    Changes:
- fix tests/system/providers/google/cloud/dataprep/example_dataprep.py
    @ahidalgob
    Beata Kossakowska authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    d783329 View commit details
    Browse the repository at this point in the history

  27. Add Dataflow Go system test

    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    b160d3b View commit details
    Browse the repository at this point in the history

  28. Update providers.yaml for google

    @ahidalgob
    Lukasz Wyszomirski authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    0bc3cc6 View commit details
    Browse the repository at this point in the history

  29. fixup! Update providers.yaml for google

    @potiuk @ahidalgob
    potiuk authored and ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    c60683c View commit details
    Browse the repository at this point in the history

  30. Google SDK Fixes after rebase 

    - Secret Manager was missing updating to v2, now expects a request dict
- Compute ssh had a bug when no cmd_timeout was passed
- Cloud Build tests were improved/refactored in community, so deleting
old ones
- googleapiclient.errors.HttpError was incorrectly used in our tests, it
it didn´t matter before but a change in the class makes HttpError()
raise an error in initialization the way we were using it before
- fix static checks

```
$ pytest tests/providers/google/cloud/
...
===== 2763 passed, 71 skipped, 21 warnings in 193.46s (0:03:13) =====
```
    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    51a6787 View commit details
    Browse the repository at this point in the history

  31. Fix Google providers type errors

    @ahidalgob
    ahidalgob committed May 17, 2023
    Configuration menu
    Copy the full SHA
    31d7cd6 View commit details
    Browse the repository at this point in the history