[BEAM-12641] Use google-auth instead of oauth2client for GCP auth #15004

chunyang · 2021-06-11T19:57:17Z

This change switches the GCP auth library from oauth2client to google-auth. google-auth-httplib2 library is used to provide authorized HTTP clients that work with the existing vendored libraries for BigQuery, GCS, etc.

I'm interested in this migration because of the need to use custom token URIs for issuing service account tokens--it's supported by google-auth but not oauth2client.

dev list discussion thread: https://lists.apache.org/x/thread.html/r4345315bfa80c6181138d556317a45d7692cd2b37d1d341238e5440a@%3Cdev.beam.apache.org%3E

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-06-11T20:15:13Z

Codecov Report

Merging #15004 (e1eadbe) into master (eeccbb0) will increase coverage by 9.68%.
The diff coverage is 70.83%.

@@            Coverage Diff             @@
##           master   #15004      +/-   ##
==========================================
+ Coverage   73.94%   83.63%   +9.68%     
==========================================
  Files         667      453     -214     
  Lines       88071    63264   -24807     
==========================================
- Hits        65128    52912   -12216     
+ Misses      21832    10352   -11480     
+ Partials     1111        0    -1111

Flag	Coverage Δ
go	`?`
python	`83.63% <70.83%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/internal/gcp/auth.py	`81.03% <70.83%> (-0.45%)`	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	`87.50% <0.00%> (-1.71%)`	⬇️
...thon/apache_beam/runners/worker/sdk_worker_main.py	`74.12% <0.00%> (-1.40%)`	⬇️
...n/apache_beam/runners/interactive/cache_manager.py	`87.58% <0.00%> (-1.01%)`	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	`90.00% <0.00%> (-1.00%)`	⬇️
sdks/python/apache_beam/coders/coder_impl.py	`93.97% <0.00%> (-0.31%)`	⬇️
...s/interactive/dataproc/dataproc_cluster_manager.py	`88.32% <0.00%> (-0.26%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`90.51% <0.00%> (-0.23%)`	⬇️
...apache_beam/runners/dataflow/internal/apiclient.py	`77.27% <0.00%> (-0.13%)`	⬇️
sdks/python/apache_beam/coders/coders.py	`87.98% <0.00%> (-0.12%)`	⬇️
... and 225 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eeccbb0...e1eadbe. Read the comment docs.

aaltay · 2021-06-14T18:35:31Z

As a reference, I was referred to: https://github.com/googleapis/google-auth-library-python-httplib2 as how some other libraries went through a similar transition.

chunyang · 2021-06-15T15:22:26Z

Thanks! I wasn't aware that library existed, will see if I can incorporate it.

:sdks:python:test-suites:tox:py38:testPy38CloudCoverage fails with ``` ImportError: cannot import name 'enquote_executable' from 'distlib.scripts' (/home/chuck.yang/beam/build/gradleenv/-1227304282/lib/python3.8/site-packages/distlib/scripts.py) ``` when I don't bump this version. It's odd because 0.3.1 should be enough to solve the problem...

Use google-auth-httplib2 library instead of implementing our own shims.

aaltay · 2021-06-29T04:21:51Z

R: @tvalentyn - Could you review this? Do you see any issues with Dataflow internals related to this?

pabloem · 2021-08-12T18:01:03Z

What's the status of this PR? @tvalentyn @chunyang

chunyang · 2021-08-12T18:27:28Z

@pabloem I'm waiting for a review, I heard that this change may not be compatible with the internal repo at Google, but hope @tvalentyn can confirm/deny. On my side I need to look at the Python PreCommit failure. It seems unrelated to me at first glance.

tvalentyn · 2021-08-31T18:45:46Z

Thanks @chunyang for working on this and your patience with this change. Adding a dependency on google-auth-httplib2 should be possible, looking closer at the change.

tvalentyn · 2021-08-31T18:52:03Z

sdks/python/apache_beam/internal/gcp/auth.py

@@ -115,29 +131,32 @@ def get_service_credentials(cls):

  @staticmethod
  def _get_service_credentials():
-    if is_running_in_gce:
-      # We are currently running as a GCE taskrunner worker.
-      return _GceAssertionCredentials(user_agent='beam-python-sdk/1.0')


I think this path was used to authenticate requests originating from GCE VMs, and we had it so that when Beam SDK is running Dataflow workers, the requests to GCP were authenticated just because the SDK is running on GCE VM.
I wonder how this will work with the new dependency.

Thanks for the review @tvalentyn. My understanding is that the google.auth.default() call in line 151 will attempt to find credentials on GCE VMs using the instance Metadata Server so we don't need a special case within the Beam code. Is this something we can check via the existing integration tests?

https://github.com/googleapis/google-auth-library-python/blob/08c987d0215c9d3e230efe5b7c13e6b8197267bc/google/auth/_default.py#L386-L389

tvalentyn · 2021-08-31T22:08:07Z

Run Python 3.7 PostCommit

tvalentyn · 2021-08-31T22:08:59Z

Yes, we should be able to check the behavior in postcommit integration tests.

tvalentyn · 2021-09-14T21:55:59Z

Run Python 3.7 PostCommit

…/auth-shim

tvalentyn · 2021-09-24T06:40:47Z

Run Python 3.7 PostCommit

chunyang · 2021-09-24T19:27:53Z

Oops sorry, I broke the postcommit. Fix pending in #15584

chunyang · 2021-09-25T04:34:24Z

Run Python 3.7 PostCommit

tvalentyn · 2022-02-10T00:04:38Z

That's good to know, thanks for sharing. We have a plan to complete this migration as well.

yeandy · 2022-03-09T21:04:46Z

Hi @chunyang, I will be doing internal verification for these changes. It seems that I don't have permission to push to your fork/branch. Please rebase or merge master. Can you give me access to this PR? In the meantime, I'll create temp PR to test internally.

chunyang · 2022-03-09T23:41:47Z

Hi @yeandy, I merged master and added you as a collaborator on my fork. Is that enough permissions?

yeandy · 2022-03-10T13:57:38Z

Thanks, that should work.

yeandy

I've done the internal verification, and everything seems to work fine.

Before final approval/merge, I have a few questions @tvalentyn.

Do we want to also remove oauth2client from the four base_image_requirements.txt files under the sdks/python/container/py3* directories too?
And do we still need oath2client in deps_urls_py.yaml?

yeandy · 2022-03-10T21:36:46Z

sdks/python/build-requirements.txt

@@ -24,4 +24,4 @@ grpcio-tools==1.37.0
 mypy-protobuf==1.18

 # Avoid https://github.com/pypa/virtualenv/issues/2006
-distlib==0.3.1


Why has this been bumped?

I ran into an error running tests locally when I didn't bump the version (see commit 0bd2c24). However, I don't think Jenkins had the same problem. We can probably revert this bump to minimize the change set?

Yes, I think we can revert this back.

tvalentyn · 2022-03-18T20:18:45Z

Yes, we should regenerate the container dependencies, can be done in separate commit/pr, or you can follow https://s.apache.org/beam-python-requirements-generate and push more commits to this branch.

we can remove the entry in deps_urls_py.yaml in the same change.

tvalentyn · 2022-03-18T20:19:10Z

Running s.apache.org/beam-python-requirements-generate currently requires a linux machine. some deps are not installable on macos.

yeandy · 2022-03-21T13:26:47Z

Thanks for the info! I'll start regenerating them.

….yaml

yeandy · 2022-03-21T21:54:59Z

R: @tvalentyn PTAL. Regenerated requirements, removed oauth2client from dep_urls_py.yaml, and bumped distlib version back down.

tvalentyn · 2022-03-21T21:57:10Z

Happy to merge when tests pass, thanks all.

yeandy · 2022-03-21T22:24:08Z

Was just reviewing again, and not sure why, but oauth2client is still in the base_image_requirements.txt files, even though I generated them with oauth2client>=2.0.1,<5 deleted from setup.py. Maybe I did the generation incorrectly?

tvalentyn · 2022-03-21T22:45:43Z

is it a transitive dep of some other dep? you can install pipdeptree to check.

yeandy · 2022-03-22T15:06:00Z

Seems that it's a dependency of google-apitools

google-apitools==0.5.31
  - fasteners [required: >=0.14, installed: 0.16.3]
    - six [required: Any, installed: 1.16.0]
  - httplib2 [required: >=0.8, installed: 0.19.1]
    - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
  - oauth2client [required: >=1.4.12, installed: 4.1.3]
    - httplib2 [required: >=0.9.1, installed: 0.19.1]
      - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
    - pyasn1 [required: >=0.1.7, installed: 0.4.8]
    - pyasn1-modules [required: >=0.0.5, installed: 0.2.8]
      - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
    - rsa [required: >=3.1.4, installed: 4.8]
      - pyasn1 [required: >=0.1.3, installed: 0.4.8]
    - six [required: >=1.6.1, installed: 1.16.0]
  - six [required: >=1.12.0, installed: 1.16.0]

yeandy · 2022-03-23T18:13:31Z

@tvalentyn If I'm understanding this correctly, this doesn't require any additional regeneration, right?

tvalentyn · 2022-03-23T18:49:15Z

Run Portable_Python PreCommit

tvalentyn · 2022-03-23T18:49:25Z

Run PythonDocker PreCommit

tvalentyn · 2022-03-23T18:49:47Z

Run Python PreCommit

tvalentyn · 2022-03-23T18:49:59Z

@tvalentyn If I'm understanding this correctly, this doesn't require any additional regeneration, right?

right

tvalentyn · 2022-03-24T00:06:52Z

We need to revert the change to dep_urls_py.yaml.

tvalentyn · 2022-03-25T19:55:31Z

retest this please

tvalentyn · 2022-03-25T19:56:17Z

hmm. not seeing test signals for some reason, might be a github issue

tvalentyn · 2022-03-25T19:56:35Z

ok, triggered now.

aaltay · 2022-03-28T20:36:43Z

Congratulations! And thank you both!

chunyang force-pushed the cyang/auth-shim branch from 61afa23 to 609999d Compare June 11, 2021 20:22

chunyang added 3 commits June 15, 2021 11:25

Add adapter to use google-auth creds with apitools

2611878

Use google-auth-httplib2 library

485c035

Use google-auth-httplib2 library instead of implementing our own shims.

chunyang force-pushed the cyang/auth-shim branch from 1642834 to 485c035 Compare June 15, 2021 18:26

chunyang added 4 commits July 20, 2021 14:07

Merge remote-tracking branch 'origin/master' into cyang/auth-shim

cb67cd5

Remove oauth2client dependency from ownership file

8b20c57

Re-add oauth2client to license dep URLs

23b0fca

Fix spacing

4ea4c56

chunyang changed the title ~~Use google-auth instead of oauth2client for GCP auth~~ [BEAM-12641] Use google-auth instead of oauth2client for GCP auth Jul 20, 2021

pabloem requested a review from tvalentyn August 12, 2021 18:01

Merge remote-tracking branch 'origin/master' into cyang/auth-shim

441bdac

tvalentyn reviewed Aug 31, 2021

View reviewed changes

Merge branch 'master' into cyang/auth-shim

aa6fec3

chunyang added 2 commits September 15, 2021 20:14

Merge remote-tracking branch 'origin/master' into cyang/auth-shim

ab47155

Merge branch 'cyang/auth-shim' of github.com:chunyang/beam into cyang…

4b29e94

…/auth-shim

Merge remote-tracking branch 'origin/master' into cyang/auth-shim

1205f58

Merge remote-tracking branch 'origin/master' into cyang/auth-shim

fc55794

yeandy reviewed Mar 18, 2022

View reviewed changes

yeandy added 2 commits March 21, 2022 21:40

Regenerate python requirements; Remove oauth2client from deps_urls_py…

5b8d05c

….yaml

Revert distlib version

543bc50

tvalentyn approved these changes Mar 21, 2022

View reviewed changes

Revert dep_urls_py.yaml

e1eadbe

tvalentyn merged commit 2e7f5f3 into apache:master Mar 28, 2022

chunyang deleted the cyang/auth-shim branch March 28, 2022 18:04

[BEAM-12641] Use google-auth instead of oauth2client for GCP auth #15004

[BEAM-12641] Use google-auth instead of oauth2client for GCP auth #15004

Conversation

chunyang commented Jun 11, 2021 • edited

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Jun 11, 2021 • edited

Codecov Report

aaltay commented Jun 14, 2021

chunyang commented Jun 15, 2021

aaltay commented Jun 29, 2021

pabloem commented Aug 12, 2021

chunyang commented Aug 12, 2021

tvalentyn commented Aug 31, 2021

tvalentyn Aug 31, 2021

Choose a reason for hiding this comment

chunyang Aug 31, 2021

Choose a reason for hiding this comment

tvalentyn commented Aug 31, 2021

tvalentyn commented Aug 31, 2021

tvalentyn commented Sep 14, 2021

tvalentyn commented Sep 24, 2021

chunyang commented Sep 24, 2021

chunyang commented Sep 25, 2021

tvalentyn commented Feb 10, 2022

yeandy commented Mar 9, 2022

chunyang commented Mar 9, 2022

yeandy commented Mar 10, 2022

yeandy left a comment

Choose a reason for hiding this comment

yeandy Mar 10, 2022

Choose a reason for hiding this comment

chunyang Mar 18, 2022

Choose a reason for hiding this comment

yeandy Mar 21, 2022

Choose a reason for hiding this comment

tvalentyn commented Mar 18, 2022

tvalentyn commented Mar 18, 2022 • edited

yeandy commented Mar 21, 2022

yeandy commented Mar 21, 2022

tvalentyn commented Mar 21, 2022

yeandy commented Mar 21, 2022

tvalentyn commented Mar 21, 2022

yeandy commented Mar 22, 2022

yeandy commented Mar 23, 2022

tvalentyn commented Mar 23, 2022

tvalentyn commented Mar 23, 2022

tvalentyn commented Mar 23, 2022

tvalentyn commented Mar 23, 2022

tvalentyn commented Mar 24, 2022

tvalentyn commented Mar 25, 2022

tvalentyn commented Mar 25, 2022

tvalentyn commented Mar 25, 2022

aaltay commented Mar 28, 2022

chunyang commented Jun 11, 2021 •

edited

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Jun 11, 2021 •

edited

tvalentyn commented Mar 18, 2022 •

edited