Add impersonation to Google operators

[olchas]

This PR adds direct impersonation of a service account in Google services. Part-of #8803

A few notes summarizing the changes:

• no changes in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator - as these operators do not use Credentials class for authentication
• in case of operators that connect to multiple Google services, all hooks use the same value of impersonation_chain (if applicable)
• in case of operators that also communicate with services of other cloud providers, the argument is named google_impersonation_chain
• changed get_credentials_and_project method of _CredentialProvider class so that if impersonation_chain argument is used then project_id returned is extracted from the e-mail of the impersonated account (target_principal). This is for the scenario when impersonated service account is from different project than the account from Connection.
  project_id will still be overridden if specified in Connection's extras or explicitly in operator's arguments.
• GCSToS3Operator was modified so that it no longer derivatives from GCSListObjectsOperator
• impersonation_chain was moved to be the last argument of hooks (if it was not already) to avoid potentially breaking changes when initializing these hooks with positional arguments

---

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[mik-laj]

@amithmathew Can you look at it?

[olchas]

Changed get_credentials_and_project method of _CredentialProvider class so that project_id returned is extracted from target_principal (e-mail of final impersonated service account) - this is for the purpose of impersonating service account from different project than the one specified in Connection. project_id can still be overwritten in Connection's extras or in operator's attributes.

[olchas]

Implemented impersonation in newly added Datastore operators (#10032)

[turbaszek]

@olchas would you mind adding an example DAG to show how users can use impersonation?

[codecov-commenter]

Codecov Report

Merging #10052 into master will decrease coverage by 53.95%.
The diff coverage is 44.38%.

@@             Coverage Diff             @@
##           master   #10052       +/-   ##
===========================================
- Coverage   89.06%   35.10%   -53.96%     
===========================================
  Files        1037     1037               
  Lines       49923    50230      +307     
===========================================
- Hits        44464    17634    -26830     
- Misses       5459    32596    +27137     

Flag	Coverage Δ
#kubernetes-tests-image-3.6-v1.16.9	?
#kubernetes-tests-image-3.6-v1.17.5	?
#kubernetes-tests-image-3.6-v1.18.6	?
#kubernetes-tests-image-3.7-v1.16.9	?
#kubernetes-tests-image-3.7-v1.17.5	?
#kubernetes-tests-image-3.7-v1.18.6	?
#mysql-tests-Core-3.7-5.7	?
#mysql-tests-Core-3.8-5.7	?
#mysql-tests-Integration-3.7-5.7	34.74% <44.38%> (-0.04%)	⬇️
#mysql-tests-Integration-3.8-5.7	?
#postgres-tests-Core-3.6-10	?
#postgres-tests-Core-3.6-9.6	?
#postgres-tests-Core-3.7-10	?
#postgres-tests-Core-3.7-9.6	?
#postgres-tests-Integration-3.6-10	34.72% <44.38%> (-0.04%)	⬇️
#postgres-tests-Integration-3.6-9.6	34.72% <44.38%> (-0.04%)	⬇️
#postgres-tests-Integration-3.7-10	34.72% <44.38%> (-0.04%)	⬇️
#postgres-tests-Integration-3.7-9.6	?
#sqlite-tests-Core-3.6	?
#sqlite-tests-Core-3.8	?
#sqlite-tests-Integration-3.6	34.17% <44.38%> (-0.04%)	⬇️
#sqlite-tests-Integration-3.8	34.43% <44.38%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
airflow/providers/google/cloud/hooks/bigquery.py	19.25% <ø> (-73.53%)	⬇️
airflow/providers/google/cloud/hooks/dataflow.py	30.71% <ø> (-58.93%)	⬇️
airflow/providers/google/cloud/hooks/datastore.py	30.76% <ø> (-67.04%)	⬇️
airflow/providers/google/cloud/hooks/gcs.py	16.43% <ø> (-70.48%)	⬇️
.../providers/google/cloud/hooks/kubernetes_engine.py	39.18% <ø> (-56.76%)	⬇️
...ow/providers/google/cloud/operators/datacatalog.py	25.56% <2.32%> (-67.86%)	⬇️
...viders/google/cloud/operators/cloud_memorystore.py	31.84% <12.00%> (-66.59%)	⬇️
...oviders/google/cloud/utils/credentials_provider.py	28.90% <16.66%> (-67.82%)	⬇️
...irflow/providers/google/cloud/operators/spanner.py	27.97% <20.00%> (-70.79%)	⬇️
airflow/providers/google/cloud/operators/tasks.py	30.76% <21.21%> (-68.38%)	⬇️
... and 973 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95328c4...34fadb3. Read the comment docs.

[codecov-commenter]

Codecov Report

Merging #10052 into master will decrease coverage by 54.33%.
The diff coverage is 44.38%.

@@             Coverage Diff             @@
##           master   #10052       +/-   ##
===========================================
- Coverage   89.44%   35.11%   -54.34%     
===========================================
  Files        1037     1037               
  Lines       49927    50234      +307     
===========================================
- Hits        44658    17639    -27019     
- Misses       5269    32595    +27326     

Flag	Coverage Δ
#kubernetes-tests-3.6-9.6	?
#kubernetes-tests-image-3.6-v1.16.9	?
#kubernetes-tests-image-3.6-v1.17.5	?
#kubernetes-tests-image-3.6-v1.18.6	?
#kubernetes-tests-image-3.7-v1.16.9	?
#kubernetes-tests-image-3.7-v1.17.5	?
#kubernetes-tests-image-3.7-v1.18.6	?
#mysql-tests-Core-3.8-5.7	?
#mysql-tests-Integration-3.7-5.7	?
#mysql-tests-Integration-3.8-5.7	35.01% <44.38%> (-0.04%)	⬇️
#postgres-tests-Core-3.6-10	?
#postgres-tests-Core-3.6-9.6	?
#postgres-tests-Core-3.7-10	?
#postgres-tests-Core-3.7-9.6	?
#postgres-tests-Integration-3.6-10	34.73% <44.38%> (-0.04%)	⬇️
#postgres-tests-Integration-3.6-9.6	?
#postgres-tests-Integration-3.7-10	34.73% <44.38%> (-0.04%)	⬇️
#postgres-tests-Integration-3.7-9.6	34.73% <44.38%> (?)
#sqlite-tests-Core-3.6	?
#sqlite-tests-Core-3.8	?
#sqlite-tests-Integration-3.6	34.18% <44.38%> (-0.04%)	⬇️
#sqlite-tests-Integration-3.8	34.44% <44.38%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
airflow/providers/google/cloud/hooks/bigquery.py	19.25% <ø> (-73.53%)	⬇️
airflow/providers/google/cloud/hooks/dataflow.py	30.71% <ø> (-58.93%)	⬇️
airflow/providers/google/cloud/hooks/datastore.py	30.76% <ø> (-67.04%)	⬇️
airflow/providers/google/cloud/hooks/gcs.py	16.43% <ø> (-70.48%)	⬇️
.../providers/google/cloud/hooks/kubernetes_engine.py	39.18% <ø> (-56.76%)	⬇️
...ow/providers/google/cloud/operators/datacatalog.py	25.56% <2.32%> (-67.86%)	⬇️
...viders/google/cloud/operators/cloud_memorystore.py	31.84% <12.00%> (-66.59%)	⬇️
...oviders/google/cloud/utils/credentials_provider.py	28.90% <16.66%> (-67.82%)	⬇️
...irflow/providers/google/cloud/operators/spanner.py	27.97% <20.00%> (-70.79%)	⬇️
airflow/providers/google/cloud/operators/tasks.py	30.76% <21.21%> (-68.38%)	⬇️
... and 973 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b3287d...9f41190. Read the comment docs.

[olchas]

Implemented impersonation in newly added Bigtable operator (#10340)

[olchas]

I have added instructions of setting up and using direct impersonation in docs/howto/connection/gcp.rst.

@mik-laj, could you take a look?

@turbaszek, do you think it will suffice for usage example? I am not sure if there is much point building a whole example dag around impersonation - usage for every Google operator is the same.

[turbaszek]

That's huge. The docs look good and I hope that the auto refactors worked as expected. Regarding the example, I think we should add it but we can create an issue for that so no need to do this now.

[mik-laj]

I would like to do a review. Please wait a moment.

[mik-laj]

Are you planning to add impersonations for DataflowCreatePythonJobOperator and DataflowCreateJavaJobOperator, GKEStartPodOperator and BigQueryTablePartitionExistenceSensor?

[olchas]

@mik-laj I have added impersonation to BigQueryTablePartitionExistenceSensor. As I mentioned in PR's description, DataflowCreatePythonJobOperator, DataflowCreateJavaJobOperator and GKEStartPodOperator use different form of authentication (they do not use Credentials class) and I would like to leave them for another PR.