[BEAM-9359] Switch to Data Catalog client

[TheNeuralBit]

Use DataCatalog client, rather than using the gRPC stub directly.

CC: @kennknowles

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[TheNeuralBit]

Run SQL PostCommit

[lukecwik]

Have you looked at the linkage checker for issues that swapping versions is going to cause?

It also looks like we are regressing the package namespace to v1beta1 even though we are increasing the version number.

[TheNeuralBit]

This isn't just swapping versions, it's switching to using the java client library, rather than using the gRPC stub and protos directly.

How do I run the linkage checker?

[lukecwik]

Linkage checker instructions:

beam/build.gradle

Line 294 in 76501ab

// Reports linkage errors across multiple Apache Beam artifact ids.

// Reports linkage errors across multiple Apache Beam artifact ids.
//
// To use (from the root of project):
//    ./gradlew -Ppublishing -PjavaLinkageArtifactIds=artifactId1,artifactId2,... :checkJavaLinkage
//
// For example:
//    ./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-core,beam-sdks-java-io-jdbc :checkJavaLinkage
//
// Note that this task publishes artifacts into your local Maven repository.

[kennknowles]

Code changes LGTM pending linkage checker. Seems there's some lack of clarity around the best way to publicly depend on this functionality.

[TheNeuralBit]

One thing I noted after testing internally is that DataCatalogClient catches and re-throws StatusRuntimeExceptions with more specific exceptions (e.g. PermissionDeniedException) I'll need to make sure we update those references.

[TheNeuralBit]

Result of running the linkage checker script is here: https://gist.github.com/TheNeuralBit/30d4b4a851dcaeed1651cc19fedd3159

I could use some help interpreting the results.

[TheNeuralBit]

Seems there's some lack of clarity around the best way to publicly depend on this functionality.

The GCP Data Catalog docs indicate people should be using this client library, rather than the gRPC stub.

[TheNeuralBit]

Run SQL PostCommit

[TheNeuralBit]

I pushed a change to replace the StatueRuntimeException with specific gax exceptions. Planning on merging when tests are green.

[lukecwik]

the linkage checker results are stale and are picking up warnings due to a jackson version change so it seems like no new linage issues have been reported