Skip to content

Latest commit

 

History

History
647 lines (550 loc) · 93.3 KB

README.md

File metadata and controls

647 lines (550 loc) · 93.3 KB

Google Cloud Java Client Libraries

Java idiomatic client for Google Cloud Platform services.

Supported APIs

Libraries are available on GitHub and Maven Central for developing Java applications that interact with individual Google Cloud services:

Client Release Level Version
AI Platform Notebooks stable Maven
API Gateway stable Maven
API Keys API stable Maven
Access Approval stable Maven
Advisory Notifications API stable Maven
AlloyDB stable Maven
AlloyDB connectors stable Maven
Analytics Hub API stable Maven
Apigee Connect stable Maven
App Engine Admin API stable Maven
Artifact Registry stable Maven
Asset Inventory stable Maven
Assured Workloads for Government stable Maven
Auto ML stable Maven
Backup and DR Service API stable Maven
BigQuery stable Maven
BigQuery Connection stable Maven
BigQuery Data Transfer Service stable Maven
BigQuery Reservation stable Maven
BigQuery Storage stable Maven
Bigtable stable Maven
Bigtable Hbase Client stable Maven
Billing stable Maven
Billing Budgets stable Maven
Binary Authorization stable Maven
Build stable Maven
CCAI Insights stable Maven
Certificate Authority Service stable Maven
Channel Services stable Maven
Composer stable Maven
Compute Engine stable Maven
Connect Gateway API stable Maven
Container Analysis stable Maven
Controls Partner API stable Maven
DNS stable Maven
Data Catalog stable Maven
Data Fusion stable Maven
Data Lineage stable Maven
Data Loss Prevention stable Maven
Database Migration Service stable Maven
Dataplex stable Maven
Dataproc stable Maven
Dataproc Metastore stable Maven
Datastore stable Maven
Datastream stable Maven
Debugger stable Maven
Deploy stable Maven
Dialogflow API stable Maven
Discovery Engine API stable Maven
Distributed Edge stable Maven
Distributed Edge Network API stable Maven
Document AI stable Maven
Domains stable Maven
Essential Contacts API stable Maven
Eventarc stable Maven
Filestore API stable Maven
Firestore stable Maven
Fleet Routing stable Maven
Functions stable Maven
GKE Hub API stable Maven
IAM stable Maven
IAM Admin API stable Maven
IAM Policy Troubleshooter API stable Maven
IAM Service Account Credentials API stable Maven
Identity Access Context Manager stable Maven
Identity-Aware Proxy API stable Maven
Infrastructure Manager API stable Maven
Internet of Things (IoT) Core stable Maven
Intrusion Detection System stable Maven
Key Management Service stable Maven
Kubernetes Engine stable Maven
Logging stable Maven
Managed Service for Microsoft Active Directory stable Maven
Memcache stable Maven
Monitoring Dashboards stable Maven
Natural Language stable Maven
Network Connectivity Center stable Maven
Network Management API stable Maven
Network Security API stable Maven
OS Config API stable Maven
OS Login stable Maven
Organization Policy stable Maven
Profiler stable Maven
Pub/Sub stable Maven
Pub/Sub Group Kafka Connector stable Maven
Pub/Sub Lite stable Maven
Pub/Sub Lite Kafka Shim stable Maven
Pub/Sub Lite Spark Connector stable Maven
Public Certificate Authority API stable Maven
Quotas API stable Maven
Recommender stable Maven
Redis stable Maven
Resource Manager API stable Maven
Resource Settings API stable Maven
Retail stable Maven
Routes API stable Maven
Scheduler stable Maven
Secret Management stable Maven
Secure Source Manager API stable Maven
Security Center Management API stable Maven
Security Command Center stable Maven
Security Posture API stable Maven
Security Scanner stable Maven
Serverless VPC Access stable Maven
Service Control API stable Maven
Service Directory stable Maven
Service Management API stable Maven
Service Usage stable Maven
Shell stable Maven
Spanner stable Maven
Spanner JDBC stable Maven
Speech stable Maven
Stackdriver Monitoring stable Maven
Stackdriver Trace stable Maven
Storage stable Maven
Storage Transfer Service stable Maven
TPU stable Maven
Talent Solution stable Maven
Tasks stable Maven
Telco Automation API stable Maven
Text-to-Speech stable Maven
Translation stable Maven
VM Migration stable Maven
Vertex AI stable Maven
VertexAI API stable Maven
Video Intelligence stable Maven
Video Stitcher API stable Maven
Video Transcoder stable Maven
Vision stable Maven
Web Risk stable Maven
Workflow Executions stable Maven
Workflows stable Maven
Workspace Add-ons API stable Maven
Workstations stable Maven
reCAPTCHA Enterprise stable Maven
Ad Manager API preview Maven
Address Validation API preview Maven
Analytics Admin preview Maven
Analytics Data preview Maven
Analytics Hub preview Maven
Anthos Multicloud preview Maven
Apache Kafka for BigQuery API preview Maven
App Hub API preview Maven
Area 120 Tables preview Maven
Backup for GKE preview Maven
Bare Metal Solution preview Maven
Batch preview Maven
BeyondCorp AppConnections preview Maven
BeyondCorp AppConnectors preview Maven
BeyondCorp AppGateways preview Maven
BeyondCorp ClientConnectorServices preview Maven
BeyondCorp ClientGateways preview Maven
BigLake preview Maven
BigQuery DataPolicy API preview Maven
BigQuery Migration preview Maven
CSS API preview Maven
Certificate Manager preview Maven
Chat API preview Maven
Commerce Consumer Procurement preview Maven
Confidential Computing API preview Maven
Data Labeling preview Maven
Dataflow preview Maven
Dataform preview Maven
Developer Connect API preview Maven
Dialogflow CX preview Maven
Document AI Warehouse preview Maven
Enterprise Knowledge Graph preview Maven
Error Reporting preview Maven
Eventarc Publishing preview Maven
GDC Hardware Management API preview Maven
KMS Inventory API preview Maven
Life Sciences preview Maven
Live Stream API preview Maven
Logging Logback Appender preview Maven
Logging Servlet Initializer preview Maven
Maps Platform Datasets API preview Maven
Media Translation API preview Maven
Meet API preview Maven
Memorystore for Redis API preview Maven
Merchant API preview Maven
Merchant API preview Maven
Merchant API preview Maven
Merchant API preview Maven
Merchant API preview Maven
Merchant API preview Maven
Merchant Conversions API preview Maven
Merchant LFP API preview Maven
Merchant Notifications API preview Maven
Merchant Quota API preview Maven
Migration Center API preview Maven
Monitoring Metrics Scopes preview Maven
NIO Filesystem Provider for Storage preview Maven
NetApp API preview Maven
Network Services API preview Maven
Parallelstore API preview Maven
Phishing Protection preview Maven
Places API (New) preview Maven
Policy Simulator API preview Maven
Private Catalog preview Maven
Pub/Sub Lite Flink Connector preview Maven
Rapid Migration Assessment API preview Maven
Recommendations AI preview Maven
Registry API preview Maven
Route Optimization API preview Maven
Run preview Maven
Security Command Center Settings API preview Maven
Service Health API preview Maven
Solar API preview Maven
Storage Insights API preview Maven
Support API preview Maven
VMware Engine preview Maven
Vision AI API preview Maven
Workspace Events API preview Maven

If the service is not listed, google-api-java-client interfaces with additional Google Cloud APIs using a legacy REST interface.

When building Java applications, preference should be given to the libraries listed in the table.

Specifying a Project ID

Most google-cloud libraries require a project ID. There are multiple ways to specify this project ID.

  1. When using google-cloud libraries from within Compute/App Engine, there's no need to specify a project ID. It is automatically inferred from the production environment.
  2. When using google-cloud elsewhere, you can do one of the following:
  • Supply the project ID when building the service options. For example, to use Datastore from a project with ID "PROJECT_ID", you can write:

    Datastore datastore = DatastoreOptions.newBuilder().setProjectId("PROJECT_ID").build().getService();
  • Specify the environment variable GOOGLE_CLOUD_PROJECT to be your desired project ID.

  • Set the project ID using the Google Cloud SDK. To use the SDK, download the SDK if you haven't already, and set the project ID from the command line. For example:

    gcloud config set project PROJECT_ID
    

google-cloud determines the project ID from the following sources in the listed order, stopping once it finds a value:

  1. The project ID supplied when building the service options
  2. Project ID specified by the environment variable GOOGLE_CLOUD_PROJECT
  3. The App Engine / Compute Engine project ID
  4. The project ID specified in the JSON credentials file pointed by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  5. The Google Cloud SDK project ID

In cases where the library may expect a project ID explicitly, we provide a helper that can provide the inferred project ID:

  import com.google.cloud.ServiceOptions;
  ...
  String projectId = ServiceOptions.getDefaultProjectId();

Authentication

google-cloud-java uses https://github.com/googleapis/google-auth-library-java to authenticate requests. google-auth-library-java supports a wide range of authentication types; see the project's README and javadoc for more details.

Google Cloud Platform environment

When using Google Cloud libraries from a Google Cloud Platform environment such as Compute Engine, Kubernetes Engine, or App Engine, no additional authentication steps are necessary.

For example:

Storage storage = StorageOptions.getDefaultInstance().getService();

or:

CloudTasksClient cloudTasksClient = CloudTasksClient.create();

Other environments

Using a service account (recommended)

  1. Generate a JSON service account key.

  2. After downloading that key, you must do one of the following:

    • Define the environment variable GOOGLE_APPLICATION_CREDENTIALS to be the location of the key. For example:
    export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json
    • Supply the JSON credentials file when building the service options. For example, this Storage object has the necessary permissions to interact with your Google Cloud Storage data:
    Storage storage = StorageOptions.newBuilder()
        .setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream("/path/to/my/key.json")))
        .build()
        .getService();

Local development/testing

If running locally for development/testing, you can use the Google Cloud SDK. Create Application Default Credentials with gcloud auth application-default login, and then google-cloud will automatically detect such credentials.

Existing OAuth2 access token

If you already have an OAuth2 access token, you can use it to authenticate (notice that in this case, the access token will not be automatically refreshed):

Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime));
Storage storage = StorageOptions.newBuilder()
    .setCredentials(credentials)
    .build()
    .getService();

or:

Credentials credentials = GoogleCredentials.create(new AccessToken(accessToken, expirationTime));
CloudTasksSettings cloudTasksSettings = CloudTasksSettings.newBuilder()
    .setCredentialProvider(FixedCredentialsProvider.create(credentials))
    .build();
CloudTasksClient cloudTasksClient = CloudTasksClient.create(cloudTasksSettings);

Application Default Credentials

If no credentials are provided, google-cloud will attempt to detect them from the environment using GoogleCredentials.getApplicationDefault() which will search for Application Default Credentials in the following locations (in order):

  1. The credentials file pointed to by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  2. Credentials provided by the Google Cloud SDK gcloud auth application-default login command
  3. Google App Engine built-in credentials
  4. Google Cloud Shell built-in credentials
  5. Google Compute Engine built-in credentials

Troubleshooting

To get help, follow the instructions in the Troubleshooting document.

Configuring a Proxy

Google Cloud client libraries use HTTPS and gRPC in underlying communication with the services. In both protocols, you can configure a proxy using https.proxyHost and (optional) https.proxyPort properties.

gRPC Custom Proxy Configuration

For a more custom proxy with gRPC, you will need supply a ProxyDetector to the ManagedChannelBuilder:

import com.google.api.core.ApiFunction;
import com.google.api.gax.rpc.TransportChannelProvider;
import com.google.cloud.tasks.v2.CloudTasksClient;
import com.google.cloud.tasks.v2.CloudTasksSettings;
import com.google.cloud.tasks.v2.stub.CloudTasksStubSettings;
import io.grpc.HttpConnectProxiedSocketAddress;
import io.grpc.ManagedChannelBuilder;
import io.grpc.ProxiedSocketAddress;
import io.grpc.ProxyDetector;

import javax.annotation.Nullable;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.SocketAddress;

public CloudTasksClient getService() throws IOException {
  TransportChannelProvider transportChannelProvider =
      CloudTasksStubSettings.defaultGrpcTransportProviderBuilder()
          .setChannelConfigurator(
              new ApiFunction<ManagedChannelBuilder, ManagedChannelBuilder>() {
                @Override
                public ManagedChannelBuilder apply(ManagedChannelBuilder managedChannelBuilder) {
                  return managedChannelBuilder.proxyDetector(
                      new ProxyDetector() {
                        @Nullable
                        @Override
                        public ProxiedSocketAddress proxyFor(SocketAddress socketAddress)
                            throws IOException {
                          return HttpConnectProxiedSocketAddress.newBuilder()
                              .setUsername(PROXY_USERNAME)
                              .setPassword(PROXY_PASSWORD)
                              .setProxyAddress(new InetSocketAddress(PROXY_HOST, PROXY_PORT))
                              .setTargetAddress((InetSocketAddress) socketAddress)
                              .build();
                        }
                      });
                }
              })
          .build();
  CloudTasksSettings cloudTasksSettings =
      CloudTasksSettings.newBuilder()
          .setTransportChannelProvider(transportChannelProvider)
          .build();
  return CloudTasksClient.create(cloudTasksSettings);
}

Long Running Operations

Long running operations (LROs) are often used for API calls that are expected to take a long time to complete (i.e. provisioning a GCE instance or a Dataflow pipeline). The initial API call creates an "operation" on the server and returns an Operation ID to track its progress. LRO RPCs have the suffix Async appended to the call name (i.e. clusterControllerClient.createClusterAsync())

Our generated clients provide a nice interface for starting the operation and then waiting for the operation to complete. This is accomplished by returning an OperationFuture. When calling get() on the OperationFuture, the client library will poll the operation to check the operation's status.

For example, take a sample createCluster Operation in google-cloud-dataproc v4.20.0:

try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create()) {
  CreateClusterRequest request =
      CreateClusterRequest.newBuilder()
          .setProjectId("{PROJECT_ID}")
          .setRegion("{REGION}")
          .setCluster(Cluster.newBuilder().build())
          .setRequestId("{REQUEST_ID}")
          .setActionOnFailedPrimaryWorkers(FailureAction.forNumber(0))
          .build();
  OperationFuture<Cluster, ClusterOperationMetadata> future =
      clusterControllerClient.createClusterOperationCallable().futureCall(request);
  // Do something.
  Cluster response = future.get();
} catch (CancellationException e) {
  // Exceeded the default RPC timeout without the Operation completing.
  // Library is no longer polling for the Operation status. Consider 
  // increasing the timeout.
}

LRO Timeouts

The polling operations have a default timeout that varies from service to service. The library will throw a java.util.concurrent.CancellationException with the message: Task was cancelled. if the timeout exceeds the operation. A CancellationException does not mean that the backend GCP Operation was cancelled. This exception is thrown from the client library when it has exceeded the total timeout without receiving a successful status from the operation. Our client libraries respect the configured values set in the OperationTimedPollAlgorithm for each RPC.

Note: The client library handles the Operation's polling mechanism for you. By default, there is no need to manually poll the status yourself.

Default LRO Values

Each LRO RPC has a set of pre-configured default values. You can find these values by searching in each Client's StubSettings's class. The default LRO settings are initialized inside the initDefaults() method in the nested Builder class.

For example, in google-cloud-aiplatform v3.24.0, the default OperationTimedPollAlgorithm has these default values:

OperationTimedPollAlgorithm.create(
    RetrySettings.newBuilder()
        .setInitialRetryDelay(Duration.ofMillis(5000L))
        .setRetryDelayMultiplier(1.5)
        .setMaxRetryDelay(Duration.ofMillis(45000L))
        .setInitialRpcTimeout(Duration.ZERO)
        .setRpcTimeoutMultiplier(1.0)
        .setMaxRpcTimeout(Duration.ZERO)
        .setTotalTimeout(Duration.ofMillis(300000L))
        .build())

Both retries and LROs share the same RetrySettings class. Note the corresponding link:

  • Total Timeout (Max Time allowed for polling): 5 minutes
  • Initial Retry Delay (Initial delay before first poll): 5 seconds
  • Max Retry Delay (Maximum delay between each poll): 45 seconds
  • Retry Delay Multiplier (Multiplier value to increase the poll delay): 1.5

The RPC Timeout values have no use in LROs and can be omitted or set to the default values (Duration.ZERO for Timeouts or 1.0 for the multiplier).

Configuring LRO Timeouts

To configure the LRO values, create an OperationTimedPollAlgorithm object and update the RPC's polling algorithm. For example:

ClusterControllerSettings.Builder settingsBuilder = ClusterControllerSettings.newBuilder();
TimedRetryAlgorithm timedRetryAlgorithm = OperationTimedPollAlgorithm.create(
		RetrySettings.newBuilder()
				.setInitialRetryDelay(Duration.ofMillis(500L))
				.setRetryDelayMultiplier(1.5)
				.setMaxRetryDelay(Duration.ofMillis(5000L))
				.setInitialRpcTimeout(Duration.ZERO) // ignored
				.setRpcTimeoutMultiplier(1.0) // ignored
				.setMaxRpcTimeout(Duration.ZERO) // ignored
				.setTotalTimeout(Duration.ofHours(24L))	// set polling timeout to 24 hours
				.build());
settingsBuilder.createClusterOperationSettings()
		.setPollingAlgorithm(timedRetryAlgorithm);
ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settingsBuilder.build());

Note: The configuration above only modifies the LRO values for the createClusterOperation RPC. The other RPCs in the Client will still use each RPC's pre-configured LRO values.

Managing Dependencies

If you are using more than one Google Cloud client library, we recommend you use one of our Bill of Material (BOM) artifacts to help manage dependency versions. For more information, see Using the Cloud Client Libraries.

Java Versions

Java 8 or above is required for using the clients in this repository.

Supported Platforms

Clients in this repository use either HTTP or gRPC for the transport layer. All HTTP-based clients should work in all environments.

For clients that use gRPC, the supported platforms are constrained by the platforms that Forked Tomcat Native supports, which for architectures means only x86_64, and for operating systems means Mac OS X, Windows, and Linux. Additionally, gRPC constrains the use of platforms with threading restrictions.

Thus, the following are not supported:

  • Android
    • Consider Firebase, which includes many of these APIs.
    • It is possible to use these libraries in many cases, although it is unsupported. You can find examples, such as this one, in this example repository but consider the risks carefully before using these libraries in an application.
  • Raspberry Pi (since it runs on the ARM architecture)
  • Google App Engine Standard Java 7

The following environments should work (among others):

  • standalone Windows on x86_64
  • standalone Mac OS X on x86_64
  • standalone Linux on x86_64
  • Google Compute Engine (GCE)
  • Google Container Engine (GKE)
  • Google App Engine Standard Java 8 (GAE Std J8)
  • Google App Engine Flex (GAE Flex)
  • Alpine Linux (Java 11+)

Testing

This library provides tools to help write tests for code that uses google-cloud services.

See TESTING to read more about using our testing helpers.

Versioning

This library follows Semantic Versioning, with some additional qualifications:

  1. Components marked with @BetaApi or @Experimental are considered to be "0.x" features inside a "1.x" library. This means they can change between minor and patch releases in incompatible ways. These features should not be used by any library "B" that itself has consumers, unless the components of library B that use @BetaApi features are also marked with @BetaApi. Features marked as @BetaApi are on a path to eventually become "1.x" features with the marker removed.

    Special exception for google-cloud-java: google-cloud-java is allowed to depend on @BetaApi features in gax-java without declaring the consuming code @BetaApi, because gax-java and google-cloud-java move in step with each other. For this reason, gax-java should not be used independently of google-cloud-java.

  2. Components marked with @InternalApi are technically public, but only because of the limitations of Java's access modifiers. For the purposes of semver, they should be considered private.

  3. Interfaces marked with @InternalExtensionOnly are public, but should only be implemented by internal classes. For the purposes of semver, we reserve the right to add to these interfaces without default implementations (for Java 7).

Please note these clients are currently under active development. Any release versioned 0.x.y is subject to backwards incompatible changes at any time.

Stable

Libraries defined at a Stable quality level are expected to be stable and all updates in the libraries are guaranteed to be backwards-compatible. Any backwards-incompatible changes will lead to the major version increment (1.x.y -> 2.0.0).

Preview

Libraries defined at a Preview quality level are still a work-in-progress and are more likely to get backwards-incompatible updates. Additionally, it's possible for Preview libraries to get deprecated and deleted before ever being promoted to Preview or Stable.

IDE Plugins

If you're using IntelliJ or Eclipse, you can add client libraries to your project using these IDE plugins:

Besides adding client libraries, the plugins provide additional functionality, such as service account key management. Refer to the documentation for each plugin for more details.

These client libraries can be used on App Engine standard for Java 8 runtime and App Engine flexible (including the Compat runtime). Most of the libraries do not work on the App Engine standard for Java 7 runtime. However, Datastore, Storage, and Bigquery should work.

Contributing

Contributions to this library are always welcome and highly encouraged.

See google-cloud's CONTRIBUTING documentation and the shared documentation for more information on how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

License

Apache 2.0 - See LICENSE for more information.