Skip to content

Java libraries and HBase client extensions for accessing Google Cloud Bigtable

License

Notifications You must be signed in to change notification settings

ksatwara/java-bigtable-hbase

 
 

Repository files navigation

Travis CI status Maven Stack Overflow

Google Cloud Bigtable is Google's NoSQL Big Data database service. It's the same database that powers many core Google services, including Search, Analytics, Maps, and Gmail.

Bigtable is designed to handle massive workloads at consistent low latency and high throughput, so it's a great choice for both operational and analytical applications, including IoT, user analytics, and financial data analysis.

Bigtable provisions and scales to hundreds of petabytes automatically, and can smoothly handle millions of operations per second. Changes to the deployment configuration are immediate, so there is no downtime during reconfiguration.

Bigtable integrates easily with popular Big Data tools like Hadoop, as well as Google Cloud Platform products like Cloud Dataflow and Dataproc. Plus, Bigtable supports the open-source, industry-standard HBase API, which makes it easy for development teams to get started.

Note: Please use google-cloud-bigtable to access Bigtable APIs instead of bigtable-core-client. These artifacts are meant to wrap HBase over Bigtable API.

Project setup, installation, and configuration

Prerequisites

Using the Java client

  • Add the appropriate Cloud Bigtable artifact dependencies to your Maven project.
    • bigtable-hbase-1.x: use for standalone applications where you are in control of your dependencies.
    • bigtable-hbase-1.x-hadoop: use in hadoop environments.
    • bigtable-hbase-1.x-mapreduce: use for map/reduce utilities.
    • bigtable-hbase-1.x-shaded: use in environments (other than hadoop) that require older versions of protobuf, guava, etc.
    • bigtable-hbase-2.x: use for standalone applications where you are in control of your dependencies. This includes an HBase async client.
    • bigtable-hbase-2.x-hadoop: use in hadoop environments.
    • bigtable-hbase-2.x-shaded: use in environments (other than hadoop) that require older versions of protobuf, guava, etc.

Maven:

<dependency>
  <groupId>com.google.cloud.bigtable</groupId>
  <artifactId>bigtable-hbase-1.x</artifactId>
  <version>1.14.1</version>
</dependency>

Gradle:

compile 'com.google.cloud.bigtable:bigtable-hbase-1.x:1.14.1'

SBT:

libraryDependencies += "com.google.cloud.bigtable" % "bigtable-hbase-1.x" % "1.14.1"

Questions and discussions

If you have questions or run into issues with Google Cloud Bigtable or the client libraries, use any of the following forums:

You can also subscribe to google-cloud-bigtable-announce@ list to receive infrequent product and client library announcements.

Developing and testing

  1. Run mvn clean install to build and install Cloud Bigtable client artifacts to your local repository, and then run the unit tests.

  2. There is also a suite of integration tests that connect to a cluster you have access to, via the authentication credentials that were loaded during the Google Cloud SDK configuration step (see above).

    Use the following command to run the Cloud Bigtable integration tests for HBase 1:

    mvn clean verify \
        -PbigtableIntegrationTest \
        -Dgoogle.bigtable.project.id=[your cloud project id] \
        -Dgoogle.bigtable.instance.id=[your cloud bigtable instance id]

    Use the following command to run the Cloud Bigtable integration tests for HBase 2:

    mvn clean verify \
        -PbigtableIntegrationTestH2 \
        -Dgoogle.bigtable.project.id=[your cloud project id] \
        -Dgoogle.bigtable.instance.id=[your cloud bigtable instance id]

    There are also tests that perform compatibility tests against an HBase Minicluster, which can be invoked with the following commands for HBase 1 and HBase 2 respectively:

    mvn clean verify -PhbaseLocalMiniClusterTest
    mvn clean verify -PhbaseLocalMiniClusterTestH2

    You can run those commands at the top of the project, or you can run them at the appropriate integration-tests project.

    Developer's NOTE: You can build the project faster by running the following command, and then run the integration test command from the appropriate integration test directory:

    mvn -pl bigtable-hbase-1.x-parent/bigtable-hbase-1.x-integration-tests \
    -pl bigtable-hbase-2.x-parent/bigtable-hbase-2.x-integration-tests \
    -am clean install
  3. Run mvn -X com.coveo:fmt-maven-plugin:format to check and enforce consistent java formatting to all of your code changes.

  4. You can use google-cloud-bigtable-emulator for local testing. The minimum configuration requires to connect the emulator with bigtable-core-client OR hbase-client:

    @Rule
    public final BigtableEmulatorRule bigtableEmulator = BigtableEmulatorRule.create();
    private BigtableSession session;
    private Connection connection;
    
    @Setup
    public void setup() {
      // for bigtable-core
      BigtableOptions opts = BigtableOptions.builder()
            .enableEmulator("localhost:" + bigtableEmulator.getPort())
            .setUserAgent("fake")
            .setProjectId("fakeproject")
            .setInstanceId("fakeinstance")
            .build();
      this.session = new BigtableSession(opts);
    
      // for hbase
      Configuration config = BigtableConfiguration.configure("fakeproject", "fakeinstance");
      config.set("google.bigtable.emulator.endpoint.host", "localhost:" + bigtableEmulator.getPort());
      this.connection = BigtableConfiguration.connect(config);
    }

NOTE: This project uses extensive shading which IDEs have trouble with. To overcome these issues, disable the with-shaded profile in your IDE to force it to resolve the dependencies from your local maven repository. When you disable that profile, attach workspace sources to the local maven repository jars.

Clients and Repositories

Name Language Repository latest version status
Cloud Bigtable Examples Java, others GoogleCloudPlatform/cloud-bigtable-examples
HBase client Java googleapis/cloud-bigtable-client Maven GA
Cloud Bigtable GoLang Go googleapis/google-cloud-go N/A GA
Cloud Bigtable Java Java googleapis/java-bigtable Maven GA
Cloud Bigtable Python Python googleapis/google-cloud-python PyPI version Beta
Cloud Bigtable Node.js Node googleaps/nodejs-bigtable npm version GA
Cloud Bigtable C# C# googleapis/google-cloud-dotnet NuGet version GA
Cloud Bigtable C++ C++ googleapis/google-cloud-cpp 0.9.0 Beta
Cloud Bigtable Ruby Ruby googleapis/google-cloud-ruby Gem Version Beta
Cloud Bigtable PHP PHP googleapis/google-cloud-php Latest Stable Version Beta

Contributing changes

See CONTRIBUTING.md for more information on how to contribute to this project.

License

Apache 2.0; see LICENSE for details.

About

Java libraries and HBase client extensions for accessing Google Cloud Bigtable

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 99.4%
  • Other 0.6%