Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(sample): Add sample for native image support in Bigquery #1829

Merged
merged 4 commits into from
Feb 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/java-bigquery/tree

| Sample | Source Code | Try it |
| --------------------------- | --------------------------------- | ------ |
| Native Image Bigquery Sample | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/native-image-sample/src/main/java/com/example/bigquery/NativeImageBigquerySample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/native-image-sample/src/main/java/com/example/bigquery/NativeImageBigquerySample.java) |
| Add Column Load Append | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AddColumnLoadAppend.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/AddColumnLoadAppend.java) |
| Add Empty Column | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AddEmptyColumn.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/AddEmptyColumn.java) |
| Auth Drive Scope | [source code](https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AuthDriveScope.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-bigquery&page=editor&open_in_editor=samples/snippets/src/main/java/com/example/bigquery/AuthDriveScope.java) |
Expand Down
47 changes: 47 additions & 0 deletions samples/native-image-sample/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# BigQuery Sample Application with Native Image

The BigQuery sample application demonstrates some common operations with [Google Cloud BigQuery](https://cloud.google.com/bigquery) and is compatible with Native Image compilation.

## Setup Instructions

1. Follow the [GCP Project Authentication and Native Image Setup Instructions](../../README.md).

2. [Enable the BigQuery APIs](https://console.cloud.google.com/apis/api/bigquery.googleapis.com).

### Run with Native Image Support

Navigate to this directory in a new terminal.

1. Compile the application using the Native Image Compiler. This step may take a few minutes.

```
mvn package -P native -DskipTests
```

2. Run the application:

```
./target/native-image-sample
```

3. The application will create a sample BigQuery dataset in your GCP project called `nativeimage_test_dataset` and perform some simple operations like creating a table, inserting data, and running a query.

If you would like to delete the BigQuery dataset later, you can manage your BigQuery resources through [Google Cloud Console](https://console.cloud.google.com/bigquery) to clean up BigQuery resources under your project.

When you run the application, you'll see output like this in the terminal:

```
Created new table: nativeimage_test_table_2351b0891d2f48af9309bd289c3bad13
Successfully inserted test row.
Queried the following records:
User id: TestUser-2f39e3ec-d81a-483f-9ec0-b9bd54155710 | age: 40
Deleted table: nativeimage_test_table_2351b0891d2f48af9309bd289c3bad13
```

### Sample Integration test with Native Image Support

In order to run the sample integration test, call the following command:

```
mvn test -Pnative
```
161 changes: 161 additions & 0 deletions samples/native-image-sample/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
<?xml version='1.0' encoding='UTF-8'?>
<!--
~ Copyright 2022 Google LLC
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.bigquery</groupId>
<artifactId>native-image-sample</artifactId>
<name>Native Image Sample</name>

<parent>
<groupId>com.google.cloud.samples</groupId>
<artifactId>shared-configuration</artifactId>
<version>1.2.0</version>
</parent>

<properties>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>24.2.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigquery</artifactId>
</dependency>

<!-- Test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>1.1.3</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.example.bigquery.NativeImageBigquerySample
</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>

<!-- Native Profile-->
<profiles>
<profile>
<id>native</id>

<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>native-image-support</artifactId>
<version>0.10.0</version>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>junit-platform-native</artifactId>
<version>0.9.9</version>
<scope>test</scope>
</dependency>

</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin
</artifactId> <!-- Must use older version of surefire plugin for native-image testing. -->
<version>2.22.2</version>
<configuration>
<includes>
<include>**/*IT</include>
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<version>0.9.9</version>
<extensions>true</extensions>
<configuration>
<mainClass>com.example.bigquery.NativeImageBigquerySample
</mainClass>
<buildArgs>
<buildArg>--no-fallback</buildArg>
<buildArg>--no-server</buildArg>
</buildArgs>
</configuration>
<executions>
<execution>
<id>build-native</id>
<goals>
<goal>build</goal>
<goal>test</goal>
</goals>
<phase>package</phase>
</execution>
<execution>
<id>test-native</id>
<goals>
<goal>test</goal>
</goals>
<phase>test</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
/*
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.example.bigquery;

import com.google.api.gax.paging.Page;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryError;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Dataset;
import com.google.cloud.bigquery.DatasetInfo;
import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.FieldValueList;
import com.google.cloud.bigquery.InsertAllRequest;
import com.google.cloud.bigquery.InsertAllResponse;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.Schema;
import com.google.cloud.bigquery.StandardSQLTypeName;
import com.google.cloud.bigquery.StandardTableDefinition;
import com.google.cloud.bigquery.Table;
import com.google.cloud.bigquery.TableDefinition;
import com.google.cloud.bigquery.TableId;
import com.google.cloud.bigquery.TableInfo;
import com.google.cloud.bigquery.TableResult;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;

/**
* Sample application demonstrating BigQuery operations.
*
* <p>Note: This application will create a BigQuery dataset in your GCP project. You can delete this
* by viewing BigQuery in Cloud Console https://console.cloud.google.com/bigquery or by uncommenting
* the call to `deleteDataset(..)` made in main().
*/
public class NativeImageBigquerySample {

private static final String DATASET_ID = "nativeimage_test_dataset";

private static final String TABLE_ID = "nativeimage_test_table";

private static final Schema TABLE_SCHEMA =
Schema.of(
Field.of("id", StandardSQLTypeName.STRING), Field.of("age", StandardSQLTypeName.INT64));

/** Entrypoint to the application. */
public static void main(String[] args) throws InterruptedException {
BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();

if (!hasDataset(bigQuery, DATASET_ID)) {
createDataset(bigQuery, DATASET_ID);
}

String tableName = TABLE_ID + "_" + UUID.randomUUID().toString().replace("-", "");
createTable(bigQuery, DATASET_ID, tableName, TABLE_SCHEMA);
String testId = "TestUser-" + UUID.randomUUID().toString();
int testAge = 40;
insertTestRecord(bigQuery, DATASET_ID, tableName, testId, testAge);
queryTable(bigQuery, DATASET_ID, tableName);

// Clean up resources.
deleteTable(bigQuery, DATASET_ID, tableName);

// Uncomment this to delete the created dataset.
// deleteDataset(bigQuery, DATASET_ID);
}

static String queryTable(BigQuery bigQuery, String datasetName, String tableName)
throws InterruptedException {
String fullyQualifiedTable = datasetName + "." + tableName;
String query = "SELECT * FROM " + fullyQualifiedTable;

QueryJobConfiguration queryConfig = QueryJobConfiguration.newBuilder(query).build();
TableResult results = bigQuery.query(queryConfig);

String result = "";
System.out.println("Queried the following records: ");
for (FieldValueList row : results.iterateAll()) {
String rowStatement =
String.format(
"User id: %s | age: %d\n",
row.get("id").getStringValue(), row.get("age").getLongValue());
result += rowStatement;
System.out.println(row);
}
return result;
}

static void insertTestRecord(
BigQuery bigQuery, String datasetName, String tableName, String id, int age) {

Map<String, Object> rowContent = new HashMap<>();
rowContent.put("id", id);
rowContent.put("age", age);

InsertAllRequest request =
InsertAllRequest.newBuilder(datasetName, tableName).addRow(rowContent).build();

InsertAllResponse response = bigQuery.insertAll(request);

if (response.hasErrors()) {
System.out.println("Insert resulted in errors:");
for (Map.Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
System.out.println("Response error: \n" + entry.getValue());
}
} else {
System.out.println("Successfully inserted test row.");
}
}

static void createTable(BigQuery bigQuery, String datasetName, String tableName, Schema schema) {

TableId tableId = TableId.of(datasetName, tableName);
TableDefinition tableDefinition = StandardTableDefinition.of(schema);
TableInfo tableInfo = TableInfo.newBuilder(tableId, tableDefinition).build();
bigQuery.create(tableInfo);
System.out.println("Created new table: " + tableName);
}

static boolean hasTable(BigQuery bigQuery, String datasetName, String tableName) {

Page<Table> tables = bigQuery.listTables(datasetName);
for (Table table : tables.iterateAll()) {
if (tableName.equals(table.getTableId().getTable())) {
return true;
}
}
return false;
}

static void createDataset(BigQuery bigQuery, String datasetName) {
DatasetInfo datasetInfo = DatasetInfo.newBuilder(datasetName).build();
Dataset newDataset = bigQuery.create(datasetInfo);
System.out.println("Created new dataset: " + newDataset.getDatasetId().getDataset());
}

static boolean hasDataset(BigQuery bigQuery, String datasetName) {
Page<Dataset> datasets = bigQuery.listDatasets();
for (Dataset dataset : datasets.iterateAll()) {
if (datasetName.equals(dataset.getDatasetId().getDataset())) {
return true;
}
}
return false;
}

static void deleteTable(BigQuery bigQuery, String datasetName, String tableName) {
bigQuery.getTable(datasetName, tableName).delete();
System.out.println("Deleted table: " + tableName);
}

static void deleteDataset(BigQuery bigQuery, String datasetName) {
bigQuery.getDataset(datasetName).delete();
System.out.println("Deleting dataset " + datasetName);
}
}