Skip to content

Commit

Permalink
[bq] 0.1 BigQuery preparation
Browse files Browse the repository at this point in the history
  • Loading branch information
dgray16 committed Jun 13, 2021
1 parent e63fe4a commit e1d5fc5
Show file tree
Hide file tree
Showing 5 changed files with 182 additions and 162 deletions.
38 changes: 38 additions & 0 deletions spring-batch-bigquery/README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# spring-batch-bigquery

Spring Batch extension which contains an `ItemWriter` implementation for https://cloud.google.com/bigquery[BigQuery] based on https://github.com/googleapis/java-bigquery[Java BigQuery]. It supports writing https://en.wikipedia.org/wiki/Comma-separated_values[CSV], https://en.wikipedia.org/wiki/JSON[JSON] using https://cloud.google.com/bigquery/docs/batch-loading-data[load jobs].

## Configuration of `BigQueryItemWriter`

Next to the https://docs.spring.io/spring-batch/reference/html/configureJob.html[configuration of Spring Batch] one needs to configure the `BigQueryItemWriter`.

```java
@Bean
BigQueryItemWriter<MyDto> bigQueryCsvWriter() {
WriteChannelConfiguration writeConfiguration = WriteChannelConfiguration
.newBuilder(TableId.of("csv_dataset", "csv_table"))
.setAutodetect(true)
.setFormatOptions(FormatOptions.csv())
.build();

BigQueryItemWriter<MyDto> writer = new BigQueryItemWriterBuilder<MyDto>()
.bigQuery(mockedBigQuery)
.writeChannelConfig(writeConfiguration)
.build();
}
```

Additional examples could be found in https://github.com/spring-projects/spring-batch-extensions/blob/main/spring-batch-bigquery/src/test/java/org/springframework/batch/extensions/bigquery/builder/BigQueryItemWriterBuilderTests.java[here].

## Configuration properties
[cols="1,1,4"]
.Properties for item writer
|===
| Property | Required | Description

| `bigQuery` | yes | BigQuery object that provided by BigQuery Java Library. Responsible for connection with BigQuery.
| `writeChannelConfig` | yes | BigQuery write channel config provided by BigQuery Java Library. Responsible for configuring data type, data channel, jobs that will be sent to BigQuery.
| `rowMapper` | no | Your own converter that specifies how to convert input CSV / JSON to byte array.
| `datasetInfo` | no | Your way to customize to how to create BigQuery dataset.
| `jobConsumer` | no | Your custom handler for BigQuery Job provided by BigQuery Java Library.
|===
45 changes: 27 additions & 18 deletions spring-batch-bigquery/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,26 @@
<artifactId>spring-batch-bigquery</artifactId>
<version>0.1.0-SNAPSHOT</version>

<developers>
<developer>
<id>Dgray16</id>
<name>Vova Perebykivskyi</name>
<email>vova235@gmail.com</email>
</developer>
</developers>

<licenses>
<license>
<name>Apache 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>

<scm>
<url>https://github.com/spring-projects/spring-batch-extensions</url>
</scm>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

Expand All @@ -36,23 +56,18 @@
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>4.3.2</version>
<version>4.3.3</version>
</dependency>

<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigquery</artifactId>
<version>1.128.0</version>
<version>1.133.0</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.3.0</version>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-csv</artifactId>
<version>2.12.3</version>
</dependency>

<dependency>
Expand All @@ -71,19 +86,13 @@
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.7.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-csv</artifactId>
<version>2.12.3</version>
<version>5.7.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.9.0</version>
<version>3.11.1</version>
<scope>test</scope>
</dependency>

Expand Down
Loading

0 comments on commit e1d5fc5

Please sign in to comment.