You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: .readme-partials.yaml
+57-3Lines changed: 57 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -28,20 +28,56 @@ custom_content: |
28
28
29
29
## Usage
30
30
31
+
### Samples
32
+
33
+
There are 3 java samples (word count, simple write, simple read) under [samples](https://github.com/googleapis/java-pubsublite-spark/tree/master/samples) that shows using the connector inside Dataproc.
Note that the connector supports both MicroBatch Processing and [Continuous Processing](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#continuous-processing).
| pubsublite.topic | String | Y | | Full topic path that the connector will write to. |
95
+
| gcp.credentials.key | String | N | [Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) | Service account JSON in base64. |
96
+
54
97
### Data Schema
55
98
56
-
The connector has fixed data schema as follows:
99
+
When reading from Pub/Sub Lite, the connector has a fixed data schema as follows:
57
100
58
101
| Data Field | Spark Data Type | Notes |
59
102
| ---------- | --------------- | ----- |
@@ -66,6 +109,17 @@ custom_content: |
66
109
| publish_timestamp | TimestampType | |
67
110
| event_timestamp | TimestampType | Nullable |
68
111
112
+
When writing to Pub/Sub Lite, the connetor matches the following data field and data types as follows:
113
+
114
+
| Data Field | Spark Data Type | Required |
115
+
| ---------- | --------------- | ----- |
116
+
| key | BinaryType | N |
117
+
| data | BinaryType | N |
118
+
| attributes | MapType\[StringType, ArrayType\[BinaryType\]\] | N |
119
+
| event_timestamp | TimestampType | N |
120
+
121
+
Note that when a data field is present in the table but the data type mismatches, the connector will throw IllegalArgumentException that terminates the query.
122
+
69
123
## Building the Connector
70
124
71
125
The connector is built using Maven. Following command creates a JAR file with shaded dependencies:
0 commit comments