You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow [the instruction](https://cloud.google.com/pubsub/lite/docs/quickstart#create_a_lite_subscription) to create a new subscription or use an existing subscription. If using an existing subscription, the connector will read from the oldest unacknowledged message in the subscription.
7
7
8
-
### Create a new subscription or use existing subscription
8
+
### Creating a Google Cloud Dataproc cluster (Optional)
9
9
10
-
Follow [the instruction](https://cloud.google.com/pubsub/lite/docs/quickstart#create_a_lite_subscription) to create a new
11
-
subscription or use existing subscription. If using existing subscription, the connector will read message from the
12
-
oldest unacknowledged.
10
+
If you do not have an Apache Spark environment, you can create a [Cloud Dataproc](https://cloud.google.com/dataproc/docs) cluster with pre-configured auth. The following examples assume you are using Cloud Dataproc, but you can use `spark-submit` on any cluster.
13
11
14
-
### Create a Google Cloud Dataproc cluster (Optional)
15
-
16
-
If you do not have an Apache Spark environment you can create a Cloud Dataproc cluster with pre-configured auth. The following examples assume you are using Cloud Dataproc, but you can use `spark-submit` on any cluster.
17
-
18
-
```
19
-
MY_CLUSTER=...
20
-
gcloud dataproc clusters create "$MY_CLUSTER"
21
-
```
12
+
```
13
+
MY_CLUSTER=...
14
+
gcloud dataproc clusters create "$MY_CLUSTER"
15
+
```
22
16
23
17
## Downloading and Using the Connector
24
18
25
19
<!--- TODO(jiangmichael): Add jar link for spark-pubsublite-latest.jar -->
26
-
The latest version connector of the connector (Scala 2.11) is publicly available in
The latest version connector of the connector (Scala 2.11) will be publicly available in `gs://spark-lib/pubsublite/spark-pubsublite-latest.jar`.
28
21
29
22
<!--- TODO(jiangmichael): Release on Maven Central and add Maven Central link -->
30
-
The connector is also available from the Maven Central
31
-
repository. It can be used using the `--packages` option or the
32
-
`spark.jars.packages` configuration property. Use the following value
23
+
The connector will also be available from the Maven Central repository. It can be used using the `--packages` option or the `spark.jars.packages` configuration property.
Note that the connector supports both MicroBatch Processing and [Continuous Processing](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#continuous-processing).
0 commit comments