feat: Add consistency vector handling to CLI and Java client #8264

vpapavas · 2021-10-18T17:04:04Z

Description

Add consistency vector handling to the CLI and Java clients. The usage is hidden behind a config flag that is false by default. Both the client and the server need to enable the config. The client needs to both enable the config and add the consistency vector in the request properties.

The consistency vector is added to the queue that stores the query results after the query has been completed successfully. This design is not needed for http1 as we could pass the token back to the client using the headers of the StreamedRow. However, this is not possible to do in http2 (at least not without heavy refactoring) since the QueryResponseMetadata is sent back to the client before even the query executes. Moreover, the only way for the Publisher to communicate/send something to the Subscriber is through the queue. Hence, and in order to keep the design the same in both endpoints, the vector is added to the queue as the last item.

The vector is serialized before being written to the wire and the clients only "see" the serialized version. In practice, the CLI does not print the vector so it is hidden from the user.

Testing done

I created the ConsistencyOffsetVectorFunctionalTest that tests that the consistency vector is correctly sent back and forth and also not sent when the flag is not enabled.

I did manual tests for the CLI to make sure the vector is not printed but used.

With the following curl command, one can see the vector:

curl -X POST   \
http://localhost:8088/query \
 -H 'content-type: application/vnd.ksql.v1+json; charset=utf-8' \
 -d '{"ksql":"SELECT * FROM tab1 where id=1;", "streamsProperties": {
      "ksql.streams.auto.offset.reset": "earliest"},"requestProperties": {
      "ksql.query.pull.consistency.token.enabled":"true", "request.ksql.query.pull.consistency.token":""}
    }'

[{"header":{"queryId":"query_1635785940946","schema":"`ID` INTEGER KEY, `KSQL_COL_0` BIGINT, `KSQL_COL_1` INTEGER"}},
{"row":{"columns":[1,1,3]}},
{"consistencyToken":{"consistencyToken":"eyJ2ZXJzaW9uIjoyLCJvZmZzZXRWZWN0b3IiOnsiZHVtbXkiOnsiNSI6NSwiNiI6NiwiNyI6N319fQ=="}}]

There is no way to avoid this since the response contains the vector and there is no client where I could remove it as I do with the CLI.

Reviewer checklist

Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
Ensure relevant issues are linked (description should include text like "Fixes #")

patrickstuedi

Thanks for the PR @vpapavas ! Just added a few comments.

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

...api-client/src/test/java/io/confluent/ksql/api/client/integration/ClientIntegrationTest.java

ksqldb-common/src/main/java/io/confluent/ksql/util/ConsistencyToken.java

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

...api-client/src/test/java/io/confluent/ksql/api/client/integration/ClientIntegrationTest.java

ksqldb-rest-model/src/main/java/io/confluent/ksql/rest/entity/StreamedRow.java

ksqldb-engine/src/main/java/io/confluent/ksql/physical/pull/PullQueryRow.java

...-rest-app/src/main/java/io/confluent/ksql/api/server/DelimitedQueryStreamResponseWriter.java

ksqldb-rest-app/src/main/java/io/confluent/ksql/api/impl/BlockingQueryPublisher.java

ksqldb-engine/src/main/java/io/confluent/ksql/physical/pull/HARouting.java

vvcephei

Hey @vpapavas , thanks for this, and I'm sorry it took me so long to review it.

I found a couple of public API "leaks" that we might want to avoid until the feature is ready. Also, it seems like the HTTP/2 (aka query-stream) API changes are both risky and complicated. What if we just skip HTTP/2 for now and plan to add consistency as part of a future API revision?

Thanks,
-John

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/Row.java

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

...-api-client/src/main/java/io/confluent/ksql/api/client/impl/ExecuteQueryResponseHandler.java

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/util/JsonMapper.java

ksqldb-common/src/main/java/io/confluent/ksql/util/ConsistencyOffsetVector.java

ksqldb-engine/src/main/java/io/confluent/ksql/physical/pull/HARouting.java

...-rest-app/src/main/java/io/confluent/ksql/api/server/DelimitedQueryStreamResponseWriter.java

ksqldb-rest-model/src/main/java/io/confluent/ksql/rest/entity/StreamedRow.java

guozhangwang

Thanks @vpapavas !

One mega comment is whether relying on Java Serializable would be sufficient for any possible future evolution of the vector serde, if we ever want to remove any fields, for example. Otherwise it looks promising.

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/StreamedQueryResultImpl.java

...-api-client/src/main/java/io/confluent/ksql/api/client/impl/ExecuteQueryResponseHandler.java

ksqldb-common/src/main/java/io/confluent/ksql/util/ConsistencyOffsetVector.java

ksqldb-engine/src/main/java/io/confluent/ksql/physical/pull/HARouting.java

ksqldb-rest-client/src/main/java/io/confluent/ksql/rest/client/KsqlTarget.java

added consistency token to rest client and cli http2 not working added CT to queue rebased and tests pass rename consistencytoken javadoc added test for ws, made ct in client string rebase and fix conflicts merge different pr's address comments fixed test after rename fix test after rename address comments and fix

vvcephei

Thanks @vpapavas !

I suspect we might want to tweak some of the serialization stuff before the next release, but it seems like this PR has accomplished what we wanted from it.

patrickstuedi

Just leaving a few comments, nothing that's a showstopper for this PR.

patrickstuedi · 2021-11-02T12:40:13Z

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

+
+  @VisibleForTesting
+  public String getSerializedConsistencyVector() {
+    return serializedConsistencyVector.get();


Isn't the idea of VisibleForTesting that you can keep the method package-private?

It is not used in a test that is in the same package. The test is an integration test and uses functionality that is available in that specific package

...est/java/io/confluent/ksql/api/client/integration/ConsistencyOffsetVectorFunctionalTest.java

ksqldb-common/src/main/java/io/confluent/ksql/util/ConsistencyOffsetVector.java

AlanConfluent · 2021-11-03T17:10:10Z

...-api-client/src/main/java/io/confluent/ksql/api/client/impl/ExecuteQueryResponseHandler.java

+    if (json instanceof JsonObject) {
+      final JsonObject jsonObject = (JsonObject) json;
+      // This is the serialized consistency vector
+      // Don't add it to the result list since the user should not see it


We talked before about supporting an older client. It seems like the server will just return the consistency token if it's enabled on the server. We should only send it down if it's enabled on the client which can be done by indicating that it's enabled in the request property.

The consistency token will be included only if it is enabled both in the client and the server. The requirements are

Client side:

Set the server property KsqlConfig.KSQL_QUERY_PULL_CONSISTENCY_OFFSET_VECTOR_ENABLED to true and Include in the request properties

Add to the request properties the actual consistency token (may be an empty string the first time a request is made)

Server side:

The server config KsqlConfig.KSQL_QUERY_PULL_CONSISTENCY_OFFSET_VECTOR_ENABLED must be set to true

The request properties must contain a consistency token (may be an empty string the first time a request is made)

So, only if both the client and the server know and enable the use of consistency tokens, will it be sent back and forth

...est-app/src/main/java/io/confluent/ksql/rest/server/resources/streaming/WSQueryEndpoint.java

AlanConfluent · 2021-11-03T17:19:36Z

ksqldb-common/src/main/java/io/confluent/ksql/util/ConsistencyOffsetVector.java


+  public static boolean isConsistencyVectorEnabled(final Map<String, Object> requestProperties) {
+    final Object consistencyEnabled


Normally request properties are defined in KsqlRequestConfig. We don't usually send KsqlConfigs up as a request property, but rather a normal property.

I wanted to avoid creating two different configs for the same thing so reused the server config in the client as well. I can change it if it is too ugly not the right way to do it

AlanConfluent · 2021-11-03T17:26:08Z

...p/src/main/java/io/confluent/ksql/rest/server/resources/streaming/StreamedQueryResource.java

@@ -489,6 +491,18 @@ private EndpointResponse handleTablePullQuery(
      pullBandRateLimiter.allow(KsqlQueryType.PULL);

      final Optional<Decrementer> optionalDecrementer = Optional.ofNullable(decrementer);
+      Optional<ConsistencyOffsetVector> consistencyOffsetVector = Optional.empty();
+      if (ksqlConfig.getBoolean(KsqlConfig.KSQL_QUERY_PULL_CONSISTENCY_OFFSET_VECTOR_ENABLED)


Just to make sure I understand, if the client doesn't send up anything, the feature is not enabled, right?

There are four main cases:

The feature is disabled on the server

The feature is enabled on the server, but not the client

The feature is enabled on the server and the client, but the client has provided no consistency token

The feature is enabled on the server and client, and the client provides a consistency token for the pull query.

This seems to cover all cases, but I just want to be sure. 2 is covered because the client indicates that the feature is enabled but no consistency token is provided by using an empty string?

The client needs to set the config to true and then only will the consistency token be included in the request properties. So, if it is disabled on the client, no consistency token will be added to the request properties and the server won't do anything.
The empty string is used only when it is enabled and signals an empty, not yet initialized consistency token.

AlanConfluent · 2021-11-03T17:28:58Z

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

@@ -136,13 +153,19 @@ public BatchedQueryResult executeQuery(
      final String sql,
      final Map<String, Object> properties
  ) {
+    if (ConsistencyOffsetVector.isConsistencyVectorEnabled(requestProperties)) {


Are you meaning to check this using properties rather than requestProperties since it's defined in KsqlConfig?

Added to properties

AlanConfluent · 2021-11-03T17:31:13Z

ksqldb-api-client/src/main/java/io/confluent/ksql/api/client/impl/ClientImpl.java

+    if (ConsistencyOffsetVector.isConsistencyVectorEnabled(requestProperties)) {
+      requestProperties.put(
+          KsqlRequestConfig.KSQL_REQUEST_QUERY_PULL_CONSISTENCY_OFFSET_VECTOR,
+          serializedConsistencyVector.get());


Can this be null rather than empty string if it's been unset?

Yeah, I had it as null before but the code for JsonObject.put throws an NPE

AlanConfluent

We talked about things offline. A couple of those smaller things would be great to get in this PR, but if not a followup is good too.

vpapavas requested review from agavra and removed request for agavra October 18, 2021 17:24

patrickstuedi reviewed Oct 19, 2021

View reviewed changes

AlanConfluent reviewed Oct 20, 2021

View reviewed changes

vpapavas force-pushed the consistency-api branch from eb9cffd to 612fd50 Compare October 22, 2021 10:03

vpapavas commented Oct 22, 2021

View reviewed changes

ksqldb-engine/src/main/java/io/confluent/ksql/physical/pull/HARouting.java Outdated Show resolved Hide resolved

vpapavas marked this pull request as ready for review October 22, 2021 14:06

vpapavas requested a review from a team as a code owner October 22, 2021 14:06

vpapavas changed the title ~~feat: Add consistency token handling to CLI and Java client~~ feat: Add consistency vector handling to CLI and Java client Oct 22, 2021

vvcephei reviewed Oct 25, 2021

View reviewed changes

vpapavas force-pushed the consistency-api branch from d6544f2 to 2d98dc7 Compare October 26, 2021 20:37

guozhangwang reviewed Oct 27, 2021

View reviewed changes

vpapavas added 2 commits November 1, 2021 17:31

clean tests, address comments

c6f25ae

vpapavas force-pushed the consistency-api branch from 111c2b3 to c6f25ae Compare November 1, 2021 17:32

vpapavas added 2 commits November 1, 2021 20:03

supress warning

3052eeb

test

586bb75

vvcephei approved these changes Nov 2, 2021

View reviewed changes

vpapavas added 3 commits November 2, 2021 11:16

test and cleaning

3899f6a

blank lines

afb009e

minor

08e20fa

patrickstuedi reviewed Nov 2, 2021

View reviewed changes

vpapavas added 2 commits November 2, 2021 18:20

address comments

7770f61

minor

cc002b5

patrickstuedi approved these changes Nov 3, 2021

View reviewed changes

patrickstuedi mentioned this pull request Nov 3, 2021

chore: passing consistency token using request properties in HARouting #8312

Merged

AlanConfluent reviewed Nov 3, 2021

View reviewed changes

AlanConfluent approved these changes Nov 3, 2021

View reviewed changes

address comments

b318bf3

vpapavas merged commit a651677 into confluentinc:master Nov 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add consistency vector handling to CLI and Java client #8264

feat: Add consistency vector handling to CLI and Java client #8264

vpapavas commented Oct 18, 2021 •

edited

Loading

patrickstuedi left a comment

vvcephei left a comment

guozhangwang left a comment

vvcephei left a comment

patrickstuedi left a comment

patrickstuedi Nov 2, 2021

vpapavas Nov 2, 2021

AlanConfluent Nov 3, 2021

vpapavas Nov 3, 2021

AlanConfluent Nov 3, 2021

vpapavas Nov 3, 2021

AlanConfluent Nov 3, 2021

vpapavas Nov 3, 2021

AlanConfluent Nov 3, 2021

vpapavas Nov 4, 2021

AlanConfluent Nov 3, 2021

vpapavas Nov 3, 2021

AlanConfluent left a comment


		public static boolean isConsistencyVectorEnabled(final Map<String, Object> requestProperties) {
		final Object consistencyEnabled

feat: Add consistency vector handling to CLI and Java client #8264

feat: Add consistency vector handling to CLI and Java client #8264

Conversation

vpapavas commented Oct 18, 2021 • edited Loading

Description

Testing done

Reviewer checklist

patrickstuedi left a comment

Choose a reason for hiding this comment

vvcephei left a comment

Choose a reason for hiding this comment

guozhangwang left a comment

Choose a reason for hiding this comment

vvcephei left a comment

Choose a reason for hiding this comment

patrickstuedi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlanConfluent left a comment

Choose a reason for hiding this comment

vpapavas commented Oct 18, 2021 •

edited

Loading