Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java Client: MessageImpl - ensure that AutoConsumeSchema downloaded the schema before decoding the payload #10248

Merged
merged 52 commits into from
Apr 19, 2021

Conversation

eolivelli
Copy link
Contributor

@eolivelli eolivelli commented Apr 16, 2021

Motivation

I saw this problem while developing integration tests for GenericObject + KeyValue with SEPARATED KeyValueEncoding.

Modifications

Before executing MessageImpl#getValue we are now ensuring that the Schema is really loaded when we call getSchemaInfo.

Initially I added an eager loading of the Schema in getSchemaInfo, but this resulted in lots of deadlocks because we are calling getSchemaInfo in many places.
The code also expects that AutoConsumeSchema returns getSchemaInfo == null in several places and we have to keep this behaviour.

Verifying this change

I added a new case to an existing integration test, the new case reproduced the problem.

@@ -78,27 +80,7 @@ public boolean supportSchemaVersioning() {

@Override
public GenericRecord decode(byte[] bytes, byte[] schemaVersion) {
if (schema == null) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have just moved this code into a separated method

new SinkSpec("test-kv-sink-input-kv-avro-json-inl-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class), KeyValueEncodingType.INLINE), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build())),
new SinkSpec("test-kv-sink-input-kv-avro-json-sep-" + randomName(8), "test-kv-sink-input-kv-string-int-" + randomName(8),
Schema.KeyValue(Schema.AVRO(PojoKey.class), Schema.JSON(Pojo.class), KeyValueEncodingType.SEPARATED), new KeyValue<>(PojoKey.builder().field1("a").build(), Pojo.builder().field1("a").field2(2).build()))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the new test case that covers the change

@eolivelli
Copy link
Contributor Author

@congbobo184 @codelipenghui now the patch is good to go. I added the test case

@linlinnn
Copy link
Contributor

the integration test PulsarGenericObjectSinkTest#testGenericObjectSink failed frequently, could you take a look?

@eolivelli
Copy link
Contributor Author

eolivelli commented Apr 17, 2021

@linlinnn
This is the fix for the problem IllegalRefCount
#10215

It is a problem related to the switch to LightProto

@eolivelli
Copy link
Contributor Author

also @linlinnn I have pushed now a change that changes that test.
before this last commit we submitted all of the sinks at once, now we will run only one sink at a time, hoping that this will require less CI resources

@eolivelli
Copy link
Contributor Author

@linlinnn @codelipenghui CI passed please take a look

@eolivelli eolivelli requested a review from sijie April 18, 2021 14:46
@codelipenghui
Copy link
Contributor

@congbobo184 Could you please also help review this PR?

@eolivelli
Copy link
Contributor Author

this patch also fixes a problem in the integration tests that made CI more flaky.
as soon as we commit this patch CI will be in better state

cc @congbobo184 @codelipenghui

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eolivelli eolivelli merged commit 54523bb into apache:master Apr 19, 2021
@eolivelli
Copy link
Contributor Author

thank you @congbobo184 @codelipenghui @lhotari for your review

eolivelli added a commit to datastax/pulsar that referenced this pull request May 12, 2021
…he schema before decoding the payload (apache#10248)

(cherry picked from commit 54523bb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants