[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

gaoran10 · 2021-02-19T13:30:43Z

Motivation

Currently, the Pulsar SQL query bytes schema data will cause an error.

Reproduce

produce bytes schema data.
query data by the Pulsar SQL.
the error log could be seen.

Error log

com.google.common.util.concurrent.UncheckedExecutionException: java.nio.BufferUnderflowException
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935)
	at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76)
	at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:471)
	at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_332__testversion____20210219_094906_2.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:509)
	at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:415)
	at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106)
	at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49)
	at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61)
	at org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
	... 18 more

Modifications

Add check for bytes schema, if the schema is bytes schema use the schema info of the bytes schema directly.

Verifying this change

Add a new integration test for different schemas.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

gaoran10 · 2021-02-21T15:00:23Z

/pulsarbot run-failure-checks

gaoran10 · 2021-02-21T16:35:43Z

/pulsarbot run-failure-checks

codelipenghui · 2021-02-21T23:03:41Z

pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java

@@ -441,9 +442,11 @@ public boolean advanceNextPosition() {
        //start time for deseralizing record
        metricsTracker.start_RECORD_DESERIALIZE_TIME();

-        SchemaInfo schemaInfo;
+        SchemaInfo schemaInfo = getBytesSchemaInfo(pulsarSplit.getSchemaType(), pulsarSplit.getSchemaName());


I noticed the current getSchemaVersion in the RawMessageImpl return null

@Override public byte[] getSchemaVersion() { if (msgMetadata != null && msgMetadata.getMetadata().hasSchemaVersion()) { return msgMetadata.getMetadata().getSchemaVersion(); } else { return null; } }

It should not throw java.nio.BufferUnderflowException. Could you please confirm here?

This method may cause the exception java.nio.BufferUnderflowException.

private SchemaInfo loadSchema(BytesSchemaVersion bytesSchemaVersion) throws PulsarAdminException { ClassLoader originalContextLoader = Thread.currentThread().getContextClassLoader(); try { Thread.currentThread().setContextClassLoader(InjectionManagerFactory.class.getClassLoader()); return pulsarAdmin.schemas() .getSchemaInfo(topicName.toString(), ByteBuffer.wrap(bytesSchemaVersion.get()).getLong()); } finally { Thread.currentThread().setContextClassLoader(originalContextLoader); } }

The schema version of Schema.BYTES is an empty bytes array. So ByteBuffer.wrap(bytesSchemaVersion.get()).getLong() will cause the exception java.nio.BufferUnderflowException.

sijie · 2021-02-19T17:57:12Z

pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java

+        if (!schemaType.equals(SchemaType.BYTES) && !schemaType.equals(SchemaType.NONE)) {
+            return null;
+        }
+        if (schemaName.equals(Schema.BYTES.getSchemaInfo().getName())) {


Can you compare SchemaType instead of comparing schemaName?

The schema types of the Schema.BYTES and Schema.BYTEBUFFER all are SchemaType.BYTES, so I use the schema name to make the comparison.

Okay. Make sense.

zymap · 2021-02-23T13:26:11Z

/pulsarbot run-failure-checks

gaoran10 · 2021-02-24T08:27:38Z

/pulsarbot run-failure-checks

gaoran10 · 2021-02-25T07:10:12Z

/pulsarbot run-failure-checks

zymap · 2021-03-01T09:44:26Z

This issue is introduced by #8422. So it not a bug fix for the 2.7.0. Removed the release/2.7.1 label.

codelipenghui assigned gaoran10 Feb 21, 2021

codelipenghui added this to the 2.8.0 milestone Feb 21, 2021

codelipenghui added area/sql Pulsar SQL related features type/bug The PR fixed a bug or issue reported a bug labels Feb 21, 2021

codelipenghui reviewed Feb 21, 2021

View reviewed changes

sijie reviewed Feb 22, 2021

View reviewed changes

codelipenghui added the release/2.7.1 label Feb 23, 2021

gaoran10 added 2 commits February 23, 2021 17:31

fix pulsar sql query bytes schema data error

8e24450

fix test

4633855

gaoran10 force-pushed the pulsar-sql-bytes-schema-fix branch from e73e177 to 4633855 Compare February 23, 2021 09:33

fix test

76f2f93

fix test

4df253d

codelipenghui approved these changes Feb 24, 2021

View reviewed changes

fix test

b83f681

gaoran10 added 3 commits February 24, 2021 19:20

fix test

1147380

fix test

0890d10

fix String schema validate

277fa8e

zymap approved these changes Feb 25, 2021

View reviewed changes

zymap merged commit 371b311 into apache:master Feb 25, 2021

gaoran10 deleted the pulsar-sql-bytes-schema-fix branch February 25, 2021 12:47

zymap removed the release/2.7.1 label Mar 1, 2021

Raven888888 mentioned this pull request Oct 6, 2021

Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1 #12284

Open

sijie mentioned this pull request Oct 6, 2021

ISSUE-12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1 streamnative/pulsar-archived#3131

Closed

sijie mentioned this pull request May 4, 2022

ISSUE-12284: Pulsar SQL BufferUnderflowException after upgrading from 2.7.0 to 2.8.1 streamnative/pulsar-archived#4166

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

gaoran10 commented Feb 19, 2021 •

edited

Loading

gaoran10 commented Feb 21, 2021

gaoran10 commented Feb 21, 2021

codelipenghui Feb 21, 2021

gaoran10 Feb 23, 2021 •

edited

Loading

codelipenghui Feb 24, 2021

sijie Feb 19, 2021

gaoran10 Feb 23, 2021 •

edited

Loading

sijie Feb 23, 2021

zymap commented Feb 23, 2021

gaoran10 commented Feb 24, 2021

gaoran10 commented Feb 25, 2021

zymap commented Mar 1, 2021

[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

[Pulsar SQL] Fix Pulsar SQL query bytes schema data error #9631

Conversation

gaoran10 commented Feb 19, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

gaoran10 commented Feb 21, 2021

gaoran10 commented Feb 21, 2021

codelipenghui Feb 21, 2021

Choose a reason for hiding this comment

gaoran10 Feb 23, 2021 • edited Loading

Choose a reason for hiding this comment

codelipenghui Feb 24, 2021

Choose a reason for hiding this comment

sijie Feb 19, 2021

Choose a reason for hiding this comment

gaoran10 Feb 23, 2021 • edited Loading

Choose a reason for hiding this comment

sijie Feb 23, 2021

Choose a reason for hiding this comment

zymap commented Feb 23, 2021

gaoran10 commented Feb 24, 2021

gaoran10 commented Feb 25, 2021

zymap commented Mar 1, 2021

gaoran10 commented Feb 19, 2021 •

edited

Loading

gaoran10 Feb 23, 2021 •

edited

Loading

gaoran10 Feb 23, 2021 •

edited

Loading