Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix pulsar sql issues when run select count(*) for the table with primary schema #10840

Merged

Conversation

codelipenghui
Copy link
Contributor

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Motivation

Fix pulsar sql issues when run select count(*) for the table with primary schema

error log:

2021-06-06T14:36:41.669+0800 ERROR SplitRunner-3-106 io.prestosql.execution.executor.TaskExecutor Error processing Split 20210606_063634_00002_6j8bm.1.0-1 PulsarSplit{splitId=1, connectorId='pulsar', originSchemaName='Bytes', schemaName='public/default', tableName='my-topic', splitSize=39324, schema='', schemaType=BYTES, startPositionEntryId=39325, endPositionEntryId=78649, startPositionLedgerId=10, endPositionLedgerId=10, schemaInfoProperties={}} (start = 2.24098441109938E8, wall = 320 ms, cpu = 0 ms, wait = 1 ms, calls = 1)
java.lang.RuntimeException: Primitive type must has only one ColumnHandle.
 at org.apache.pulsar.sql.presto.decoder.primitive.PulsarPrimitiveRowDecoderFactory.createRowDecoder(PulsarPrimitiveRowDecoderFactory.java:67)
 at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createRowDecoder(PulsarDispatchingRowDecoderFactory.java:59)
 at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:545)
 at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
 at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
 at io.prestosql.operator.Driver.processInternal(Driver.java:379)
 at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
 at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
 at io.prestosql.operator.Driver.processFor(Driver.java:276)
 at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
 at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
 at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
 at io.prestosql.$gen.Presto_332__testversion____20210606_063438_2.run(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)

Modifications

Remove check for the column size when on the primary schema since when run select count(*) on the presto, we will get empty column handlers

Verifying this change

New integration test added.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

…mary schema

error log:

```
2021-06-06T14:36:41.669+0800 ERROR SplitRunner-3-106 io.prestosql.execution.executor.TaskExecutor Error processing Split 20210606_063634_00002_6j8bm.1.0-1 PulsarSplit{splitId=1, connectorId='pulsar', originSchemaName='Bytes', schemaName='public/default', tableName='my-topic', splitSize=39324, schema='', schemaType=BYTES, startPositionEntryId=39325, endPositionEntryId=78649, startPositionLedgerId=10, endPositionLedgerId=10, schemaInfoProperties={}} (start = 2.24098441109938E8, wall = 320 ms, cpu = 0 ms, wait = 1 ms, calls = 1)
java.lang.RuntimeException: Primitive type must has only one ColumnHandle.
 at org.apache.pulsar.sql.presto.decoder.primitive.PulsarPrimitiveRowDecoderFactory.createRowDecoder(PulsarPrimitiveRowDecoderFactory.java:67)
 at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createRowDecoder(PulsarDispatchingRowDecoderFactory.java:59)
 at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:545)
 at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
 at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
 at io.prestosql.operator.Driver.processInternal(Driver.java:379)
 at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
 at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
 at io.prestosql.operator.Driver.processFor(Driver.java:276)
 at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
 at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
 at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
 at io.prestosql.$gen.Presto_332__testversion____20210606_063438_2.run(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)
 ```
@codelipenghui codelipenghui self-assigned this Jun 6, 2021
@codelipenghui codelipenghui added this to the 2.8.0 milestone Jun 6, 2021
@codelipenghui codelipenghui added the release/blocker Indicate the PR or issue that should block the release until it gets resolved label Jun 6, 2021
Copy link
Contributor

@gaoran10 gaoran10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui added the area/sql Pulsar SQL related features label Jun 6, 2021
Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui merged commit 4a1076a into apache:master Jun 6, 2021
@codelipenghui codelipenghui deleted the penghui/fix-sql-count-star branch June 6, 2021 09:40
yangl pushed a commit to yangl/pulsar that referenced this pull request Jun 23, 2021
…mary schema (apache#10840)

error log:

```
2021-06-06T14:36:41.669+0800 ERROR SplitRunner-3-106 io.prestosql.execution.executor.TaskExecutor Error processing Split 20210606_063634_00002_6j8bm.1.0-1 PulsarSplit{splitId=1, connectorId='pulsar', originSchemaName='Bytes', schemaName='public/default', tableName='my-topic', splitSize=39324, schema='', schemaType=BYTES, startPositionEntryId=39325, endPositionEntryId=78649, startPositionLedgerId=10, endPositionLedgerId=10, schemaInfoProperties={}} (start = 2.24098441109938E8, wall = 320 ms, cpu = 0 ms, wait = 1 ms, calls = 1)
java.lang.RuntimeException: Primitive type must has only one ColumnHandle.
 at org.apache.pulsar.sql.presto.decoder.primitive.PulsarPrimitiveRowDecoderFactory.createRowDecoder(PulsarPrimitiveRowDecoderFactory.java:67)
 at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createRowDecoder(PulsarDispatchingRowDecoderFactory.java:59)
 at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:545)
 at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
 at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
 at io.prestosql.operator.Driver.processInternal(Driver.java:379)
 at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
 at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
 at io.prestosql.operator.Driver.processFor(Driver.java:276)
 at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
 at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
 at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
 at io.prestosql.$gen.Presto_332__testversion____20210606_063438_2.run(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)
 ```
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
…mary schema (apache#10840)

error log:

```
2021-06-06T14:36:41.669+0800 ERROR SplitRunner-3-106 io.prestosql.execution.executor.TaskExecutor Error processing Split 20210606_063634_00002_6j8bm.1.0-1 PulsarSplit{splitId=1, connectorId='pulsar', originSchemaName='Bytes', schemaName='public/default', tableName='my-topic', splitSize=39324, schema='', schemaType=BYTES, startPositionEntryId=39325, endPositionEntryId=78649, startPositionLedgerId=10, endPositionLedgerId=10, schemaInfoProperties={}} (start = 2.24098441109938E8, wall = 320 ms, cpu = 0 ms, wait = 1 ms, calls = 1)
java.lang.RuntimeException: Primitive type must has only one ColumnHandle.
 at org.apache.pulsar.sql.presto.decoder.primitive.PulsarPrimitiveRowDecoderFactory.createRowDecoder(PulsarPrimitiveRowDecoderFactory.java:67)
 at org.apache.pulsar.sql.presto.PulsarDispatchingRowDecoderFactory.createRowDecoder(PulsarDispatchingRowDecoderFactory.java:59)
 at org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:545)
 at io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90)
 at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
 at io.prestosql.operator.Driver.processInternal(Driver.java:379)
 at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
 at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
 at io.prestosql.operator.Driver.processFor(Driver.java:276)
 at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
 at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
 at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
 at io.prestosql.$gen.Presto_332__testversion____20210606_063438_2.run(Unknown Source)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)
 ```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sql Pulsar SQL related features release/blocker Indicate the PR or issue that should block the release until it gets resolved
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants