Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate key __pfn_input_topic__ in presto server #9686

Merged

Conversation

pointearth
Copy link
Contributor

@pointearth pointearth commented Feb 23, 2021

Motivation

When I query a topic with presto
`presto> select * from xxxTable from pulsar."xxxTenant/xxxNamespace"."last-topic-value";

Query 20210222_101426_00019_jvnf5, FAILED, 2 nodes
Splits: 18 total, 0 done (0.00%)
0.79 [0 rows, 0B] [0 rows/s, 0B/s]

You can see it cast a long time, and will not stop.
And you can see many error messages in presto server:
Query 20210222_101426_00019_jvnf5 failed: Duplicate key pfn_input_topic (attempted merging values persistent://first-tenant-value/first-namespace-value/first-topic-value and persistent://second-tenant-value/second-namespace-value/second-topic-valu)`
image

To Reproduce
Steps to reproduce the behavior:
In pulsar version before 2.7.0

  1. create 3 topics: first-topic-value, second-topic-value, last-topic-value
  2. create 2 pulsar functions, f1 pulls messages from first-topic-value, writes to second-topic-value, f2 pulls messages from second-topic-value, writes to last-topic-value.
  3. In all pulsar version (2.7.0 as newest currently)`presto> select * from xxxTable from pulsar."xxxTenant/xxxNamespace"."last-topic-value";
  4. See these errors

Expected behavior
query successfully, and No error.

Screenshots
Screen Shot 2021-02-24 at 00 16 09

Desktop (please complete the following information):

  • OS: CentOS , pulsar 2.7.0

Additional Information###

presto can't run in the ruined data came from old pulsar funcations.

Modifications

if key dumplicated, use new one.

Verifying this change

This change added tests and can be verified as follows:

  • Added unit test in RawMessageImplTest.java

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why? it is a bug
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
  • It is just like <fix duplicate key to send propertys #6390>, but it causes presto issue.

@pointearth pointearth closed this Feb 23, 2021
@pointearth pointearth force-pushed the fix_duplicate_key_RawMessageImpl branch from 8bc05d1 to b8170ee Compare February 23, 2021 16:46
@pointearth pointearth reopened this Feb 23, 2021
@pointearth
Copy link
Contributor Author

@codelipenghui please review the code.

@sijie
Copy link
Member

sijie commented Feb 23, 2021

@pointearth Great contribution!

@sijie sijie added area/sql Pulsar SQL related features release/2.7.1 type/bug The PR fixed a bug or issue reported a bug labels Feb 23, 2021
@sijie sijie added this to the 2.8.0 milestone Feb 23, 2021
@codelipenghui codelipenghui merged commit d2f3557 into apache:master Feb 24, 2021
zymap pushed a commit that referenced this pull request Mar 1, 2021
* fix Duplicate key __pfn_input_topic__ in presto server

(cherry picked from commit d2f3557)
@zymap zymap added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sql Pulsar SQL related features cherry-picked/branch-2.7 Archived: 2.7 is end of life release/2.7.1 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants