Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pulsar SQL can not query the latest ingested event in a topic , but Pulsar Consumer/Read always can. #4976

Closed
frank-dkvan opened this issue Aug 19, 2019 · 13 comments
Closed

Pulsar SQL can not query the latest ingested event in a topic , but Pulsar Consumer/Read always can. #4976

frank-dkvan opened this issue Aug 19, 2019 · 13 comments
Assignees
@codelipenghui
Labels
area/sql Pulsar SQL related features type/bug The PR fixed a bug or issue reported a bug
Milestone
2.6.0

Comments

@frank-dkvan
Copy link

Describe the bug
Pulsar SQL can not query the latest ingested event in a topic , but Pulsar Consumer/Read always can.

the testing Pulsar version is 2.4.0

from the two screenshots below, we can see the latest event is always missed in Palsar SQL. but Pulsar reader client can fetch the latest one correctly.

I am new to Pulsar but very interested in Pulsar. don't know this is a bug, or just a kind interesting feature which needs some hidden config I don't know yet?

Screenshots
this is the testing code and screenshot of Pulsar Reader:
image
this is the testing query SQL and screenshot in Pulsar SQL CLI:
image
@frank-dkvan frank-dkvan added the type/bug The PR fixed a bug or issue reported a bug label Aug 19, 2019
@frank-dkvan
Copy link
Author

looking at the screenshots above, we can find Pulsar Reader returns 5 records with the latest ingested event, but Pulsar SQL only returns 4 records without the latest ingested event. always missing one latest ingested event from Pulsar SQL while new event comes into Pulsar Topic

@sijie
Copy link
Member

sijie commented Aug 21, 2019

@frank-dkvan this is a known issue of #3828. because Pulsar doesn't flush LAC so when Presto SQL reads data from bookkeeper bypassing broker, it will miss the latest message because the advanced LAC is not flushed back.

The fix is waiting for BK 4.10 release.

@sijie sijie added area/sql Pulsar SQL related features triage/week-33 labels Aug 21, 2019
@frank-dkvan
Copy link
Author

@sijie got it! thanks

@frank-dkvan
Copy link
Author

frank-dkvan commented Dec 8, 2019 •
edited
Loading

@frank-dkvan this is a known issue of #3828. because Pulsar doesn't flush LAC so when Presto SQL reads data from bookkeeper bypassing broker, it will miss the latest message because the advanced LAC is not flushed back.

The fix is waiting for BK 4.10 release.

@sijie I tested and found this bug hasn't yet been fixed in latest 2.5.0 snapshot with BK 4.10 release.
will this bug get fixed in 2.5.0 official release? or it there any workaround solution/configuration to solve that?

@sijie
Copy link
Member

sijie commented Dec 8, 2019

@frank-dkvan I think it requires a change at Pulsar side. since we are cutting 2.5.0 next week, I don't think we are able to include this in time. We can do this for 2.5.1 or 2.6.0.

you might be able to work around with 4.10 with a bk client configuraiton. @codelipenghui should be able to help you with this.

@frank-dkvan
Copy link
Author

thanks @sijie
Hi @codelipenghui wld you pls help with this: providing more details on bk client configuration at pulsar side...sample code if any would be nice..thanks

@frank-dkvan frank-dkvan mentioned this issue Dec 9, 2019
Closed
@codelipenghui codelipenghui mentioned this issue Dec 9, 2019
Merged
@codelipenghui
Copy link
Contributor

@frank-dkvan @sijie I have create a new pull request #5822 for expose the config in broker.conf.

@sijie
Copy link
Member

sijie commented Dec 9, 2019

@frank-dkvan

I think you can add bookkeeperExplicitLacIntervalInMills=5 in the JVM system properties when starting the broker.

@codelipenghui
Copy link
Contributor

@sijie I think we need to use explicitLacInterval, this is the name in Bookkeeper client configuration. The bookkeeperExplicitLacIntervalInMills is named in broker side.

@sijie
Copy link
Member

sijie commented Dec 9, 2019

You are right!

@frank-dkvan
Copy link
Author

@frank-dkvan

I think you can add bookkeeperExplicitLacIntervalInMills=5 in the JVM system properties when starting the broker.

Hi @codelipenghui "5" means there would be 5-ms delay before we can see the latest message in Pulsar SQL? Am I understanding correctly?

@frank-dkvan
Copy link
Author

@frank-dkvan @sijie I have create a new pull request #5822 for expose the config in broker.conf.

Thanks @codelipenghui

@sijie sijie mentioned this issue Dec 27, 2019
Closed
codelipenghui added a commit that referenced this issue Feb 10, 2020
@codelipenghui
Expose bookkeeper expose explicit lac in broker.conf (#5822)
4fd17d4 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to #3828 #4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.
@codelipenghui
Copy link
Contributor

close via #5822

@codelipenghui codelipenghui self-assigned this Feb 10, 2020
@codelipenghui codelipenghui added this to the 2.6.0 milestone Feb 10, 2020
tuteng pushed a commit to AmateurEvents/pulsar that referenced this issue Feb 23, 2020
@codelipenghui @tuteng
Expose bookkeeper expose explicit lac in broker.conf (apache#5822)
c8eb719 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to apache#3828 apache#4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.
tuteng pushed a commit to AmateurEvents/pulsar that referenced this issue Mar 21, 2020
@codelipenghui @tuteng
Expose bookkeeper expose explicit lac in broker.conf (apache#5822)
f98d4c1 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to apache#3828 apache#4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.

(cherry picked from commit 4fd17d4)
tuteng pushed a commit that referenced this issue Apr 13, 2020
@codelipenghui @tuteng
Expose bookkeeper expose explicit lac in broker.conf (#5822)
c421d26 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to #3828 #4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.

(cherry picked from commit 4fd17d4)
@codelipenghui codelipenghui mentioned this issue May 7, 2020
Merged
jiazhai pushed a commit to jiazhai/pulsar that referenced this issue May 18, 2020
@codelipenghui @jiazhai
Expose bookkeeper expose explicit lac in broker.conf (apache#5822)
652ada3 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to apache#3828 apache#4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.
(cherry picked from commit 4fd17d4)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@codelipenghui @huangdx0726
Expose bookkeeper expose explicit lac in broker.conf (apache#5822)
6eca5dd 
### Motivation

Expose bookkeeper expose explicit lac configuration in broker.conf
It's related to apache#3828 apache#4976, some Pulsar SQL users need to enable the explicitLacInterval, so that they can get the last message in Pulsar SQL.
@AntiBit82 AntiBit82 mentioned this issue Feb 18, 2022
Closed
@sijie sijie mentioned this issue Feb 18, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codelipenghui codelipenghui

Labels
area/sql Pulsar SQL related features type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
2.6.0
Development

No branches or pull requests
3 participants
@sijie @codelipenghui @frank-dkvan