Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get messages implementation improvement #253

Closed
iliax opened this issue Mar 16, 2021 · 0 comments · Fixed by #286
Closed

Get messages implementation improvement #253

iliax opened this issue Mar 16, 2021 · 0 comments · Fixed by #286
Assignees
@iliax
Labels
scope/backend type/enhancement En enhancement to an already existing feature
Projects
Kafka UI
Milestone
0.1

Comments

@iliax
Copy link
Contributor

iliax commented Mar 16, 2021 •
edited
Loading

Currently ConsumingService.RecordEmitter has following limitations & problems:

https://github.com/provectus/kafka-ui/blob/master/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ConsumingService.java#L118

  1. will block 3 sec even if topic was read to the end (or empty)
  2. always will do 3 polls (bug?)
  3. has a very strict max records limit (100)

Suggestion is to rewrite RecordEmitter as a Flow.Publisher with end offsets check. Also backpressure support can be useful if you'll decide to use json streaming (application/stream+json) to return huge number of messages in the future.
@germanosin germanosin added this to the 0.1 milestone Mar 17, 2021
@germanosin germanosin added scope/backend type/enhancement En enhancement to an already existing feature labels Mar 17, 2021
@workshur workshur linked a pull request Mar 18, 2021 that will close this issue
#253: Get messages implementation improvement #286
Merged
@iliax iliax added this to In progress in Kafka UI Mar 19, 2021
@iliax iliax self-assigned this Mar 22, 2021
germanosin pushed a commit that referenced this issue Mar 23, 2021
#253: Get messages implementation improvement (#286)
992e8b0 
* 1. End offsets check added to RecordEmitter
2. Tests for OffsetsSeek, RecordEmitter added

* Tests improvements

* ISSUE-257: checkstyle

* ISSUE-257: withSchemaType sonar fix

* ISSUE-257: withSchemaType sonar fix

Co-authored-by: Ilya Kuramshin <ikuramshin@provectus.com>
@iliax iliax moved this from In progress to Done in Kafka UI Mar 24, 2021
javalover123 pushed a commit to javalover123/kafka-ui that referenced this issue Dec 7, 2022
provectus#253: Get messages implementation improvement (provectus#286)
5a8c9ec 
* 1. End offsets check added to RecordEmitter
2. Tests for OffsetsSeek, RecordEmitter added

* Tests improvements

* ISSUE-257: checkstyle

* ISSUE-257: withSchemaType sonar fix

* ISSUE-257: withSchemaType sonar fix

Co-authored-by: Ilya Kuramshin <ikuramshin@provectus.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iliax iliax

Labels
scope/backend type/enhancement En enhancement to an already existing feature
Projects
No open projects
Kafka UI
  
Done
Milestone
0.1
Development

Successfully merging a pull request may close this issue.

#253: Get messages implementation improvement provectus/kafka-ui
2 participants
@iliax @germanosin