Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consumption of partition stops if segment expires #100

Closed
horkhe opened this issue Mar 24, 2017 · 1 comment
Closed

Consumption of partition stops if segment expires #100

horkhe opened this issue Mar 24, 2017 · 1 comment
Labels
bug

Comments

@horkhe
Copy link
Member

horkhe commented Mar 24, 2017

If a segment of a topic partition is deleted by Kafka due to retention period exceeded while Kafka-Pixy reads from it, then consumption from the partition stops. There is no even log record with ERROR severity reported.

Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> started
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> partition claimed: via=/_[0]/cons[0]/G:analytics[1]/member[0], retries=0, took=280ms
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> started
Mar 23 07:01:38  </_[0]/offset_mgr_f[0]/mapper[0]/reassign[164]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0] -> /_[0]/offset_mgr_f[0]/broker_2_aggr[0] (ref=9)
Mar 23 07:01:38  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/offset_mgr[0]> assigned /_[0]/offset_mgr_f[0]/broker_2_aggr[0]
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]> initialized: offset=485662619, sparseAcks=
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> started
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/mapper[0]/reassign[5]> assign /_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0] -> /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0] (ref=2)
Mar 23 07:01:40  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> assigned /_[0]/cons[0]/G:analytics[1]/msg_stream_f[0]/broker_0_aggr[0]
Mar 23 07:02:39  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> fetch failed: err=kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.
Mar 23 07:02:39  </_[0]/cons[0]/G:analytics[1]/P:use1_pipeline_events_35[0]/msg_stream[0]> stopped
@horkhe horkhe added the bug label Mar 24, 2017
@horkhe horkhe closed this as completed in 8c3146f Jun 14, 2017
horkhe added a commit that referenced this issue Jun 14, 2017
@horkhe
Merge pull request #104 from mailgun/maxim/develop
1eda773 
Recover after segment expires (fixes #100)
@horkhe horkhe reopened this Aug 16, 2017
@horkhe
Copy link
Member Author

horkhe commented Aug 16, 2017

Apparently this issue was not fixed, we still experience it on v0.13.0

horkhe added a commit that referenced this issue Aug 17, 2017
@horkhe
Recover after fetcher fatal failure (fixes #100)
e1fa238 
This is the second attemp to fix this issue. The first
solution was not complete. This time we have a test to
prove it.
horkhe added a commit that referenced this issue Aug 17, 2017
@horkhe
Recover after fetcher fatal failure (fixes #100)
aaa3964 
This is the second attemp to fix this issue. The first
solution was not complete. This time we have a test to
prove it.
@horkhe horkhe closed this as completed in 8eade93 Aug 18, 2017
horkhe added a commit that referenced this issue Aug 18, 2017
@horkhe
Merge pull request #113 from mailgun/maxim/develop
74b35d7 
Recover after fetcher fatal failure (fixes #100)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@horkhe