Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues when forced to rebuild corrupted index #206

Closed
stigok opened this issue Sep 28, 2018 · 1 comment
Closed

Issues when forced to rebuild corrupted index #206

stigok opened this issue Sep 28, 2018 · 1 comment
Milestone
4.0

Comments

@stigok
Copy link

stigok commented Sep 28, 2018

Found an issue with kafka nodes are coming up again after having failed. If they wake up to a corrupted index, it will attempt to fix itself. This seems to have two major implications:

  • Memory consumptions goes out the roof, resulting in pod getting killed due to OOM (due to limits, of course)
  • Readiness probe fails, and will kill the pod if it hasn't already OOM'ed

Any thoughts on how to remedy this?
@stigok stigok changed the title OOM when rebuilding corrupted index Issues when forced to rebuild corrupted index Sep 28, 2018
@solsson
Copy link
Contributor

solsson commented Sep 29, 2018

We didn't get around to implementing it yet, but the idea is to fix the default image to support the ./kafka-server-stop.sh command (solsson/dockerfiles@4fb7b5d) and to use a preStop pod lifecycle hook to invoke it.

@solsson solsson added this to the 4.0 milestone Sep 29, 2018
solsson added a commit to StreamingMicroservicesPlatform/docker-kafka that referenced this issue Sep 29, 2018
@solsson
Adds the ps command as a minimal layer on top of
5df87ac 
the kafka image used in current stable github.com/Yolean/kubernetes-kafka

See:
 - #6
 - Yolean/kubernetes-kafka#206
@solsson solsson mentioned this issue Sep 29, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
4.0
Development

No branches or pull requests
2 participants
@solsson @stigok