New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outputs that cannot be initialized are being tried to initialize constantly #741

Closed
lennartkoopmann opened this Issue Oct 19, 2014 · 2 comments

Comments

Projects
None yet
4 participants
@lennartkoopmann
Member

lennartkoopmann commented Oct 19, 2014

When a MessageOuput is throwing an exception in its initialize method, the server tries to initialize it again for every message that goes through the OutputRouter.

screen shot 2014-10-19 at 02 12 29

@lennartkoopmann lennartkoopmann added this to the 0.92 milestone Oct 19, 2014

@lennartkoopmann lennartkoopmann changed the title from Outputs that cannot be initialized are being tried to initialized constantly to Outputs that cannot be initialized are being tried to initialize constantly Oct 19, 2014

@dennisoelkers

This comment has been minimized.

Member

dennisoelkers commented Dec 3, 2014

We are initialising them lazily because it's much simpler. I am in the process of changing this right now, because when we/3rd party implement outputs that need to hold resources (like network/database connections) we cannot do this anymore and need proper state handling. Won't be fixed in 0.92.x, but in 0.93.0.

@dennisoelkers dennisoelkers self-assigned this Dec 3, 2014

@joschi joschi modified the milestones: 0.93, 0.92 Dec 4, 2014

@kroepke

This comment has been minimized.

Member

kroepke commented Jan 14, 2015

What is the state of this issue, @dennisoelkers ?
Is it already in 1.0?

dennisoelkers added a commit that referenced this issue Jan 16, 2015

Disabling stream outputs temporarily after failing during initializat…
…ion repeatedly.

We do not handle outputs the same way as we do with inputs right now. We
don't know which outputs are going to run on which nodes, so we are just
starting them lazily when an output's stream is catching a message and
they are kept running until the server restarts or the output is
deleted.

This means that we cannot just flag them as failed, so the approach here
is to temporarily refrain from trying to instantiate them after a certain
configurable fault count threshold was crossed. The fault count is
tracked using an internal counter which is reset after the fault penalty
time frame passed (even if the threshold was not crossed).

As a plus, fixing output removal, which is actually stopping the output
now.

Fixes #741

joschi added a commit that referenced this issue Jan 21, 2015

joschi added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 21, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment