/graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outputs that cannot be initialized are being tried to initialize constantly #741

Closed
lennartkoopmann opened this Issue Oct 19, 2014 · 2 comments

Comments

Assignees

@dennisoelkers dennisoelkers

Labels
None yet
Projects
None yet
Milestone
  1.0.0
4 participants
@lennartkoopmann @dennisoelkers @kroepke @joschi
@lennartkoopmann
Member

lennartkoopmann commented Oct 19, 2014

When a MessageOuput is throwing an exception in its initialize method, the server tries to initialize it again for every message that goes through the OutputRouter.

screen shot 2014-10-19 at 02 12 29

@lennartkoopmann lennartkoopmann added this to the 0.92 milestone Oct 19, 2014

@lennartkoopmann lennartkoopmann changed the title from Outputs that cannot be initialized are being tried to initialized constantly to Outputs that cannot be initialized are being tried to initialize constantly Oct 19, 2014

@dennisoelkers

This comment has been minimized.

Sign in to view
Member

dennisoelkers commented Dec 3, 2014

We are initialising them lazily because it's much simpler. I am in the process of changing this right now, because when we/3rd party implement outputs that need to hold resources (like network/database connections) we cannot do this anymore and need proper state handling. Won't be fixed in 0.92.x, but in 0.93.0.

@dennisoelkers dennisoelkers self-assigned this Dec 3, 2014

@joschi joschi modified the milestones: 0.93, 0.92 Dec 4, 2014

@kroepke

This comment has been minimized.

Sign in to view
Member

kroepke commented Jan 14, 2015

What is the state of this issue, @dennisoelkers ?
Is it already in 1.0?

@dennisoelkers dennisoelkers closed this in c935e90 Jan 16, 2015

dennisoelkers added a commit that referenced this issue Jan 16, 2015

@dennisoelkers
Disabling stream outputs temporarily after failing during initializat… 
…ion repeatedly.

We do not handle outputs the same way as we do with inputs right now. We
don't know which outputs are going to run on which nodes, so we are just
starting them lazily when an output's stream is catching a message and
they are kept running until the server restarts or the output is
deleted.

This means that we cannot just flag them as failed, so the approach here
is to temporarily refrain from trying to instantiate them after a certain
configurable fault count threshold was crossed. The fault count is
tracked using an internal counter which is reset after the fault penalty
time frame passed (even if the threshold was not crossed).

As a plus, fixing output removal, which is actually stopping the output
now.

Fixes #741
Loading status checks…
e5e8753

joschi added a commit that referenced this issue Jan 21, 2015

@joschi
Add system notification if output has been disabled 
Refs #741
Loading status checks…
52ef686

joschi added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 21, 2015

@joschi
Add support for OUTPUT_DISABLED notification 
Refs Graylog2/graylog2-server#741
Loading status checks…
86717a4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment