New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could Not retrieve Inputs (Fetching inputs failed with status: 500) #1923

Closed
123dev opened this Issue Mar 13, 2016 · 4 comments

Comments

Projects
None yet
4 participants
@123dev

123dev commented Mar 13, 2016

Problem description

I was editing one of the inputs to set the source value when it failed to save (can't remember the error, a red popup from the bottom of the screen was seen)
Afterwards the input was stopped.
Stopping Graylog, and restarting it back, or even rebooting the system, hasn't helped.
Since this error, I'm unable to access any of the inputs.
Although one of my browser windows was open on the extractors page of one the inputs, which is still accessible
http://10.1.1.94:9000/system/inputs/00c2670c-064b-415a-8768-104eb0e7ce08/56e021bade20c60caec10dde/extractors

But http://10.1.1.94:9000/system/inputs/00c2670c-064b-415a-8768-104eb0e7ce08/56e021bade20c60caec10dde is not

I have more inputs,

I see the following messages in log

2016-03-13 07:36:57,777 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2016-03-13 07:36:58,019 WARN : org.graylog2.inputs.PersistedInputsImpl - Cannot instantiate persisted input. Exception caught:
java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList
        at org.graylog2.inputs.InputImpl.getStaticFields(InputImpl.java:120) ~[graylog.jar:?]
        at org.graylog2.inputs.InputServiceImpl.getMessageInput(InputServiceImpl.java:337) ~[graylog.jar:?]
        at org.graylog2.inputs.PersistedInputsImpl.iterator(PersistedInputsImpl.java:54) [graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher.launchAllPersisted(InputLauncher.java:115) [graylog.jar:?]
        at org.graylog2.shared.initializers.InputSetupService.run(InputSetupService.java:92) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$3.run(Callables.java:100) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
2016-03-13 07:36:58,095 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Raw/Plaintext UDP/56db757cde20c6076e325ce0] is now STARTING
2016-03-13 07:36:58,111 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Syslog UDP/56dc3a45de20c6076e332b8b] is now STARTING
2016-03-13 07:36:58,119 INFO : org.graylog2.shared.inputs.InputStateListener - Input [GELF UDP/56dc3ae1de20c6076e332c33] is now STARTING
2016-03-13 07:36:58,130 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Raw/Plaintext UDP/56dc4fbede20c6076e3342ba] is now STARTING
2016-03-13 07:36:58,162 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Syslog UDP/56e021bade20c60caec10dde] is now STARTING
2016-03-13 07:36:58,324 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog UDP [Router], type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=00c2670c-064b-415a-8768-104eb0e7ce08} should be 262144 but is 212992.
2016-03-13 07:36:58,324 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Syslog UDP/56e021bade20c60caec10dde] is now RUNNING
2016-03-13 07:36:58,330 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Syslog UDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=00c2670c-064b-415a-8768-104eb0e7ce08} should be 262144 but is 212992.
2016-03-13 07:36:58,331 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Cisco Catalyst, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=00c2670c-064b-415a-8768-104eb0e7ce08} should be 262144 but is 212992.
2016-03-13 07:36:58,331 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Syslog UDP/56dc3a45de20c6076e332b8b] is now RUNNING
2016-03-13 07:36:58,332 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Raw/Plaintext UDP, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=00c2670c-064b-415a-8768-104eb0e7ce08} should be 262144 but is 212992.
2016-03-13 07:36:58,334 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=GELF UDP, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=00c2670c-064b-415a-8768-104eb0e7ce08} should be 262144 but is 212992.
2016-03-13 07:36:58,343 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Raw/Plaintext UDP/56dc4fbede20c6076e3342ba] is now RUNNING
2016-03-13 07:36:58,347 INFO : org.graylog2.shared.inputs.InputStateListener - Input [GELF UDP/56dc3ae1de20c6076e332c33] is now RUNNING
2016-03-13 07:36:58,353 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Raw/Plaintext UDP/56db757cde20c6076e325ce0] is now RUNNING
2016-03-13 07:37:12,943 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Enabling session validation scheduler...
2016-03-13 07:37:13,274 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2016-03-13 07:37:26,215 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList
        at org.graylog2.inputs.InputImpl.getStaticFields(InputImpl.java:120) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.InputsResource.getInputSummary(InputsResource.java:208) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.InputsResource.lambda$list$0(InputsResource.java:108) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_65]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_65]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_65]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_65]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_65]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_65]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_65]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_65]
        at org.graylog2.rest.resources.system.inputs.InputsResource.list(InputsResource.java:109) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
2016-03-13 07:37:28,602 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017
2016-03-13 07:38:07,068 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.ClassCastException: com.mongodb.BasicDBObject cannot be cast to com.mongodb.BasicDBList
        at org.graylog2.inputs.InputImpl.getStaticFields(InputImpl.java:120) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.InputsResource.getInputSummary(InputsResource.java:208) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.InputsResource.lambda$list$0(InputsResource.java:108) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_65]

Steps to reproduce the problem

Don't know how to reproduce the problem, as I have other inputs that I managed to edit the source without issues,
The Input in question was the Netflow plugin input.
https://marketplace.graylog.org/addons/29bcffe3-816e-402f-89c8-06a8e2657c34

The alpha 5 was in general working very nicely (great job folks) aside from few glitches here and there which I will report (if they are not already reported) I haven't had any serious issue.
However now it's quite broken

Thanks

Environment

  • Graylog Version: 2.0 alpha 5
  • Elasticsearch Version: 2.2.0
  • MongoDB Version: 3.2.4
  • Operating System: ubuntu 14.04 TLS
  • Browser version: Chrome x64
@123dev

This comment has been minimized.

123dev commented Mar 13, 2016

I Managed to fix the problem.

I looked at the MongoDB entries for the inputs and noticed the following.

  • The source value was properly overwritten
  • However during saving, the static field that was previously added, was badly saved.

Instead of saving it in an array such as

    "static_fields" : [
        {
            "value" : "Netflow_UDP", 
            "key" : "GraylogInput"
        }
    ]

It was saved like this:

    "static_fields" : {
        "GraylogInput" : "Netflow_UDP"
    }

Deleting the input from MongoDB and adding it back from the UI fixed the problem.

Though I suppose there could be a glitch in the saving / update code that caused the corruption.

@kroepke

This comment has been minimized.

Member

kroepke commented Mar 13, 2016

Actually it looks like this is related to 03927cb

This might be a bug regarding backwards compatibility

@joschi

This comment has been minimized.

Contributor

joschi commented Mar 14, 2016

@kroepke kroepke added bug S1 P1 labels Mar 18, 2016

@kroepke kroepke added this to the 2.0.0 milestone Mar 18, 2016

@bernd bernd self-assigned this Mar 30, 2016

bernd referenced this issue Mar 30, 2016

@bernd

This comment has been minimized.

Member

bernd commented Mar 30, 2016

This has been fixed in b612c97. Thank you for the report!

@bernd bernd closed this Mar 30, 2016

@kroepke kroepke added the triaged label Sep 21, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment