New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add raw message loader #2438

Merged
merged 21 commits into from Jul 4, 2016

Conversation

Projects
None yet
2 participants
@edmundoa
Member

edmundoa commented Jun 29, 2016

This PR adds a raw message loader that can be used standalone or integrated in our LoaderTabs component. It is required as part of the pipeline simulator, so the component is still not in use.

Summary of the changes done in here:

  • Add rest resource to parse raw message
  • Add rest resource returning codec types and their configuration
  • Convert old MessagesStore into a Reflux store
  • Add stores and actions to be able to use the previous resources
  • Add component capable of loading a raw message
  • Adapt LoaderTabs to offer the possibility of using the raw message loader

As this PR is already too big, I have avoided refactoring and reformatting LoaderTabs and the old message loaders. I will work on that once this PR is merged, so it is a bit more sane reviewing that afterwards.

I mentioned that the RawMessageLoader component is still not used, so in order to test it you can modify one of the components using LoaderTabs. This is an example for AddExtractorWizard:

       const extractorFieldActions = <MessageFieldExtractorActions fieldName="" message={{}}/>;
       extractorForm = (
         <div className="stream-loader">
-          <LoaderTabs selectedInputId={this.props.inputId} customFieldActions={extractorFieldActions} />
+          <LoaderTabs tabs={['recent', 'messageId', 'raw']} selectedInputId={this.props.inputId} customFieldActions={extractorFieldActions} />
         </div>
       );
     }

@edmundoa edmundoa added the web label Jun 29, 2016

@edmundoa edmundoa added this to the 2.1.0 milestone Jun 29, 2016

@edmundoa edmundoa force-pushed the raw-message-loader branch from 535ac56 to 5c7728d Jun 30, 2016

@bernd bernd self-assigned this Jul 1, 2016

@bernd

This comment has been minimized.

Member

bernd commented Jul 1, 2016

I created a simple raw message and switched to the "Recent Message" tab. The message from the "Raw Message" tab was sill visible. It only disappears when I pressed "Load Message" for the "Recent Message". This was a bit confusing. Not a big issue though.

Might be more confusing if there is no recent message. What happens then?

acccaaaiiiiaaaggggibfaorbi apiwaccccaaaiiiiaaaghykuaytfkf2ggggaaccccaaaiiiicarqkeqyvgkysaaggggaaccccaaaiiwcnwfx6avjl4bflvaaaaaelftksuqmcc

@bernd

This comment has been minimized.

Member

bernd commented Jul 1, 2016

I am getting the following error when trying to analyze a message with the netflow coded from the netflow plugin. Looks like the message parser cannot handle multi-message codecs yet.

2016-07-01 10:43:26,421 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.UnsupportedOperationException: MultiMessageCodec class org.graylog.plugins.netflow.codecs.NetFlowCodec does not support decode()
    at org.graylog.plugins.netflow.codecs.NetFlowCodec.decode(NetFlowCodec.java:56) ~[?:?]
    at org.graylog2.rest.resources.messages.MessageResource.decodeMessage(MessageResource.java:155) ~[classes/:?]
    at org.graylog2.rest.resources.messages.MessageResource.parse(MessageResource.java:149) ~[classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_91]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_91]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.22.1.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.22.1.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [jersey-container-grizzly2-http-2.22.1.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [grizzly-http-server-2.3.23.jar:2.3.23]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [metrics-core-3.1.2.jar:3.1.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
@bernd

This comment has been minimized.

Member

bernd commented Jul 1, 2016

I was also wondering if it makes sense to configure the codec when we want to create an extractor for an existing input. Maybe the component needs a flag to disable the codec config?

stream_ids: message.streams,
highlight_ranges: messageSummary.highlight_ranges,

This comment has been minimized.

@bernd

bernd Jul 1, 2016

Member

The highlight_ranges isn't needed anymore?

This comment has been minimized.

@edmundoa

edmundoa Jul 1, 2016

Member

Probably I removed it by mistake, will investigate. Nice catch anyway 👍

This comment has been minimized.

@edmundoa

edmundoa Jul 1, 2016

Member

It was indeed needed, so I added it back

@edmundoa

This comment has been minimized.

Member

edmundoa commented Jul 1, 2016

It's hard to reply in here making sense, as messages are not following any order, so I'll just quote from your comments:

I created a simple raw message and switched to the "Recent Message" tab. The message from the "Raw Message" tab was sill visible. It only disappears when I pressed "Load Message" for the "Recent Message". This was a bit confusing. Not a big issue though.

I think we should remove the preview when the tabs change. The inputs should still have the data, so it's a matter of click "load message" again. Will take care of that.

I am getting the following error when trying to analyze a message with the netflow coded from the netflow plugin. Looks like the message parser cannot handle multi-message codecs yet.

Multi-message codecs are not supported yet, and I think we only have one of those. Not sure how important it is, but I can take a look, probably it's not too hard to do.

I was also wondering if it makes sense to configure the codec when we want to create an extractor for an existing input. Maybe the component needs a flag to disable the codec config?

For now we are not using the loader in there, and there will be some changes to do if we want to do so. For instance, the REST resource for that only supports loading messages with an ID and index. I would rather wait with any changes for that until we do it and see what we actually need for it.

@bernd

This comment has been minimized.

Member

bernd commented Jul 1, 2016

Multi-message codecs are not supported yet, and I think we only have one of those. Not sure how important it is, but I can take a look, probably it's not too hard to do.

I think it's not important right know. A better error message would be good though. The user currently gets a 500 error message that doesn't say much.

edmundoa added some commits Jun 21, 2016

Add rest endpoint to decode raw messages
This is needed to allow users to work with raw messages in the UI, being
able to visualize modifications on messages not processed by Graylog.
Use arrow function, to set the right `this`
Using `MessageFormatter.formatMessageSummary()` directly in Array's
`map()` sets the wrong `this` value in the callback. We replace the
direct call with an arrow function that takes care of that.
Add raw message loader
Still not connected with `LoaderTabs`, but loads the message in the
console for now using the given parameters.
Include raw message parameters in callback
Allow customers of the loader to know which configuration was used to
load the message.

@edmundoa edmundoa force-pushed the raw-message-loader branch from 0abc533 to c59263c Jul 1, 2016

edmundoa added some commits Jul 1, 2016

Reset loaded message preview after changing tabs
When a message is loader in `LoaderTabs` and the user switches to
another tab, the preview was still present, which could be confusing
when trying to load another message (is the preview for the previous
message or the new one?).
Add highlight ranges into formatted message
The key was removed by mistake, and is used in the search results.
Make decoding error handling more robust
Catch exceptions during decoding, and also show a more helpful error
message in the frontend when a decoding exception occurred.
@bernd

This comment has been minimized.

Member

bernd commented Jul 4, 2016

LGTM 👍

@bernd bernd merged commit 2ed9d94 into master Jul 4, 2016

4 checks passed

ci-server-integration Jenkins build graylog2-server-integration-pr 1056 has succeeded
Details
ci-web-linter Jenkins build graylog-pr-linter-check 542 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@bernd bernd deleted the raw-message-loader branch Jul 4, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment