New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create JSON extractor #3630

Closed
edmundoa opened this Issue Mar 16, 2017 · 3 comments

Comments

Projects
None yet
5 participants
@edmundoa
Member

edmundoa commented Mar 16, 2017

On the current master, creating a JSON extractor fails with the current server output:

2017-03-16 17:51:32,969 ERROR: org.graylog2.rest.resources.system.inputs.ExtractorsResource - Extractor persist validation failed.
org.graylog2.plugin.database.ValidationException: null
	at org.graylog2.database.PersistedServiceImpl.embed(PersistedServiceImpl.java:248) ~[classes/:?]
	at org.graylog2.inputs.InputServiceImpl.addExtractor(InputServiceImpl.java:187) ~[classes/:?]
	at org.graylog2.rest.resources.system.inputs.ExtractorsResource.create(ExtractorsResource.java:124) [classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [grizzly-http-server-2.3.28.jar:2.3.28]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [metrics-core-3.2.0.jar:3.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Using a breakpoint before the PersistedServiceImpl validation, I could see that the validation error is an expected target_field field in the extractor configuration. This field cannot be set for JSON extractors, as the extractor may create many fields from a single one.

Steps to Reproduce (for bugs)

  1. Fill out the create JSON extractor form. Here is the configuration I used:
{
  "title": "JSON extractor",
  "cut_or_copy": "copy",
  "source_field": "message",
  "target_field": "",
  "extractor_type": "json",
  "extractor_config": {
    "list_separator": ", ",
    "key_separator": "_",
    "kv_separator": "=",
    "key_prefix": "",
    "replace_key_whitespace": true,
    "key_whitespace_replacement": "_"
  },
  "converters": {},
  "condition_type": "none",
  "condition_value": ""
}
  1. Press on create extractor and see that the server returns a 400 code and the error mentioned above is in the logs

Your Environment

  • Graylog Version: Graylog 2.3.0-SNAPSHOT (4802bbb)

@edmundoa edmundoa added the bug label Mar 16, 2017

@joschi

This comment has been minimized.

Contributor

joschi commented Mar 17, 2017

This was probably introduced in 1aefec0.

The validators for extractors were never run before (see

public Map<String, Validator> getEmbeddedValidations(String key) {
if (EMBEDDED_EXTRACTORS.equals(key)) {
final ImmutableMap.Builder<String, Validator> validations = ImmutableMap.builder();
validations.put(Extractor.FIELD_ID, new FilledStringValidator());
validations.put(Extractor.FIELD_TITLE, new FilledStringValidator());
validations.put(Extractor.FIELD_TYPE, new FilledStringValidator());
validations.put(Extractor.FIELD_CURSOR_STRATEGY, new FilledStringValidator());
validations.put(Extractor.FIELD_TARGET_FIELD, new FilledStringValidator());
validations.put(Extractor.FIELD_SOURCE_FIELD, new FilledStringValidator());
validations.put(Extractor.FIELD_CREATOR_USER_ID, new FilledStringValidator());
validations.put(Extractor.FIELD_EXTRACTOR_CONFIG, new MapValidator());
return validations.build();
}
).

So the question is, should we:

  1. Fix the validations (e. g. target_field is optional)?
  2. Remove the validations altogether, leading to the same (wrong) behavior as before?
@dennisoelkers

This comment has been minimized.

Member

dennisoelkers commented Mar 17, 2017

My vote goes for 1.).

@kroepke

This comment has been minimized.

Member

kroepke commented Mar 17, 2017

I also vote for 1

joschi added a commit that referenced this issue Mar 17, 2017

@joschi joschi added this to the 2.3.0 milestone Mar 17, 2017

@joschi joschi self-assigned this Mar 17, 2017

@joschi joschi added the in progress label Mar 17, 2017

@jalogisch jalogisch added the triaged label Mar 27, 2017

@wafflebot wafflebot bot removed the in progress label Apr 19, 2017

kroepke added a commit that referenced this issue Apr 19, 2017

Fix OptionalStringValidator and validations for extractors (#3633)
* Fix OptionalStringValidator and validations for extractors

Fixes #3630

* Update LimitedOptionalStringValidatorTest to allow empty strings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment