Cannot create JSON extractor

[edmundoa]

On the current master, creating a JSON extractor fails with the current server output:

2017-03-16 17:51:32,969 ERROR: org.graylog2.rest.resources.system.inputs.ExtractorsResource - Extractor persist validation failed.
org.graylog2.plugin.database.ValidationException: null
	at org.graylog2.database.PersistedServiceImpl.embed(PersistedServiceImpl.java:248) ~[classes/:?]
	at org.graylog2.inputs.InputServiceImpl.addExtractor(InputServiceImpl.java:187) ~[classes/:?]
	at org.graylog2.rest.resources.system.inputs.ExtractorsResource.create(ExtractorsResource.java:124) [classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [grizzly-http-server-2.3.28.jar:2.3.28]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [metrics-core-3.2.0.jar:3.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Using a breakpoint before the PersistedServiceImpl validation, I could see that the validation error is an expected target_field field in the extractor configuration. This field cannot be set for JSON extractors, as the extractor may create many fields from a single one.

Steps to Reproduce (for bugs)

1. Fill out the create JSON extractor form. Here is the configuration I used:

{
  "title": "JSON extractor",
  "cut_or_copy": "copy",
  "source_field": "message",
  "target_field": "",
  "extractor_type": "json",
  "extractor_config": {
    "list_separator": ", ",
    "key_separator": "_",
    "kv_separator": "=",
    "key_prefix": "",
    "replace_key_whitespace": true,
    "key_whitespace_replacement": "_"
  },
  "converters": {},
  "condition_type": "none",
  "condition_value": ""
}

2. Press on create extractor and see that the server returns a 400 code and the error mentioned above is in the logs

Your Environment

• Graylog Version: Graylog 2.3.0-SNAPSHOT (4802bbb)

[joschi]

This was probably introduced in 1aefec0.

The validators for extractors were never run before (see

graylog2-server/graylog2-server/src/main/java/org/graylog2/inputs/InputImpl.java

Lines 76 to 88 in 1aefec0

	public Map<String, Validator> getEmbeddedValidations(String key) {
	if (EMBEDDED_EXTRACTORS.equals(key)) {
	final ImmutableMap.Builder<String, Validator> validations = ImmutableMap.builder();
	validations.put(Extractor.FIELD_ID, new FilledStringValidator());
	validations.put(Extractor.FIELD_TITLE, new FilledStringValidator());
	validations.put(Extractor.FIELD_TYPE, new FilledStringValidator());
	validations.put(Extractor.FIELD_CURSOR_STRATEGY, new FilledStringValidator());
	validations.put(Extractor.FIELD_TARGET_FIELD, new FilledStringValidator());
	validations.put(Extractor.FIELD_SOURCE_FIELD, new FilledStringValidator());
	validations.put(Extractor.FIELD_CREATOR_USER_ID, new FilledStringValidator());
	validations.put(Extractor.FIELD_EXTRACTOR_CONFIG, new MapValidator());
	return validations.build();
	}

).

So the question is, should we:

1. Fix the validations (e. g. target_field is optional)?
2. Remove the validations altogether, leading to the same (wrong) behavior as before?

[dennisoelkers]

My vote goes for 1.).

[kroepke]

I also vote for 1