Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create JSON extractor #3630

edmundoa opened this issue Mar 16, 2017 · 3 comments

Cannot create JSON extractor #3630

edmundoa opened this issue Mar 16, 2017 · 3 comments


Copy link

@edmundoa edmundoa commented Mar 16, 2017

On the current master, creating a JSON extractor fails with the current server output:

2017-03-16 17:51:32,969 ERROR: - Extractor persist validation failed.
org.graylog2.plugin.database.ValidationException: null
	at org.graylog2.database.PersistedServiceImpl.embed( ~[classes/:?]
	at org.graylog2.inputs.InputServiceImpl.addExtractor( ~[classes/:?]
	at [classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
	at sun.reflect.NativeMethodAccessorImpl.invoke( ~[?:1.8.0_112]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke( ~[?:1.8.0_112]
	at java.lang.reflect.Method.invoke( ~[?:1.8.0_112]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$ [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$ [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$ [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors$ [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process( [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process( [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process( [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope( [jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle( [jersey-server-2.25.1.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service( [jersey-container-grizzly2-http-2.25.1.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$ [grizzly-http-server-2.3.28.jar:2.3.28]
	at com.codahale.metrics.InstrumentedExecutorService$ [metrics-core-3.2.0.jar:3.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$ [?:1.8.0_112]
	at [?:1.8.0_112]

Using a breakpoint before the PersistedServiceImpl validation, I could see that the validation error is an expected target_field field in the extractor configuration. This field cannot be set for JSON extractors, as the extractor may create many fields from a single one.

Steps to Reproduce (for bugs)

  1. Fill out the create JSON extractor form. Here is the configuration I used:
  "title": "JSON extractor",
  "cut_or_copy": "copy",
  "source_field": "message",
  "target_field": "",
  "extractor_type": "json",
  "extractor_config": {
    "list_separator": ", ",
    "key_separator": "_",
    "kv_separator": "=",
    "key_prefix": "",
    "replace_key_whitespace": true,
    "key_whitespace_replacement": "_"
  "converters": {},
  "condition_type": "none",
  "condition_value": ""
  1. Press on create extractor and see that the server returns a 400 code and the error mentioned above is in the logs

Your Environment

  • Graylog Version: Graylog 2.3.0-SNAPSHOT (4802bbb)
@edmundoa edmundoa added the bug label Mar 16, 2017
Copy link

@joschi joschi commented Mar 17, 2017

This was probably introduced in 1aefec0.

The validators for extractors were never run before (see

public Map<String, Validator> getEmbeddedValidations(String key) {
if (EMBEDDED_EXTRACTORS.equals(key)) {
final ImmutableMap.Builder<String, Validator> validations = ImmutableMap.builder();
validations.put(Extractor.FIELD_ID, new FilledStringValidator());
validations.put(Extractor.FIELD_TITLE, new FilledStringValidator());
validations.put(Extractor.FIELD_TYPE, new FilledStringValidator());
validations.put(Extractor.FIELD_CURSOR_STRATEGY, new FilledStringValidator());
validations.put(Extractor.FIELD_TARGET_FIELD, new FilledStringValidator());
validations.put(Extractor.FIELD_SOURCE_FIELD, new FilledStringValidator());
validations.put(Extractor.FIELD_CREATOR_USER_ID, new FilledStringValidator());
validations.put(Extractor.FIELD_EXTRACTOR_CONFIG, new MapValidator());

So the question is, should we:

  1. Fix the validations (e. g. target_field is optional)?
  2. Remove the validations altogether, leading to the same (wrong) behavior as before?
Copy link

@dennisoelkers dennisoelkers commented Mar 17, 2017

My vote goes for 1.).

Copy link

@kroepke kroepke commented Mar 17, 2017

I also vote for 1

joschi pushed a commit that referenced this issue Mar 17, 2017
Jochen Schalanda
Fixes #3630
@joschi joschi added this to the 2.3.0 milestone Mar 17, 2017
@joschi joschi self-assigned this Mar 17, 2017
@joschi joschi added the in progress label Mar 17, 2017
@jalogisch jalogisch added the triaged label Mar 27, 2017
@ghost ghost removed the in progress label Apr 19, 2017
kroepke added a commit that referenced this issue Apr 19, 2017
* Fix OptionalStringValidator and validations for extractors

Fixes #3630

* Update LimitedOptionalStringValidatorTest to allow empty strings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants