New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in Import content pack #2663

Closed
greenx opened this Issue Aug 11, 2016 · 7 comments

Comments

Projects
None yet
5 participants
@greenx

greenx commented Aug 11, 2016

Hi, ALL!
I have two similar graylog systems (test->prod).
I created on test system: a few syslog udp inputs and grok patter extractor on each input.
I wanted copy this setting to production system.
In "Create a content pack" I filling field "Name", "Description", "Category", selecting needed "inpurs" and "grok pattern" (I don't have output, stream, a dashboard) and download content pack.
Next, on product system in "Content packs" - I pressed "import Content pack", select file and upload him.
Now, I select my content pack and press "apply content" and receive error... something like "error see logs"

Expected Behavior

I expected that import settings (inputs,extractors, grok patterns)

Current Behavior

in logs I saw the following:

2016-08-11T12:02:06.758+03:00 ERROR [BundleImporter] Error while creating dashboards. Starting rollback.
org.graylog2.plugin.database.ValidationException: Missing parameters source_field or target_field.
        at org.graylog2.bundles.BundleImporter.addExtractor(BundleImporter.java:313) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.addExtractors(BundleImporter.java:302) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.createMessageInput(BundleImporter.java:292) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.createInputs(BundleImporter.java:253) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:133) [graylog.jar:?]
...skip...
2016-08-11T12:02:06.759+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: ValidationException{message=Missing parameters source_field or target_field., errors={_=[ValidationFailed{passed=false, error=Missing parameters source_field or target_field.}]}}
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:142) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:112) ~[graylog.jar:?]
        at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:105) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.bundles.BundleResource.applyBundle(BundleResource.java:178) ~[graylog.jar:?]
..skip..

I understand that there are two errors:
1.) Wrong text in error description. I don't use dashboards (absolutely). In this place -

LOG.error("Error while creating dashboards. Starting rollback.", e);

2.) And main error - "Missing parameters source_field or target_field."
2.1.) In my json - target_field is empty:

    "inputs": [
        {
            "configuration": {
                "allow_override_date": true,
                "bind_address": "0.0.0.0",
                "expand_structured_data": false,
                "force_rdns": true,
                "override_source": null,
                "port": 51403,
                "recv_buffer_size": 262144,
                "store_full_message": false
            },
            "extractors": [
                {
                    "condition_type": "NONE",
                    "condition_value": "",
                    "configuration": {
                        "grok_pattern": "%{_swift_container}"
                    },
                    "converters": [],
                    "cursor_strategy": "COPY",
                    "order": 0,
                    "source_field": "message",
                    "target_field": "",
                    "title": "container message parse",
                    "type": "GROK"
                }
            ],
...skip...

2.2.) Next, I found it string

// grok extractor, for example, has no target field, so set it to source to satisf

That is, the field can be left blank?
Here - https://github.com/Graylog2/graylog2-web-interface/blob/eb084699edf5686bf124b4086d81ecfdc2e3fb2c/app/controllers/ExtractorsController.java#L137

2.3.) In this case, adding grok patter always is TRUE in this code:

        if (extractorDescription.getSourceField().isEmpty() || extractorDescription.getTargetField().isEmpty()) {
            throw new ValidationException("Missing parameters source_field or target_field.");
}

Here -

if (extractorDescription.getSourceField().isEmpty() || extractorDescription.getTargetField().isEmpty()) {

Possible Solution

I do not know much about the graylog - maybe it will be enough to change a logical expression.

Steps to Reproduce (for bugs)

  1. Create content pack with grok pattern
  2. Import this content pack
  3. Apple it.

Environment

  • Graylog Version: 2.0.3
  • Elasticsearch Version: 2.3.5
  • MongoDB Version: 3.2.8
  • Operating System: CentOS 7.2.1511
  • Browser version: FireFox 47.0.1 for linux
@joschi

This comment has been minimized.

Contributor

joschi commented Aug 11, 2016

@greenx Please attach the complete content pack which triggers the error.

@greenx

This comment has been minimized.

greenx commented Aug 11, 2016

attached in bottom

@greenx

This comment has been minimized.

greenx commented Aug 11, 2016

By the way, a rollback in case of failure does not work.
Each pattern must be deleted as much time, how many times tried to apply content pack!

@bernd bernd added the to-verify label Aug 15, 2016

@bernd bernd added this to the 2.1.0 milestone Aug 15, 2016

@bernd bernd added bug S2 P2 labels Aug 15, 2016

@edmundoa edmundoa self-assigned this Aug 15, 2016

@edmundoa

This comment has been minimized.

Member

edmundoa commented Aug 16, 2016

Hi @greenx,

Just to double check: does your _swift_expirer grok pattern has a new line in it? At least the content pack you posted in the issue has one.

@joschi

This comment has been minimized.

Contributor

joschi commented Aug 16, 2016

@greenx As a quick side note, you should upload the JSON file of the content pack and not paste it into the comment field as text.

@greenx

This comment has been minimized.

greenx commented Aug 16, 2016

@edmundoa

This comment has been minimized.

Member

edmundoa commented Aug 16, 2016

Thank you @greenx!

The new line was only in the copied text, I guess it was a typo while copy/pasting the long text.

I have been able to reproduce the issue you described, will create a patch for it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment