Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create GROK pattern with underscores in field name. #5563

Closed
ddimick opened this Issue Jan 21, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@ddimick
Copy link

ddimick commented Jan 21, 2019

Expected Behavior

Graylog should accept underscores in field name and create the grok pattern.

Current Behavior

Graylog returns "Could not test Grok pattern
Testing Grok pattern "test" failed with status: cannot POST https://graylog.my.domain/api/system/grok (400)"

Steps to Reproduce (for bugs)

Create a grok pattern like (?<test_field>test), it will fail. Remove the underscore and create (?<testfield>test), it will succeed.

Your Environment

  • Graylog Version: 3.0.0-beta.3-1
  • Elasticsearch Version: 6.5.4
  • MongoDB Version: 4.0.5
  • Operating System: Server CoreOS 4.14.88, docker 18.06.1-ce, client Windows 10 Pro
  • Browser version: Chrome 71.0.3578.98, Firefox 62.0.3 and Edge 42.17134.1.0
@kmerz

This comment has been minimized.

Copy link
Member

kmerz commented Jan 22, 2019

Unfortunately this is not fixable. The java Pattern matcher does not allow to have underscores in its named groups:

https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html#groupname

A capturing group can also be assigned a "name", a named-capturing group, and then be back-
referenced later by the "name". Group names are composed of the following characters. The first
character must be a letter.

The uppercase letters 'A' through 'Z' ('\u0041' through '\u005a'),
The lowercase letters 'a' through 'z' ('\u0061' through '\u007a'),
The digits '0' through '9' ('\u0030' through '\u0039'),

The only alternative would be to implement our own Pattern Matcher which would be slightly out of scope.

@kmerz kmerz closed this Jan 22, 2019

bernd added a commit that referenced this issue Mar 25, 2019

Switch back to a repackaged and fixed version of java-grok
To support underscores ("_") in Grok match group names, we had to modify
the java-grok library to use the old regexp engine again.

See: graylog-labs/java-grok#2

This also adds a test for the Grok extractor to make sure that using
underscores works.

Fixes #5704
Fixes #5563

kmerz added a commit that referenced this issue Mar 26, 2019

Switch back to a repackaged and fixed version of java-grok (#5800)
* Switch back to a repackaged and fixed version of java-grok

To support underscores ("_") in Grok match group names, we had to modify
the java-grok library to use the old regexp engine again.

See: graylog-labs/java-grok#2

This also adds a test for the Grok extractor to make sure that using
underscores works.

Fixes #5704
Fixes #5563

* Fix GrokPatternService#extractPatternNames and add a test for it

* Add missing license header to GrokPatternServiceTest

* Add test for named group with underscore

Prior to this change, there was no test for named groups
with underscores in the FunctionSnippetsTest

This change enhances the grok() test to run with a
named group with underscore.
@bernd

This comment has been minimized.

Copy link
Member

bernd commented Mar 26, 2019

@ddimick We fixed this issue in master and it will be backported into the upcoming 3.0.1. release.

@bernd bernd self-assigned this Mar 26, 2019

@bernd bernd added bug triaged labels Mar 26, 2019

bernd added a commit that referenced this issue Mar 26, 2019

Switch back to a repackaged and fixed version of java-grok (#5800)
* Switch back to a repackaged and fixed version of java-grok

To support underscores ("_") in Grok match group names, we had to modify
the java-grok library to use the old regexp engine again.

See: graylog-labs/java-grok#2

This also adds a test for the Grok extractor to make sure that using
underscores works.

Fixes #5704
Fixes #5563

* Fix GrokPatternService#extractPatternNames and add a test for it

* Add missing license header to GrokPatternServiceTest

* Add test for named group with underscore

Prior to this change, there was no test for named groups
with underscores in the FunctionSnippetsTest

This change enhances the grok() test to run with a
named group with underscore.

(cherry picked from commit e642a41)

kmerz added a commit that referenced this issue Mar 26, 2019

Switch back to a repackaged and fixed version of java-grok (#5800) (#…
…5807)

* Switch back to a repackaged and fixed version of java-grok

To support underscores ("_") in Grok match group names, we had to modify
the java-grok library to use the old regexp engine again.

See: graylog-labs/java-grok#2

This also adds a test for the Grok extractor to make sure that using
underscores works.

Fixes #5704
Fixes #5563

* Fix GrokPatternService#extractPatternNames and add a test for it

* Add missing license header to GrokPatternServiceTest

* Add test for named group with underscore

Prior to this change, there was no test for named groups
with underscores in the FunctionSnippetsTest

This change enhances the grok() test to run with a
named group with underscore.

(cherry picked from commit e642a41)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.