FEATURE : Recursive pattern for Grok

[M0dM]

Hi,

I didn't arrived to use recursivity inside grok custom patterns.
I think this could be an awesome feature.

Benoit

Description :

Grok pattern matching the two following lines :

2014-07-11 18:26:21,335 - INFO  - 1712933>-<>-<text1>-<>-<text2

2014-07-11 18:26:21,335 - INFO  - 1712933>-<>-<text1>-<>-<text2>-<>-<text3

I want to match both of the lines and extract data like this :

%{CUSTOM_DATE}[\s-]*%{LOGLEVEL}[\s-]*%{POSINT}%{AMA_VALUES_LIST_DATA}

CUSTOM_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
CUSTOM_VALUE ((?!>-<>-<).)*
CUSTOM_LIST_VALUE >-<>-<%{CUSTOM_VALUE}
CUSTOM_VALUES_LIST_COMPLEX %{CUSTOM_LIST_VALUE}%           {CUSTOM_LIST_VALUE_COMPLEX} | %{CUSTOM_LIST_VALUE}

What I would like to get :

     {
      "CUSTOM_DATE": [
        [
          "2014-07-11 18:26:21,335"
        ]
      ],
      "YEAR": [
        [
          "2014"
        ]
      ],
      "MONTHNUM": [
        [
          "07"
        ]
      ],
      "MONTHDAY": [
        [
          "11"
        ]
      ],
      "HOUR": [
        [
          "18"
        ]
      ],
      "MINUTE": [
        [
          "26"
        ]
      ],
      "SECOND": [
        [
          "21,335"
        ]
      ],
      "LOGLEVEL": [
        [
          "INFO"
        ]
      ],
      "POSINT": [
        [
          "1712933"
        ]
      ],
      "CUSTOM_LIST_COMPLEX": [
        [
          ">-<>-<text1>-<>-<text2>-<>-<text3"
        ]
      ],
      "CUSTOM_LIST_VALUE": [
        [
          ">-<>-<text1",
          ">-<>-<text2",
          ">-<>-<text3"
        ]
      ]
      "CUSTOM_VALUE": [
        [
          "text1",
          "text2",
          "text3"
        ]
      ]
    }

[jordansissel]

For Logstash 1.5.0, we've moved all plugins to individual repositories, so I have moved this issue to logstash-plugins/logstash-filter-grok#35. Let's continue the discussion there! :)