[Zeek] Moving edge processing to ingest pipelines

[P1llus]

What does this PR do?

This PR moves all edge processing for the zeek filesets to ES ingest pipelines, adds pipeline tests to each fileset, and adds splunk data to each fileset golden files to test.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.

Related issues

• Closes Convert Zeek's edge processing to Ingest Node pipeline #664

[P1llus]

On the topic of splitting out the splunk specific parts into a specific pipeline, we decided to move that to a separate PR, once we have the possibility to store a single pipeline that can be shared between filesets.

[leehinman]

I tried a diff noticed a couple of things ( diff src )
zeek.irc.command is missing
zeek.irc.user is difference
zeek.notice.actions is new
zeek.notice.suppress_for now has decimal point
in http url.port removed
in http url.path added
in dhcp destination.port changed to 68
zeek.dhcp.domain missing
zeek.dhcp.lease_time now has decimal point
zeek.files.duration now has decimal point
zeek.files.md5 missing
zeek.files.mime_type missing
zeek.files.sha1 missing
and dns is different but I think that is expected.

[P1llus]

I tried a diff noticed a couple of things ( diff src )
zeek.irc.command is missing
zeek.irc.user is difference
zeek.notice.actions is new
zeek.notice.suppress_for now has decimal point
in http url.port removed
in http url.path added
in dhcp destination.port changed to 68
zeek.dhcp.domain missing
zeek.dhcp.lease_time now has decimal point
zeek.files.duration now has decimal point
zeek.files.md5 missing
zeek.files.mime_type missing
zeek.files.sha1 missing
and dns is different but I think that is expected.

Thanks for taking the time @leehinman . I think what happen here is that I renamed a few that was originally "converted". I was a bit unsure why we store the same data in both zeek fields and ECS fields. I am going through all the logs now and fixing up the small bits and pieces.

[P1llus]

Manually went through all the golden files and applied some small fixes. The DNS fileset can still be ignored for now.

The only issue left is a few float values that does not want to convert, so I removed the convert processors for now, maybe someone else has an idea, the issues are in:

zeek.dhcp.duration
zeek.dhcp.lease_time
zeek.notice.suppress_for
zeek.files.duration

Running a convert processor with type long on these would return errors like:
zeek/files test-files.log:
[0] unexpected pipeline error: For input string: \"0.0\"
[1] unexpected pipeline error: For input string: \"5.316734313964844E-5\"
zeek/notice test-notice.log:
[0] unexpected pipeline error: For input string: \"3600.0\"
zeek/files test-files.log:
[0] unexpected pipeline error: For input string: \"0.0\"
[1] unexpected pipeline error: For input string: \"5.316734313964844E-5\"
zeek/notice test-notice.log:
[0] unexpected pipeline error: For input string: \"3600.0\"

[P1llus]

Any handling of event.original, duplicate fields etc discussed earlier will be handled in a separate PR to this.

[andrewkroh]

The only issue left is a few float values that does not want to convert, so I removed the convert processors for now, maybe someone else has an idea, the issues are in:

I did a quick test with convert and had no issues with those float values as inputs. Are the quotes part of the input value?

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_source": {
          "float_output": 0.000053167343,
          "float_in": "5.316734313964844E-5"
        },
        "_ingest": {
          "timestamp": "2021-04-13T14:06:12.744211906Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_source": {
          "float_output": 0,
          "float_in": "0.0"
        },
        "_ingest": {
          "timestamp": "2021-04-13T14:06:12.744218811Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_source": {
          "float_output": 3600,
          "float_in": "3600.0"
        },
        "_ingest": {
          "timestamp": "2021-04-13T14:06:12.744221446Z"
        }
      }
    }
  ]
}

[andrewkroh]

Nice work, this is a big undertaking. Apart from DNS changes that are pending, LGTM.

[leehinman]

LGTM. awesome work.

[P1llus]

jenkins run tests