create a field mappings template for elasticsearch #127
Assignees
Comments
mmguero
added
elastic
mmguero
referenced
this issue
Jun 25, 2020
* added Sniffpass and HTTP-Attack plugins for zeek * documentation updates * clean up stuff from web generation we don't want * removed fixed timezone from dashboards (?) and updated notices * rudimentary detection of telnet protocol * added telnet to list of insecure protocols * bump version to 2.0.1 for development * include basic telnet detection in sensor iso * more work on the telnet detection feature * more work on the telnet detection feature * ensure local zeek policy gets set correctly * issue #120, detect telnet, rsh, and rlogin traffic with zeek Squashed commit of the following: commit fb5c313 Author: SG <13872653+mmguero@users.noreply.github.com> Date: Tue Apr 14 10:17:44 2020 -0600 ensure local zeek policy gets set correctly commit 7ca7add Author: SG <13872653+mmguero@users.noreply.github.com> Date: Tue Apr 14 10:01:11 2020 -0600 more work on the telnet detection feature commit d921cf9 Author: SG <13872653+mmguero@users.noreply.github.com> Date: Mon Apr 13 16:29:45 2020 -0600 more work on the telnet detection feature commit b643c44 Author: SG <13872653+mmguero@users.noreply.github.com> Date: Mon Apr 13 08:03:45 2020 -0600 include basic telnet detection in sensor iso commit 5952a30 Author: SG <13872653+mmguero@users.noreply.github.com> Date: Mon Apr 13 07:14:19 2020 -0600 bump version to 2.0.1 for development commit ea06a8a Author: SG <13872653+mmguero@users.noreply.github.com> Date: Mon Apr 13 07:13:20 2020 -0600 added telnet to list of insecure protocols commit 3774c69 Author: SG <13872653+mmguero@users.noreply.github.com> Date: Mon Apr 13 07:08:28 2020 -0600 rudimentary detection of telnet protocol commit 99a9710 Merge: e95d736 18b98db Author: SG <13872653+mmguero@users.noreply.github.com> Date: Thu Apr 9 14:07:52 2020 -0600 Merge remote-tracking branch 'upstream/development' into development * according to semantic versioning, this version will be 2.1.0 since it introduces new backwards-compatible features * update zeek to 3.0.4 to address a security vulnerability * update documentation * bump version to 2.0.1 for patch release for zeek 3.0.4 (see issue #123) * meh, might as well be 3.0.5 with the compilation fix for older compilers * meh, might as well be 3.0.5 with the compilation fix for older compilers * added telnet/rsh/rlogin dashboard for idaholab#120 * update sha256 sums * fix idaholab#122 by installing bro-xor-exe-plugin correctly see also: - corelight/zeek-xor-exe-plugin#2 - zeek/zeek#916 * added a build-time sanity check for the docker image to make sure all of the third-party plugins install and load correctly * update version for docs * include network visualization for possible use in future dashboards * dockerfile cleanpu * use Dockerfile ADD instead of 'git clone' to get certain repositories * categorize xor-decrypted files by saving the original FUID in parent_fuid and normalizing the source * make sure both original and decrypted FUID show up in notice log for pe_xor decrypted files * fix recognition of names of file extracted by mitre-attack/bzar when scanned and triggering signatures there are other extracted files that come from the mitre-attack/bzar scripts, they are formatted like this: local fname = fmt("%s_%s%s", c$uid, f$id, subst_string(smb_name, "\\", "_")); CR7X4q2hmcXKqP0vVj_F3jZ2VjYttqhKaGfh__172.16.1.8_C$_WINDOWS_sny4u_un1zbd94ytwj99hcymmsad7j54gr4wdskwnqs0ki252jdsrf763zsm531b.exe └----------------┘ └---------------┘└------------------------------------------------------------------------------------------┘ UID FID subst_string(smb_name, "\\", "_")) (see https://github.com/mitre-attack/bzar/blob/master/scripts/bzar_files.bro#L50) * make sure SNMP Registers actions (GetResponse, GetRequest, SetRequest, GetBulkRequest) * added missing file for kibana plugin patch * for idaholab#127, create a field mapping template for elasticsearch * disabled by default, but starting to work on idaholab#79 mapping fields to ECS fields * bump netsniff version to 0.6.7 * fix issue with defaults not being set right for ldap * bump zeek version * documentation updates * documentation updates * documentation updates * documentation updates, and save hedgehog build artifacts * documentation fixes * documentation fixes * documentation fixes * bump moloch version to 2.3.0 * updated elasticsearch version, working on ecs fields * more work on ecs normalization * more work on ecs normalization * Revert "updated elasticsearch version" due to discovery of elastic/elasticsearch#57006; should be fixed in 7.7.1 This partially reverts commit 4beaa09. * update download shas * update download shas * added sankey visualization * testing on my own fork * sankey visualization fixes * sankey visualization fixes * added drilldown plugin for experimentation * use fork of drilldown plugin * specify nginx rewrite rule for idkib2mol to allow kibana -> moloch drilldowns * for idaholab#133, specify drill-down mapping for zeek fields for kibana -> moloch drill-down * for idaholab#133, handle strings correctly with quotes for moloch expression * for idaholab#133, even though moloch fields won't map correctly (for now), still create URL drilldown mappings * moloch test harness * take ECS stuff out of development branch (will work on it in topic/ecs) * take drilldown stuff out of development branch (will work on it in topic/drilldown) * test harness * use db: prefix for moloch (see arkime/arkime#1461) for constructing kibana -> moloch drilldown URLs * kibana network visualization having issues with 7.7.1, disabling for now * bump elasticsearch version to 7.7.1 and moloch version to 2.3.1 * added -w option to allow elasticsearch to be populated with logs before starting curator, elastalert * Several of my kibana plugins are not working correctly in Kibana 7.7.x, so I am going to switch back to 7.6.x until I can work through those issues * fix something borked by copy/paste * fix install of drilldown plugin for 7.6.2 * have Kibana set up drilldown url mappings on each startup * added some more drilldown links for kibana * match drilldowns in moloch and kibana * more working on drilldowns for common fields * more working on drilldowns for common fields * fix drilldowns from moloch side * fix drilldowns on kibana side * reduce verbosity of message * fix drilldowns on kibana side * update comments * added plugin for zeek to detect cve_2020_0601 * update zeek to 3.0.7 (https://github.com/zeek/zeek/releases/tag/v3.0.7) * added more actions (smtp, ssh, socks, ssl, rfb, etc.) * more working on result normalization * various fixes for results * more working on result normalization * more working on result normalization * fix connection state map * updated various dashboards to include result * updated various dashboards to include result * fix freq lookups by url encoding query parameters * sort dns randomness charts correctly * fix DNP3 IIN flags and ftp dashboard * fix way with more recent vagrant/virtualbox for checking output from vagrant run * fix issue applying iin_flags to action if they weren't specified * fix issue applying iin_flags to action if they weren't specified * more tweaks to dnp3 action/result * dashboard tweaks * dashboard fixes/cleanup * fixes to HTTP And SNMP dashboards * ended up with some bad JSON in a dashboard somehow :/ * fix issue with split pie charts in kibana * fix issue with split pie charts in kibana
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now almost all of the fields we're populating via logstash are being done with dynamic mappings. There are reasons not to do this.
We're already maintaining some of these different fields elsewhere (wise, zeek-log-fields.json, 11_zeek_logs.conf and the other conf files in the pipelines), but we need to tell elasticsearch about these fields to avoid conflicts and store/search things efficiently.
We already have the beginnings of this file: zeek_template.json.
It would be nice if there was a way we could auto-create this file. Like, I wonder if I can create a script that reads (certain values from) the wise.js file to create the template file... The only problem is that some of the files should be text and some are keywords so I'm not sure there'd be a straightforward way to do it. We may just have to keep the entire lists in two places (basically the wise.js and the zeek_template.json; the zeek-log-fields.json is only a small subset) and make sure we update both of them when needed.
This may also relate to issue #79 and #65.
The text was updated successfully, but these errors were encountered: