Skip to content

Commit

Permalink
work for 2.1.0 (#135)
Browse files Browse the repository at this point in the history
* added Sniffpass and HTTP-Attack plugins for zeek

* documentation updates

* clean up stuff from web generation we don't want

* removed fixed timezone from dashboards (?) and updated notices

* rudimentary detection of telnet protocol

* added telnet to list of insecure protocols

* bump version to 2.0.1 for development

* include basic telnet detection in sensor iso

* more work on the telnet detection feature

* more work on the telnet detection feature

* ensure local zeek policy gets set correctly

* issue #120, detect telnet, rsh, and rlogin traffic with zeek

Squashed commit of the following:

commit fb5c313
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Tue Apr 14 10:17:44 2020 -0600

    ensure local zeek policy gets set correctly

commit 7ca7add
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Tue Apr 14 10:01:11 2020 -0600

    more work on the telnet detection feature

commit d921cf9
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon Apr 13 16:29:45 2020 -0600

    more work on the telnet detection feature

commit b643c44
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon Apr 13 08:03:45 2020 -0600

    include basic telnet detection in sensor iso

commit 5952a30
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon Apr 13 07:14:19 2020 -0600

    bump version to 2.0.1 for development

commit ea06a8a
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon Apr 13 07:13:20 2020 -0600

    added telnet to list of insecure protocols

commit 3774c69
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Mon Apr 13 07:08:28 2020 -0600

    rudimentary detection of telnet protocol

commit 99a9710
Merge: e95d736 18b98db
Author: SG <13872653+mmguero@users.noreply.github.com>
Date:   Thu Apr 9 14:07:52 2020 -0600

    Merge remote-tracking branch 'upstream/development' into development

* according to semantic versioning, this version will be 2.1.0 since it introduces new backwards-compatible features

* update zeek to 3.0.4 to address a security vulnerability

* update documentation

* bump version to 2.0.1 for patch release for zeek 3.0.4 (see issue #123)

* meh, might as well be 3.0.5 with the compilation fix for older compilers

* meh, might as well be 3.0.5 with the compilation fix for older compilers

* added telnet/rsh/rlogin dashboard for idaholab#120

* update sha256 sums

* fix idaholab#122 by installing bro-xor-exe-plugin correctly

see also:

- corelight/zeek-xor-exe-plugin#2
- zeek/zeek#916

* added a build-time sanity check for the docker image to make sure all of the third-party plugins install and load correctly

* update version for docs

* include network visualization for possible use in future dashboards

* dockerfile cleanpu

* use Dockerfile ADD instead of 'git clone' to get certain repositories

* categorize xor-decrypted files by saving the original FUID in parent_fuid and normalizing the source

* make sure both original and decrypted FUID show up in notice log for pe_xor decrypted files

* fix recognition of names of file extracted by mitre-attack/bzar when scanned and triggering signatures

there are other extracted files that come from the mitre-attack/bzar scripts, they are formatted like this:

local fname = fmt("%s_%s%s", c$uid, f$id, subst_string(smb_name, "\\", "_"));

CR7X4q2hmcXKqP0vVj_F3jZ2VjYttqhKaGfh__172.16.1.8_C$_WINDOWS_sny4u_un1zbd94ytwj99hcymmsad7j54gr4wdskwnqs0ki252jdsrf763zsm531b.exe
└----------------┘ └---------------┘└------------------------------------------------------------------------------------------┘
        UID              FID          subst_string(smb_name, "\\", "_"))

(see https://github.com/mitre-attack/bzar/blob/master/scripts/bzar_files.bro#L50)

* make sure SNMP Registers actions (GetResponse, GetRequest, SetRequest, GetBulkRequest)

* added missing file for kibana plugin patch

* for idaholab#127, create a field mapping template for elasticsearch

* disabled by default, but starting to work on idaholab#79 mapping fields to ECS fields

* bump netsniff version to 0.6.7

* fix issue with defaults not being set right for ldap

* bump zeek version

* documentation updates

* documentation updates

* documentation updates

* documentation updates, and save hedgehog build artifacts

* documentation fixes

* documentation fixes

* documentation fixes

* bump moloch version to 2.3.0

* updated elasticsearch version, working on ecs fields

* more work on ecs normalization

* more work on ecs normalization

* Revert "updated elasticsearch version" due to discovery of elastic/elasticsearch#57006; should be fixed in 7.7.1

This partially reverts commit 4beaa09.

* update download shas

* update download shas

* added sankey visualization

* testing on my own fork

* sankey visualization fixes

* sankey visualization fixes

* added drilldown plugin for experimentation

* use fork of drilldown plugin

* specify nginx rewrite rule for idkib2mol to allow kibana -> moloch drilldowns

* for idaholab#133, specify drill-down mapping for zeek fields for kibana -> moloch drill-down

* for idaholab#133, handle strings correctly with quotes for moloch expression

* for idaholab#133, even though moloch fields won't map correctly (for now), still create URL drilldown mappings

* moloch test harness

* take ECS stuff out of development branch (will work on it in topic/ecs)

* take drilldown stuff out of development branch (will work on it in topic/drilldown)

* test harness

* use db: prefix for moloch (see arkime/arkime#1461) for constructing kibana -> moloch drilldown URLs

* kibana network visualization having issues with 7.7.1, disabling for now

* bump elasticsearch version to 7.7.1 and moloch version to 2.3.1

* added -w option to allow elasticsearch to be populated with logs before starting curator, elastalert

* Several of my kibana plugins are not working correctly in Kibana 7.7.x, so I am going to switch back to 7.6.x until I can work through those issues

* fix something borked by copy/paste

* fix install of drilldown plugin for 7.6.2

* have Kibana set up drilldown url mappings on each startup

* added some more drilldown links for kibana

* match drilldowns in moloch and kibana

* more working on drilldowns for common fields

* more working on drilldowns for common fields

* fix drilldowns from moloch side

* fix drilldowns on kibana side

* reduce verbosity of message

* fix drilldowns on kibana side

* update comments

* added plugin for zeek to detect cve_2020_0601

* update zeek to 3.0.7 (https://github.com/zeek/zeek/releases/tag/v3.0.7)

* added more actions (smtp, ssh, socks, ssl, rfb, etc.)

* more working on result normalization

* various fixes for results

* more working on result normalization

* more working on result normalization

* fix connection state map

* updated various dashboards to include result

* updated various dashboards to include result

* fix freq lookups by url encoding query parameters

* sort dns randomness charts correctly

* fix DNP3 IIN flags and ftp dashboard

* fix way with more recent vagrant/virtualbox for checking output from vagrant run

* fix issue applying iin_flags to action if they weren't specified

* fix issue applying iin_flags to action if they weren't specified

* more tweaks to dnp3 action/result

* dashboard tweaks

* dashboard fixes/cleanup

* fixes to HTTP And SNMP dashboards

* ended up with some bad JSON in a dashboard somehow :/

* fix issue with split pie charts in kibana

* fix issue with split pie charts in kibana
  • Loading branch information
mmguero committed Jun 25, 2020
1 parent 7459e01 commit 244d5aa
Show file tree
Hide file tree
Showing 104 changed files with 7,432 additions and 1,099 deletions.
2 changes: 1 addition & 1 deletion Dockerfiles/curator.Dockerfile
Expand Up @@ -69,7 +69,7 @@ RUN sed -i "s/buster main/buster main contrib non-free/g" /etc/apt/sources.list
apt-get -q -y autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
bash -c 'echo -e "${CRON} su -c \"/usr/local/bin/curator --config ${CONFIG_FILE} ${ACTION_FILE}\" ${CURATOR_USER} >/proc/1/fd/1 2>/proc/1/fd/2\n@reboot su -c \"/usr/local/bin/elastic_search_status.sh && /usr/local/bin/register-elasticsearch-snapshot-repo.sh\" ${CURATOR_USER} >/proc/1/fd/1 2>/proc/1/fd/2" | crontab -'
bash -c 'echo -e "${CRON} su -c \"/usr/local/bin/curator --config ${CONFIG_FILE} ${ACTION_FILE}\" ${CURATOR_USER} >/proc/1/fd/1 2>/proc/1/fd/2\n@reboot su -c \"/usr/local/bin/elastic_search_status.sh -w && /usr/local/bin/register-elasticsearch-snapshot-repo.sh\" ${CURATOR_USER} >/proc/1/fd/1 2>/proc/1/fd/2" | crontab -'

ADD shared/bin/cron_env_deb.sh /usr/local/bin/
ADD shared/bin/elastic_search_status.sh /usr/local/bin/
Expand Down
90 changes: 67 additions & 23 deletions Dockerfiles/kibana.Dockerfile
Expand Up @@ -32,21 +32,7 @@ ENV ELASTICSEARCH_URL $ELASTICSEARCH_URL

USER root

RUN yum install -y epel-release && \
yum update -y && \
yum install -y curl cronie inotify-tools npm psmisc python-requests python-setuptools zip unzip && \
yum clean all && \
easy_install supervisor && \
npm install -g http-server

ADD kibana/scripts /data/
ADD shared/bin/elastic_search_status.sh /data/
ADD shared/bin/cron_env_centos.sh /data/
ADD kibana/kibana-standard.yml /opt/kibana/config/kibana-standard.yml
ADD kibana/kibana-offline-maps.yml /opt/kibana/config/kibana-offline-maps.yml
ADD kibana/supervisord.conf /etc/supervisord.conf
ADD kibana/dashboards /opt/kibana/dashboards
ADD kibana/maps /opt/maps
ADD kibana/plugin-patches /tmp/plugin-patches
ADD kibana/elastalert-kibana-plugin/server/routes/elastalert.js /tmp/elastalert-server-routes.js

# todo: these extra plugins are kind of gutted right now with 7.x, need to fix
Expand All @@ -71,14 +57,20 @@ ADD kibana/elastalert-kibana-plugin/server/routes/elastalert.js /tmp/elastalert-
# /usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-calendar.zip --allow-root && \
# rm -rf /tmp/kibana-calendar.zip /tmp/kibana && \

RUN curl -sSL -o /tmp/kibana-comments.zip "https://github.com/gwintzer/kibana-comments-app-plugin/releases/download/7.4.0/kibana-comments-app-plugin-7.4.0-latest.zip" && \
RUN sed -i "s/d\.name\.split/d\.name\.toString()\.split/" /usr/share/kibana/src/legacy/ui/public/vislib/visualizations/pie_chart.js && \
curl -sSL -o /tmp/kibana-comments.zip "https://github.com/gwintzer/kibana-comments-app-plugin/releases/download/7.4.0/kibana-comments-app-plugin-7.4.0-latest.zip" && \
curl -sSL -o /tmp/kibana-swimlane.zip "https://github.com/prelert/kibana-swimlane-vis/releases/download/v7.6.2/prelert_swimlane_vis-7.6.2.zip" && \
curl -sSL -o /tmp/elastalert-kibana-plugin.zip "https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip" && \
chmod 755 /data/*.sh /data/*.py && \
chown -R kibana:kibana /opt/kibana/dashboards /opt/maps /opt/kibana/config/kibana*.yml && \
chmod 400 /opt/maps/* && \
mkdir -p /var/log/supervisor && \
(echo -e "*/2 * * * * su -c /data/kibana-create-moloch-sessions-index.sh kibana >/dev/null 2>&1\n0 * * * * su -c /data/kibana_index_refresh.py kibana >/dev/null 2>&1\n" | crontab -) && \
curl -sSL -o /tmp/kibana-network.zip "https://codeload.github.com/dlumbrer/kbn_network/zip/7-dev" && \
curl -sSL -o /tmp/kibana-sankey.zip "https://codeload.github.com/mmguero-dev/kbn_sankey_vis/zip/master" && \
curl -sSL -o /tmp/kibana-drilldown.zip "https://codeload.github.com/mmguero-dev/kibana-plugin-drilldownmenu/zip/master" && \
yum install -y epel-release && \
yum update -y && \
yum install -y curl cronie inotify-tools npm patch psmisc python-requests python-setuptools zip unzip && \
yum clean all && \
easy_install supervisor && \
npm install -g http-server && \
mkdir -p /var/log/supervisor && \
cd /tmp && \
echo "Installing ElastAlert plugin..." && \
unzip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/public/components/main/main.js && \
Expand All @@ -93,6 +85,43 @@ RUN curl -sSL -o /tmp/kibana-comments.zip "https://github.com/gwintzer/kibana-co
cd /usr/share/kibana/plugins && \
/usr/share/kibana/bin/kibana-plugin install file:///tmp/elastalert-kibana-plugin.zip --allow-root && \
rm -rf /tmp/elastalert-kibana-plugin.zip /tmp/elastalert.js /tmp/kibana && \
cd /tmp && \
echo "Installing Sankey visualization..." && \
unzip /tmp/kibana-sankey.zip && \
mkdir ./kibana &&\
mv ./kbn_sankey_vis-* ./kibana/sankey_vis && \
cd ./kibana/sankey_vis && \
sed -i "s/7\.6\.3/7\.6\.2/g" ./package.json && \
npm install && \
cd /tmp && \
zip -r sankey_vis.zip kibana --exclude ./kibana/sankey_vis/.git\* && \
cd /usr/share/kibana/plugins && \
/usr/share/kibana/bin/kibana-plugin install file:///tmp/sankey_vis.zip --allow-root && \
rm -rf /tmp/kibana /tmp/*sankey* && \
cd /tmp && \
echo "Installing Drilldown menu plugin..." && \
unzip /tmp/kibana-drilldown.zip && \
mkdir ./kibana &&\
mv ./kibana-plugin-drilldownmenu-* ./kibana/kibana-plugin-drilldownmenu && \
cd ./kibana/kibana-plugin-drilldownmenu && \
sed -i "s/7\.6\.2/7\.6\.2/g" ./package.json && \
npm install && \
cd /tmp && \
zip -r drilldown.zip kibana --exclude ./kibana/kibana-plugin-drilldownmenu/.git\* && \
cd /usr/share/kibana/plugins && \
/usr/share/kibana/bin/kibana-plugin install file:///tmp/drilldown.zip --allow-root && \
rm -rf /tmp/kibana /tmp/*drilldown* && \
cd /tmp && \
echo "Installing Network visualization..." && \
cd /usr/share/kibana/plugins && \
unzip /tmp/kibana-network.zip && \
mv ./kbn_network-* ./network_vis && \
cd ./network_vis && \
sed -i "s/7\.5\.2/7\.6\.2/g" ./package.json && \
rm -rf ./images && \
patch -p 1 < /tmp/plugin-patches/kbn_network_7.6.x.patch && \
npm install && \
rm -rf /tmp/kibana-network.zip && \
cd /tmp && \
echo "Installing Comments visualization..." && \
unzip kibana-comments.zip kibana/kibana-comments-app-plugin/package.json && \
Expand All @@ -109,10 +138,25 @@ RUN curl -sSL -o /tmp/kibana-comments.zip "https://github.com/gwintzer/kibana-co
cd /usr/share/kibana/plugins && \
/usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-swimlane.zip --allow-root && \
bash -c "find /usr/share/kibana/plugins/prelert_swimlane_vis/ -type f -exec chmod 644 '{}' \;" && \
rm -rf /tmp/kibana-swimlane.zip /tmp/kibana
rm -rf /tmp/kibana-swimlane.zip /tmp/kibana && \
rm -rf /tmp/plugin-patches /tmp/elastalert-server-routes.js /tmp/npm-*

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "-u", "root", "-n"]
ADD kibana/dashboards /opt/kibana/dashboards
ADD kibana/kibana-offline-maps.yml /opt/kibana/config/kibana-offline-maps.yml
ADD kibana/kibana-standard.yml /opt/kibana/config/kibana-standard.yml
ADD kibana/maps /opt/maps
ADD kibana/scripts /data/
ADD kibana/supervisord.conf /etc/supervisord.conf
ADD kibana/zeek_template.json /data/zeek_template.json
ADD shared/bin/cron_env_centos.sh /data/
ADD shared/bin/elastic_search_status.sh /data/

RUN chmod 755 /data/*.sh /data/*.py && \
chown -R kibana:kibana /opt/kibana/dashboards /opt/maps /opt/kibana/config/kibana*.yml && \
chmod 400 /opt/maps/* && \
(echo -e "*/2 * * * * su -c /data/kibana-create-moloch-sessions-index.sh kibana >/dev/null 2>&1\n0 10 * * * su -c /data/kibana_index_refresh.py kibana >/dev/null 2>&1\n" | crontab -)

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf", "-u", "root", "-n"]

# to be populated at build-time:
ARG BUILD_DATE
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/moloch.Dockerfile
Expand Up @@ -4,7 +4,7 @@ FROM debian:buster-slim AS build

ENV DEBIAN_FRONTEND noninteractive

ENV MOLOCH_VERSION "2.3.0"
ENV MOLOCH_VERSION "2.3.1"
ENV MOLOCHDIR "/data/moloch"
ENV MOLOCH_URL "https://codeload.github.com/aol/moloch/tar.gz/v${MOLOCH_VERSION}"

Expand Down
8 changes: 4 additions & 4 deletions Dockerfiles/zeek.Dockerfile
Expand Up @@ -16,7 +16,7 @@ ENV SRC_BASE_DIR "/usr/local/src"
ENV ZEEK_DIR "/opt/zeek"
ENV ZEEK_PATCH_DIR "${SRC_BASE_DIR}/zeek-patches"
ENV ZEEK_SRC_DIR "${SRC_BASE_DIR}/zeek-${ZEEK_VERSION}"
ENV ZEEK_VERSION "3.0.6"
ENV ZEEK_VERSION "3.0.7"

# using clang now instead of gcc because Spicy depends on it
ENV LLVM_VERSION "10"
Expand Down Expand Up @@ -152,13 +152,13 @@ ADD zeek/config/*.zeek ${ZEEK_DIR}/share/zeek/site/

# sanity check to make sure the plugins installed and copied over correctly
# these ENVs should match the number of third party plugins installed by zeek_install_plugins.sh
ENV ZEEK_THIRD_PARTY_PLUGINS_COUNT 18
ENV ZEEK_THIRD_PARTY_GREP_STRING "(Bro_LDAP/scripts/main|Corelight/PE_XOR/main|Salesforce/GQUIC/main|Zeek_AF_Packet/scripts/init|bzar/main|cve-2020-0601/cve-2020-0601|hassh/hassh|ja3/ja3|zeek-community-id/main|zeek-EternalSafety/main|zeek-httpattacks/main|zeek-plugin-bacnet/main|zeek-plugin-enip/main|zeek-plugin-profinet/main|zeek-plugin-s7comm/main|zeek-plugin-tds/main|zeek-sniffpass/main|spicy/main)\.(zeek|bro)"
ENV ZEEK_THIRD_PARTY_PLUGINS_COUNT 19
ENV ZEEK_THIRD_PARTY_GREP_STRING "(Bro_LDAP/scripts/main|Corelight/PE_XOR/main|Salesforce/GQUIC/main|Zeek_AF_Packet/scripts/init|bzar/main|cve-2020-0601/cve-2020-0601|cve-2020-13777/cve-2020-13777|hassh/hassh|ja3/ja3|zeek-community-id/main|zeek-EternalSafety/main|zeek-httpattacks/main|zeek-plugin-bacnet/main|zeek-plugin-enip/main|zeek-plugin-profinet/main|zeek-plugin-s7comm/main|zeek-plugin-tds/main|zeek-sniffpass/main|spicy/main)\.(zeek|bro)"

RUN mkdir -p /tmp/logs && \
cd /tmp/logs && \
$ZEEK_DIR/bin/zeek -C -r /tmp/pcaps/udp.pcap local policy/misc/loaded-scripts 2>/dev/null && \
bash -c "(( $(grep -cP "$ZEEK_THIRD_PARTY_GREP_STRING" loaded_scripts.log) == $ZEEK_THIRD_PARTY_PLUGINS_COUNT)) && echo 'Zeek plugins loaded correctly' || (echo 'One or more Zeek plugins did not load correctly' && exit 1)" && \
bash -c "(( $(grep -cP "$ZEEK_THIRD_PARTY_GREP_STRING" loaded_scripts.log) == $ZEEK_THIRD_PARTY_PLUGINS_COUNT)) && echo 'Zeek plugins loaded correctly' || (echo 'One or more Zeek plugins did not load correctly' && cat loaded_scripts.log && exit 1)" && \
cd /tmp && \
rm -rf /tmp/logs /tmp/pcaps

Expand Down

0 comments on commit 244d5aa

Please sign in to comment.