Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Malcolm v24.04.0 #466

Merged
merged 99 commits into from
Apr 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
8d01223
bump for v24.04.0 development
mmguero Mar 28, 2024
6255044
bump beats to v8.13.0 (https://www.elastic.co/guide/en/beats/libbeat/…
mmguero Mar 28, 2024
de3330c
Compose v2 ignores the top-level version element in docker-compose fi…
mmguero Mar 28, 2024
0fd6ab8
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Mar 29, 2024
af919bd
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Mar 29, 2024
0bce99d
fix hedgehog build
mmguero Mar 29, 2024
d0202b3
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Mar 29, 2024
ec5acf9
attempt to fix hedgheog iso build
mmguero Mar 29, 2024
7883057
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Mar 29, 2024
742c193
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
cedf0d6
workflow updates
mmguero Apr 1, 2024
6ddc051
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
6c0a9e7
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
5b2359c
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
695fe5d
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
625eab0
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
ffd9903
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
d319392
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
9a51ee4
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
4a21efc
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
c53fea7
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 1, 2024
d31a04a
added ARKIME_QUERY_ALL_INDICES env. variable to set queryAllIndices (…
mmguero Apr 2, 2024
6e84f82
bump beats to v8.13.1 (https://www.elastic.co/guide/en/beats/libbeat/…
mmguero Apr 2, 2024
fdf0afc
fixed assets directory on sensor
mmguero Apr 2, 2024
bebe677
show malcolm columns if malcolm redirect header is in place (idaholab…
mmguero Apr 2, 2024
d07440e
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 2, 2024
c58ccef
work in progress for idaholab/Malcolm#331, add transform vis. to File…
mmguero Apr 2, 2024
e77885f
work in progress for idaholab/Malcolm#331, add transform vis. to File…
mmguero Apr 2, 2024
03375b4
documentation and screenshot changes
mmguero Apr 3, 2024
127b5ff
documentation changes
mmguero Apr 3, 2024
8996d95
bump opensearch and dashboards to v2.13.0
mmguero Apr 3, 2024
fbfcba5
documentation changes
mmguero Apr 3, 2024
efd1800
testing out htadmin fixes
mmguero Apr 3, 2024
19ac5e1
point back at master branch for htadmin
mmguero Apr 4, 2024
46e4859
Bump Arkime to v5.1.0 (https://github.com/arkime/arkime/blob/2908ebf4…
mmguero Apr 4, 2024
4d6e4ed
make sure ja4+ plugin gets put into ini file correctly
mmguero Apr 4, 2024
26f6528
be smarter about how we read the docker storage driver
mmguero Apr 8, 2024
a6bf5f5
remove capture_loss and stats from LOGSTASH_ZEEK_IGNORED_LOGS
mmguero Apr 8, 2024
a02badb
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 8, 2024
1a4969c
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 8, 2024
c1f45a7
work in progress for idaholab/Malcolm#331, improvements to extracted_…
mmguero Apr 8, 2024
e4d5162
resolve idaholab/Malcolm#455, allow user to specify prefix for dashbo…
mmguero Apr 8, 2024
e38b00a
bump fluent-bit to v3.0.1
mmguero Apr 8, 2024
1a69873
for idaholab/Malcolm#453, allow specifying a maximum gigabytes thresh…
mmguero Apr 8, 2024
8379655
for idaholab/Malcolm#453, allow specifying a maximum gigabytes thresh…
mmguero Apr 8, 2024
9517b9a
for idaholab/Malcolm#453, allow specifying a maximum gigabytes thresh…
mmguero Apr 8, 2024
8250577
for idaholab/Malcolm#453, work in progress for pruning zeek extracted…
mmguero Apr 9, 2024
80cfb67
bump beats and logstash to v8.13.2
mmguero Apr 9, 2024
a0fcc31
install.py tweaks
mmguero Apr 9, 2024
1a42db0
documentation updates
mmguero Apr 9, 2024
97ebeef
for idaholab/Malcolm#456, adjustments to how Zeek intel files are gen…
mmguero Apr 10, 2024
3e18b88
handle ZEEK_INTEL_REFRESH_CRON_EXPRESSION on Hedgehog Linux installat…
mmguero Apr 10, 2024
cc846a0
documentation updates to go along with mmguero-dev/Malcolm@3e18b88b46…
mmguero Apr 10, 2024
b728879
documentation updates to go along with mmguero-dev/Malcolm@3e18b88b46…
mmguero Apr 10, 2024
dd92cef
documentation updates to go along with mmguero-dev/Malcolm@3e18b88b46…
mmguero Apr 10, 2024
154d3cd
for idaholab/Malcolm#456, use a unique string when generating the tem…
mmguero Apr 10, 2024
f2f4615
have zeek's crontab be in /opt/zeek so it can be edited in-place by n…
mmguero Apr 10, 2024
ea11612
little bit better cleanup for zeek_intel_setup.sh
mmguero Apr 10, 2024
c0bae29
for idaholab/Malcolm#421, review and update Windows prep and installa…
mmguero Apr 11, 2024
2ce8914
bump idna to 3.7 in hedgehog interface build
mmguero Apr 12, 2024
a12be91
bump fluent-bit to v3.0.2
mmguero Apr 12, 2024
641e0ef
bump gunicorn to 22.0.0 for CVE-2024-1135
mmguero Apr 17, 2024
7d44153
for idaholab/Malcolm#456, use zeekctl deploy rather than just zeekctl…
mmguero Apr 18, 2024
5439f6c
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 18, 2024
43f9087
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 18, 2024
d24177b
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 18, 2024
7f8bc5b
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
9c44fa9
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
0a6ac3f
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
fd2fbaa
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
4ca6d87
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
6647c1f
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
827227a
for idaholab/Malcolm#415, beginning work on using DNS for populating …
mmguero Apr 22, 2024
b669237
documentation updates
mmguero Apr 23, 2024
9d72051
for idaholab/Malcolm#415, work on using DNS for populating hostnames …
mmguero Apr 23, 2024
dc6bd91
quiet a few warnings
mmguero Apr 24, 2024
12d32eb
set category fields in default anomaly detectors to give a better br…
mmguero Apr 25, 2024
212c590
for idaholab/Malcolm#415, added exception debugging for for populatin…
mmguero Apr 25, 2024
b60715d
bump transform OSD version to 2.13.0
mmguero Apr 25, 2024
b7c674f
for idaholab/Malcolm#415, realizing there's an issue with devices vs.…
mmguero Apr 25, 2024
4acf58f
for idaholab/Malcolm#415, added debugging for for populating hostname…
mmguero Apr 25, 2024
263525b
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 25, 2024
fad7372
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 25, 2024
faabbcf
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 25, 2024
57c21bc
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 25, 2024
197c4a1
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 25, 2024
2e16a9d
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 26, 2024
653a644
make debug just a bit less verbose
mmguero Apr 26, 2024
df9123a
bump arkime to v5.1.2 (https://github.com/arkime/arkime/releases)
mmguero Apr 26, 2024
41f8bd6
for idaholab/Malcolm#415, work in progress on converting devices -> vms
mmguero Apr 26, 2024
8ca5572
fix verbosity of logs for netbox
mmguero Apr 26, 2024
354060b
fluent bit 3.0.3
mmguero Apr 29, 2024
d348266
work in progress for idaholab/Malcolm#208, improvements to creation o…
mmguero Apr 29, 2024
276502e
work in progress for idaholab/Malcolm#208, improvements to creation o…
mmguero Apr 29, 2024
4a74e7e
work in progress for idaholab/Malcolm#208, improvements to creation o…
mmguero Apr 29, 2024
43954a8
work in progress for idaholab/Malcolm#208, improvements to creation o…
mmguero Apr 29, 2024
f0104f7
work in progress for idaholab/Malcolm#208, improvements to creation o…
mmguero Apr 29, 2024
a7368de
restore image refererces for kubernetes
mmguero Apr 29, 2024
bceee46
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Apr 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/api-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/arkime-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-helper-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/file-upload-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/filebeat-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/freq-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@ jobs:
cp -r ./arkime/patch ./hedgehog-iso/shared/arkime_patch
mkdir -p ./hedgehog-iso/suricata
cp -r ./suricata/rules-default ./hedgehog-iso/suricata/
mkdir -p ./hedgehog-iso/nginx
cp -r ./nginx/landingpage/css ./hedgehog-iso/nginx/
cp -r ./nginx/landingpage/js ./hedgehog-iso/nginx/
pushd ./hedgehog-iso
echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/htadmin-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/logstash-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ on:
- 'malcolm-iso/**'
- 'shared/bin/*'
- '!shared/bin/configure-capture.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_iso_workflow_build'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/netbox-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/nginx-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/opensearch-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-capture-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/postgresql-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/redis-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/suricata-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/zeek-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ on:
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/extracted_files_http_server.py'
- '!shared/bin/web-ui-asset-download.sh'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
10 changes: 7 additions & 3 deletions Dockerfiles/arkime.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ ENV TERM xterm
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

ENV ARKIME_VERSION "v5.0.1"
ENV ARKIME_VERSION "v5.1.2"
ENV ARKIME_DIR "/opt/arkime"
ENV ARKIME_URL "https://github.com/arkime/arkime.git"
ENV ARKIME_LOCALELASTICSEARCH no
Expand All @@ -16,7 +16,8 @@ ENV ARKIME_INET yes
ADD arkime/scripts/bs4_remove_div.py /opt/
ADD arkime/patch/* /opt/patches/

RUN apt-get -q update && \
RUN export DEBARCH=$(dpkg --print-architecture) && \
apt-get -q update && \
apt-get -y -q --no-install-recommends upgrade && \
apt-get install -q -y --no-install-recommends \
binutils \
Expand Down Expand Up @@ -73,7 +74,10 @@ RUN apt-get -q update && \
make install && \
npm cache clean --force && \
rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"
bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded" && \
mkdir -p "${ARKIME_DIR}"/plugins && \
curl -fsSL -o "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" "https://github.com/arkime/arkime/releases/download/${ARKIME_VERSION}/ja4plus.${DEBARCH}.so" && \
chmod 755 "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so"

FROM debian:12-slim

Expand Down
4 changes: 2 additions & 2 deletions Dockerfiles/dashboards-helper.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ ADD scripts/malcolm_utils.py /data/

RUN apk update --no-cache && \
apk upgrade --no-cache && \
apk --no-cache add bash python3 py3-pip curl openssl procps psmisc npm rsync shadow jq tini && \
apk --no-cache add bash python3 py3-pip curl openssl procps psmisc moreutils npm rsync shadow jq tini && \
npm install -g http-server && \
pip3 install --break-system-packages supervisor humanfriendly requests && \
curl -fsSLO "$SUPERCRONIC_URL" && \
Expand Down Expand Up @@ -95,7 +95,7 @@ RUN apk update --no-cache && \
/opt/templates && \
chmod 755 /data/*.sh /data/*.py /data/init && \
chmod 400 /opt/maps/* && \
(echo -e "*/2 * * * * /data/create-arkime-sessions-index.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
(echo -e "*/2 * * * * /data/shared-object-creation.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})

EXPOSE $OFFLINE_REGION_MAPS_PORT

Expand Down
8 changes: 4 additions & 4 deletions Dockerfiles/dashboards.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM opensearchproject/opensearch-dashboards:2.12.0
FROM opensearchproject/opensearch-dashboards:2.13.0

LABEL maintainer="malcolm@inl.gov"
LABEL org.opencontainers.image.authors='malcolm@inl.gov'
Expand All @@ -20,7 +20,7 @@ ENV PUSER_PRIV_DROP true
ENV TERM xterm

ENV TINI_VERSION v0.19.0
ENV OSD_TRANSFORM_VIS_VERSION 2.12.0
ENV OSD_TRANSFORM_VIS_VERSION 2.13.0

ARG NODE_OPTIONS="--max_old_space_size=4096"
ENV NODE_OPTIONS $NODE_OPTIONS
Expand All @@ -40,8 +40,8 @@ RUN yum upgrade -y && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root && \
cd /tmp && \
# unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/package.json && \
# zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
cd /usr/share/opensearch-dashboards/plugins && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \
Expand Down
40 changes: 21 additions & 19 deletions Dockerfiles/file-monitor.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ ARG EXTRACTED_FILE_SCANNER_START_SLEEP=10
ARG EXTRACTED_FILE_LOGGER_START_SLEEP=5
ARG EXTRACTED_FILE_MIN_BYTES=64
ARG EXTRACTED_FILE_MAX_BYTES=134217728
ARG EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
ARG EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
ARG EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300
ARG VTOT_API2_KEY=0
ARG VTOT_REQUESTS_PER_MINUTE=4
ARG EXTRACTED_FILE_ENABLE_CLAMAV=false
Expand Down Expand Up @@ -65,6 +68,9 @@ ENV EXTRACTED_FILE_SCANNER_START_SLEEP $EXTRACTED_FILE_SCANNER_START_SLEEP
ENV EXTRACTED_FILE_LOGGER_START_SLEEP $EXTRACTED_FILE_LOGGER_START_SLEEP
ENV EXTRACTED_FILE_MIN_BYTES $EXTRACTED_FILE_MIN_BYTES
ENV EXTRACTED_FILE_MAX_BYTES $EXTRACTED_FILE_MAX_BYTES
ENV EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE $EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE
ENV EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT $EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT
ENV EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS $EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS
ENV VTOT_API2_KEY $VTOT_API2_KEY
ENV VTOT_REQUESTS_PER_MINUTE $VTOT_REQUESTS_PER_MINUTE
ENV EXTRACTED_FILE_ENABLE_CLAMAV $EXTRACTED_FILE_ENABLE_CLAMAV
Expand Down Expand Up @@ -103,6 +109,11 @@ ENV SUPERCRONIC_SHA1SUM "cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/
ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
COPY --chmod=755 shared/bin/web-ui-asset-download.sh /usr/local/bin/

RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
apt-get -q update && \
Expand All @@ -129,7 +140,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
pkg-config \
tini \
unzip && \
apt-get -y -q install \
apt-get -y -q install \
inotify-tools \
libzmq5 \
psmisc \
Expand All @@ -143,6 +154,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
python3 -m pip install --break-system-packages --no-compile --no-cache-dir \
clamd \
dominate \
humanfriendly \
psutil \
pycryptodome \
python-magic \
Expand Down Expand Up @@ -170,6 +182,8 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
rm -rf "${SRC_BASE_DIR}"/yara* && \
cd "${YARA_RULES_SRC_DIR}" && \
/usr/local/bin/yara_rules_setup.sh -r "${YARA_RULES_SRC_DIR}" -y "${YARA_RULES_DIR}" && \
cd /tmp && \
/usr/local/bin/web-ui-asset-download.sh -o "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css" && \
cd /tmp && \
curl -fsSL -o ./capa.zip "${CAPA_URL}" && \
unzip ./capa.zip && \
Expand All @@ -190,9 +204,6 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
libtool \
make \
python3-dev && \
apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* && \
mkdir -p /var/log/clamav "${CLAMAV_RULES_DIR}" && \
groupadd --gid ${DEFAULT_GID} ${PGROUP} && \
useradd -m --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
Expand All @@ -214,31 +225,22 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/clam_scan.py && \
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/yara_scan.py && \
ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/capa_scan.py && \
echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB}
echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB} && \
apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/*

USER ${PUSER}

RUN /usr/bin/freshclam freshclam --config-file=/etc/clamav/freshclam.conf

USER root

ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI9w2_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u8w4BMUTPHjxsAXC-v.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI5wq_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh7USSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6uyw4BMUTPHjx4wWw.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh6UVSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff2?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff2"
ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff"

COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
COPY --chmod=755 shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
COPY --chmod=755 shared/bin/prune_files.sh /usr/local/bin/
COPY --chmod=755 shared/bin/service_check_passthrough.sh /usr/local/bin/
COPY --chmod=755 shared/bin/zeek_carve*.py /usr/local/bin/
COPY --chmod=755 file-monitor/scripts/*.py /usr/local/bin/
COPY --chmod=755 shared/bin/extracted_files_http_server.py /usr/local/bin/
COPY --chmod=644 shared/bin/watch_common.py /usr/local/bin/
COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
COPY --chmod=644 file-monitor/supervisord.conf /etc/supervisord.conf
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/filebeat.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/beats/filebeat-oss:8.12.1
FROM docker.elastic.co/beats/filebeat-oss:8.13.2

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="malcolm@inl.gov"
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/logstash.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.elastic.co/logstash/logstash-oss:8.12.1
FROM docker.elastic.co/logstash/logstash-oss:8.13.2

LABEL maintainer="malcolm@inl.gov"
LABEL org.opencontainers.image.authors='malcolm@inl.gov'
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/opensearch.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM opensearchproject/opensearch:2.12.0
FROM opensearchproject/opensearch:2.13.0

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="malcolm@inl.gov"
Expand Down