idaholab#475, more zeek deploy changes for supporting more loggers

docs/live-analysis.md

@@ -84,14 +84,20 @@ The relevant environment variables related to tuning Zeek for live packet captur
 
 - `ZEEK_AF_PACKET_BUFFER_SIZE` - AF_Packet [ring buffer size](https://docs.zeek.org/en/master/scripts/builtin-plugins/Zeek_AF_Packet/init.zeek.html#id-AF_Packet::buffer_size) in bytes (default `67108864`)
 - `ZEEK_AF_PACKET_FANOUT_MODE` - AF_Packet [fanout mode](https://docs.zeek.org/en/master/scripts/base/bif/plugins/Zeek_AF_Packet.af_packet.bif.zeek.html#type-AF_Packet::FanoutMode) (default `FANOUT_HASH`)
-- `ZEEK_LB_PROCS_WORKER_DEFAULT` - ["Zeek is not multithreaded, so once the limitations of a single processor core are reached the only option currently is to spread the workload across many cores"](https://docs.zeek.org/en/master/cluster-setup.html#cluster-architecture). This value defines the number of processors to be assigned to each group of workers created for each capture interface for [load balancing](https://docs.zeek.org/en/master/cluster-setup.html#load-balancing) (default `1`). A value of `0` means "autocalculate based on the number of CPUs present in the system."
-- `ZEEK_LB_PROCS_CPUS_RESERVED` - If `ZEEK_LB_PROCS_WORKER_DEFAULT` is `0` ("autocalculate"), exclude this number of CPUs from the autocalculation (defaults to `4` for the kernel, and for Zeek's manager, proxy, and logger processes)
-- `ZEEK_PIN_CPUS_WORKER_AUTO` - Automatically [pin worker CPUs](https://en.wikipedia.org/wiki/Processor_affinity) (default `false`)
+- `ZEEK_LB_PROCS_WORKER_DEFAULT` - ["Zeek is not multithreaded, so once the limitations of a single processor core are reached the only option currently is to spread the workload across many cores"](https://docs.zeek.org/en/master/cluster-setup.html#cluster-architecture). This value defines the number of processors to be assigned to each group of [workers](https://docs.zeek.org/en/master/frameworks/cluster.html#worker) created for each capture interface for [load balancing](https://docs.zeek.org/en/master/cluster-setup.html#load-balancing) (default `1`). A value of `0` means "autocalculate based on the number of CPUs present in the system."
 - `ZEEK_LB_PROCS_WORKER_n` - Explicitly defines the number of processor to be assigned to the group of workers for the *n*-th capture interface. If unspecified this defaults to the number of CPUs `ZEEK_PIN_CPUS_WORKER_n` if defined, or `ZEEK_LB_PROCS_WORKER_DEFAULT` otherwise.
+- `ZEEK_LB_PROCS_CPUS_RESERVED` - If `ZEEK_LB_PROCS_WORKER_DEFAULT` is `0` ("autocalculate"), exclude this number of CPUs from the autocalculation (defaults to `1` (kernel) + `1` (manager) + `ZEEK_LB_PROCS_LOGGER` + `ZEEK_LB_PROCS_PROXY`)
+- `ZEEK_PIN_CPUS_WORKER_AUTO` - Automatically [pin worker CPUs](https://en.wikipedia.org/wiki/Processor_affinity) (default `false`)
 - `ZEEK_PIN_CPUS_WORKER_n` - Explicitly defines the processor IDs to be to be assigned to the group of workers for the *n*-th capture interface (e.g., `0` means "the first CPU"; `12,13,14,15` means "the last four CPUs" on a 16-core system)
-- `ZEEK_PIN_CPUS_LOGGER` - list of CPUs to pin for [zeekctl's](https://github.com/zeek/zeekctl?tab=readme-ov-file#configuration) logger process, or `true` to auto-pin one CPU (default is unset)
-- `ZEEK_PIN_CPUS_MANAGER` - list of CPUs to pin for [zeekctl's](https://github.com/zeek/zeekctl?tab=readme-ov-file#configuration) manager process, or `true` to auto-pin one CPU (default is unset)
-- `ZEEK_PIN_CPUS_PROXY` - list of CPUs to pin for [zeekctl's](https://github.com/zeek/zeekctl?tab=readme-ov-file#configuration) proxy process, or `true` to auto-pin one CPU (default is unset)
+- `ZEEK_PIN_CPUS_OTHER_AUTO` - automatically pin CPUs for manager, loggers, and proxies if possible (default `false`)
+- `ZEEK_PIN_CPUS_MANAGER` - list of CPUs to pin for the [manager](https://docs.zeek.org/en/master/frameworks/cluster.html#manager) process (default is unset; only used if `ZEEK_PIN_CPUS_OTHER_AUTO` is `false`)
+- `ZEEK_PIN_CPUS_LOGGER` - list of CPUs to pin for the logger processes (default is unset; only used if `ZEEK_PIN_CPUS_OTHER_AUTO` is `false`)
+- `ZEEK_PIN_CPUS_PROXY` - list of CPUs to pin for the proxy processes (default is unset; only used if `ZEEK_PIN_CPUS_OTHER_AUTO` is `false`)
+
+These variables will aslo be honored, but it is not recommended to set them to any value greater than `1` as the Malcolm processes that monitor Zeek logs do not yet handle logs generated by multiple loggers to disparate locations:
+
+- `ZEEK_LB_PROCS_LOGGER` - Defines the number of processors to be assigned to the [loggers](https://docs.zeek.org/en/master/frameworks/cluster.html#logger) (default `1`)
+- `ZEEK_LB_PROCS_PROXY` - Defines the number of processors to be assigned to the [proxies](https://docs.zeek.org/en/master/frameworks/cluster.html#proxy) (default `1`)
 
 ### <a name="LiveAnalysisTuningArkime"></a>Arkime
 

hedgehog-iso/interface/sensor_ctl/control_vars.conf

@@ -70,27 +70,38 @@ export ZEEK_PRUNE_CHECK_SECONDS=90
 # Zeek performance tuning
 #   See idaholab/Malcolm#475 and idaholab/Malcolm#36 for details)
 
-# AF_Packet ring buffer size
+# AF_Packet ring buffer size in bytes
 export ZEEK_AF_PACKET_BUFFER_SIZE=67108864
-# default for ZEEK_LB_PROCS_WORKER_n if unspecified; defaults to '1'
+
+# Zeek Workers
+# default number of processes for each worker (if ZEEK_LB_PROCS_WORKER_n is unspecified)
 #   a value of '0' means "autocalculate based on the number of CPUs the system has"
 export ZEEK_LB_PROCS_WORKER_DEFAULT=1
-# if ZEEK_LB_PROCS_WORKER_DEFAULT is '0' (autocalculate), exclude this
-#   number of CPUs from the autocalculation (defaults to '4' for
-#   kernel, manager, proxy, and logger; may wish to increase as there
-#   are other processes using CPU as well)
-export ZEEK_LB_PROCS_CPUS_RESERVED=4
 # automatically pin worker CPUs (default 'false')
-export ZEEK_PIN_CPUS_WORKER_AUTO=
-# zeekdeploy.sh will also use (if present, where n is the number of capture interfaces):
+export ZEEK_PIN_CPUS_WORKER_AUTO=false
+# zeekdeploy.sh will also use these for workers (if present, where n is the number of capture interfaces):
 #   ZEEK_PIN_CPUS_WORKER_1 .. ZEEK_PIN_CPUS_WORKER_n
 #   ZEEK_LB_PROCS_WORKER_1 .. ZEEK_LB_PROCS_WORKER_n
-# ZEEK_PIN_CPUS_(LOGGER|MANAGER|PROXY) specify either a list of CPUs to pin or "true"
-#   to auto-pin one CPU each, respectively, if there are enough CPUs to do so
+
+# Zeek Loggers, Proxies, and Manager
+# the number of processors for loggers
+export ZEEK_LB_PROCS_LOGGER=1
+# the number of processors for proxies
+export ZEEK_LB_PROCS_PROXY=1
+# automatically pin CPUs for manager, loggers, and proxies if possible (default false)
+export ZEEK_PIN_CPUS_OTHER_AUTO=false
+# ZEEK_PIN_CPUS_(LOGGER|MANAGER|PROXY) specify either a list of CPUs to pin for those
+#   respective processes (only used if ZEEK_PIN_CPUS_OTHER_AUTO is false)
 export ZEEK_PIN_CPUS_LOGGER=
 export ZEEK_PIN_CPUS_MANAGER=
 export ZEEK_PIN_CPUS_PROXY=
 
+# if ZEEK_LB_PROCS_WORKER_DEFAULT is '0' (autocalculate), exclude this
+#   number of CPUs from the autocalculation (defaults to
+#   1 (kernel) + 1 (manager) + ZEEK_LB_PROCS_LOGGER + ZEEK_LB_PROCS_PROXY;
+#   may wish to increase as there are non-Zeek processes using CPU as well)
+export ZEEK_LB_PROCS_CPUS_RESERVED=
+
 export ZEEK_LOCAL_NETS=
 export ZEEK_JSON=
 export ZEEK_RULESET=local

shared/bin/zeek-deb-download.sh

@@ -23,20 +23,24 @@ if [[ -n $VERBOSE ]]; then
   set -x
 fi
 
-DEB_URL="https://downloadcontentcdn.opensuse.org/repositories/security:/zeek/${DISTRO}"
+URL_PREFIX="https://downloadcontentcdn.opensuse.org/repositories/security:/zeek/${DISTRO}"
+URLS=(
+  "${URL_PREFIX}/${ARCH}/libbroker-dev_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/${ARCH}/zeek-core-dev_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/${ARCH}/zeek-core_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/${ARCH}/zeek-spicy-dev_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/${ARCH}/zeek_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/${ARCH}/zeekctl_${ZEEK_VERSION}_${ARCH}.deb"
+  "${URL_PREFIX}/all/zeek-client_${ZEEK_VERSION}_all.deb"
+  "${URL_PREFIX}/all/zeek-zkg_${ZEEK_VERSION}_all.deb"
+  "${URL_PREFIX}/all/zeek-btest_${ZEEK_VERSION}_all.deb"
+  "${URL_PREFIX}/all/zeek-btest-data_${ZEEK_VERSION}_all.deb"
+)
 
 pushd "$OUTPUT_DIR" >/dev/null 2>&1
-curl --fail-early -fsSL --remote-name-all \
-  "${DEB_URL}/${ARCH}/libbroker-dev_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/${ARCH}/zeek-core-dev_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/${ARCH}/zeek-core_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/${ARCH}/zeek-spicy-dev_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/${ARCH}/zeek_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/${ARCH}/zeekctl_${ZEEK_VERSION}_${ARCH}.deb" \
-  "${DEB_URL}/all/zeek-client_${ZEEK_VERSION}_all.deb" \
-  "${DEB_URL}/all/zeek-zkg_${ZEEK_VERSION}_all.deb" \
-  "${DEB_URL}/all/zeek-btest_${ZEEK_VERSION}_all.deb" \
-  "${DEB_URL}/all/zeek-btest-data_${ZEEK_VERSION}_all.deb"
+for URL in ${URLS[@]}; do
+  curl -fsSL -O -J "${URL}"
+done
 popd >/dev/null 2>&1
 
 if [[ -n $VERBOSE ]]; then