diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md index f9bcd43618..7cddca0637 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/_index.md @@ -1,14 +1,18 @@ --- -title: Scaling Snort3 - How To Leverage Multithreading For Better Performance +title: Scaling Snort3 - use multithreading for improved performance + +draft: true +cascade: + draft: true minutes_to_complete: 45 who_is_this_for: This blog is for engineers familiar with Snort who want to enhance its performance by leveraging the benefits of multithreading. learning_objectives: - - Install Snort with all of its dependencies - - Configure Snort Lua files to enable multithreading - - Use multithreading to process capture files and measure performance + - Install Snort with all of its dependencies. + - Configure Snort Lua files to enable multithreading. + - Use multithreading to process capture files and measure performance. prerequisites: - An Arm-based instance from a cloud provider or an Arm server running Ubuntu 20.04 or 22.04. diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md index aed4e45bc3..e395286e6e 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/build-and-install.md @@ -1,212 +1,233 @@ ---- -title: Install Snort3 along with all its required dependencies. -weight: 2 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- - -Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. - -In order to enable multithreading in Snort3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. - -{{%notice Note%}} - The instruction provided have been tested on AWS EC2 Graviton4 metal instance (Neoverse V2) -{{%/notice%}} - -## Compile and build Snort3 -Run the script to download and install Snort3 and its dependent libraries. -Skip this step if Snort3 is already installed. - - -``` bash -#!/usr/bin/env bash - -# Copyright (c) 2022-2024, Arm Limited. -# -# SPDX-License-Identifier: Apache-2.0 -# author : PreemaMerlin.Dsouza@arm.com - -# Define a list of dependency package URLs -declare -a PACKAGE_URLS=( -"https://github.com/snort3/snort3/archive/refs/tags/3.3.5.0.tar.gz" -"https://sourceforge.net/projects/pcre/files/pcre/8.45/pcre-8.45.tar.gz" -"https://github.com/VectorCamp/vectorscan/archive/refs/tags/vectorscan/5.4.11.tar.gz" -"https://github.com/snort3/libdaq/archive/refs/tags/v3.0.16.tar.gz" -"https://boostorg.jfrog.io/artifactory/main/release/1.86.0/source/boost_1_86_0.tar.gz" -"https://github.com/rurban/safeclib/releases/download/v3.8.1/safeclib-3.8.1.tar.gz" -"https://github.com/gperftools/gperftools/releases/download/gperftools-2.13/gperftools-2.13.tar.gz" -) - -downlaodPackages() -{ - for url in "${PACKAGE_URLS[@]}"; do - # Extract the file name from the URL - fname=$(basename "$url") - fpath="${ROOT_DIR}/${fname}" - # Check if the file already exists - if [[ -f "$fpath" ]]; then - echo "File $fname already exists. Skipping download." - else - # Download the file using wget - - echo "File $fname not found. Downloading..." - - wget -O "$fpath" "$url" - if [[ $? -eq 0 ]]; then - echo "$fname download complete" - else - echo "ERROR:$fname download Fail." - fi - fi - done -} - -installPackages() -{ - echo "@@@@@@@@@@@@@@@@@@ Installing packages ... @@@@@@@@@@@@@@@@@@@@" - if [[ -r /etc/os-release ]]; then - OS_NAME=$(grep -w "NAME" /etc/os-release | cut -d= -f2 | tr -d '"') - OS_VERSION_ID=$(grep -w "VERSION_ID" /etc/os-release | cut -d= -f2 | tr -d '"') - if [[ "${OS_NAME}" == "Ubuntu" ]]; then - echo "OS: ${OS_NAME} ${OS_VERSION_ID}" - else - echo "Error: This script is only for ubuntu" - exit 1 - fi - if [[ "${OS_VERSION_ID}" != "22.04" ] || [ "${OS_VERSION_ID}" != "20.04" ]];then - echo "Warning: OS: ${OS_NAME} ${OS_VERSION_ID}" - echo "Warning: Ubuntu 20.04 or 22.04 is recommended" - fi - else - echo "Error: OS information detection failed" - exit 1 - fi - - apt-get update - apt-get install -y $LIST_OF_APPS - - # required to get optimised result from Snort3 - downlaodPackages - mkdir -p ${ROOT_DIR}/snort3 - tar -xzf 3.3.5.0.tar.gz --directory ${ROOT_DIR}/snort3 --strip-components=1 - echo "@@@@@@@@@@@@@@@@@@ Installing Snort3 Dependencies ... @@@@@@@@@@@@@@@@@@@@" - mkdir -p ${SNORT_DIR} - mkdir -p $SNORT_DIR/pcre - tar -xvf pcre-8.45.tar.gz --directory $SNORT_DIR/pcre --strip-components=1 - #vector scan - mkdir -p $SNORT_DIR/vectorscan - tar -xzvf 5.4.11.tar.gz --directory $SNORT_DIR/vectorscan --strip-components=1 - - #libdaq - mkdir -p $SNORT_DIR/libdaq - tar -xvzf v3.0.16.tar.gz --directory $SNORT_DIR/libdaq --strip-components=1 - - #required to get optimized result from vectorscan - mkdir -p $SNORT_DIR/boost - tar -xvf boost_1_86_0.tar.gz -C $SNORT_DIR/boost --strip-components=1 - - #safeclib - mkdir -p $SNORT_DIR/safeclib - tar -xzvf safeclib-3.8.1.tar.gz --directory $SNORT_DIR/safeclib --strip-components=1 - - #gperftools - mkdir -p $SNORT_DIR/gperftools - tar -xzvf gperftools-2.13.tar.gz --directory $SNORT_DIR/gperftools --strip-components=1 - - echo "@@@@@@@@@@@@@@@@@@ Packages installed @@@@@@@@@@@@@@@@@@@@" -} - -buildInstall() -{ - echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Start @@@@@@@@@@@@@@@@@@@@" - cd $SNORT_DIR/libdaq - mkdir -p ${SNORT_DIR}/libdaq/install - ./bootstrap - ./configure --prefix=${SNORT_DIR}/libdaq/install - make -j${NUM_JOBS} - make install - - cd ${SNORT_DIR}/safeclib - ./configure - make -j${NUM_JOBS} - make -j${NUM_JOBS} install - - cd $SNORT_DIR/gperftools - ./configure --with-tcmalloc-pagesize=64 - make -j${NUM_JOBS} - make -j${NUM_JOBS} - - cd $SNORT_DIR/pcre - ./configure - make -j${NUM_JOBS} - make -j${NUM_JOBS} - - cd ${SNORT_DIR}/vectorscan - cmake -DBOOST_ROOT=$(SNORT_DIR)/boost -DCMAKE_BUILD_TYPE=Release . - make -j${NUM_JOBS} - make -j${NUM_JOBS} - - cd ${ROOT_DIR}/snort3 - ./configure_cmake.sh --build-type=Release --with-daq-includes=${SNORT_DIR}/libdaq/install/include/ --with-daq-libraries=${SNORT_DIR}/libdaq/install/lib/ --enable-unit-tests --enable-tcmalloc - cd ${ROOT_DIR}/snort3/build - make -j$NUM_JOBS - make -j$NUM_JOBS install - echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Done @@@@@@@@@@@@@@@@@@@@" -} - -#------ Execution Start ----------# -# provide nproc count to the scripts , it will be used as -j for make -if [[ $# -ne 2 ]]; then - echo "Usage: $0 " - exit 1 -fi - -ROOT_DIR=$(pwd)/"$1" -NUM_JOBS="$2" -SNORT_DIR=${ROOT_DIR}/snort3/dependencies -set -e - -LIST_OF_APPS="sudo net-tools build-essential manpages-dev libnuma-dev python3 - python3-venv cmake meson pkg-config python3-pyelftools lshw - util-linux iperf3 nginx libboost-all-dev ragel libsqlite3-dev - libpcap-dev libdumbnet-dev libluajit-5.1-dev zlib1g-dev - libhwloc-dev liblzma-dev libssl-dev libgoogle-perftools-dev - libpcre++-dev flex openssl libunwind-dev autotools-dev - libhugetlbfs-bin autoconf libmnl-dev bats wget unzip iproute2 - git pkg-config cpputest libtool bison libcmocka-dev - libnetfilter-queue-dev ethtool" - -# nprc should be a positive integer) -if ! [[ "$NUM_JOBS" =~ ^[0-9]+$ ]] || [[ "$NUM_JOBS" -le 0 ]]; then - echo "Error: nprc should be a positive integer." - exit 1 -fi - -mkdir -p ${ROOT_DIR} -cd ${ROOT_DIR} -installPackages -buildInstall -``` - -To check if the installation is complete, run the command below. -```bash{ output_lines = "2-20" } - snort -V -,,_ -*> Snort++ <*- - o" )~ Version 3.3.5.0 - '''' By Martin Roesch & The Snort Team - http://snort.org/contact#team - Copyright (C) 2014-2024 Cisco and/or its affiliates. All rights reserved. - Copyright (C) 1998-2013 Sourcefire, Inc., et al. - Using DAQ version 3.0.16 - Using Hyperscan version 5.4.11 2024-09-12 - Using libpcap version 1.10.1 (with TPACKET_V3) - Using LuaJIT version 2.1.0-beta3 - Using LZMA version 5.2.5 - Using OpenSSL 3.0.2 15 Mar 2022 - Using PCRE version 8.45 2021-06-15 - Using ZLIB version 1.2.11 - -``` - - +--- +title: Install Snort3 and the required dependencies +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +Snort is an Open Source Intrusion Prevention System (IPS). Snort uses a series of rules to define malicious network activity. If malicious activity is found, Snort generates alerts. + +Multithreading in Snort 3 refers to the ability to associate multiple threads with a single Snort instance enabling the concurrent processing of multiple packet files. This optimization frees up additional memory for further packet processing. + +In order to enable multithreading in Snort3, specify the quantity of threads designated for processing network traffic using either the '--max-packet-threads' or '-z' option. + +{{%notice Note%}} + The instructions provided have been tested on AWS EC2 Graviton4 instance, based on Neoverse V2. The examples are easiest to use if you have at least 16 cores in the system. +{{%/notice%}} + +## Compile and build Snort3 + +To install Snort3, use a text editor to save the script below on your Arm server in a file named `install-snort.sh`. + + +``` bash +#!/usr/bin/env bash + +# Copyright (c) 2022-2024, Arm Limited. +# +# SPDX-License-Identifier: Apache-2.0 +# author : PreemaMerlin.Dsouza@arm.com + +# Define a list of dependency package URLs +declare -a PACKAGE_URLS=( +"https://github.com/snort3/snort3/archive/refs/tags/3.3.5.0.tar.gz" +"https://sourceforge.net/projects/pcre/files/pcre/8.45/pcre-8.45.tar.gz" +"https://github.com/VectorCamp/vectorscan/archive/refs/tags/vectorscan/5.4.11.tar.gz" +"https://github.com/snort3/libdaq/archive/refs/tags/v3.0.16.tar.gz" +"https://boostorg.jfrog.io/artifactory/main/release/1.86.0/source/boost_1_86_0.tar.gz" +"https://github.com/rurban/safeclib/releases/download/v3.8.1/safeclib-3.8.1.tar.gz" +"https://github.com/gperftools/gperftools/releases/download/gperftools-2.13/gperftools-2.13.tar.gz" +) + +downlaodPackages() +{ + for url in "${PACKAGE_URLS[@]}"; do + # Extract the file name from the URL + fname=$(basename "$url") + fpath="${ROOT_DIR}/${fname}" + # Check if the file already exists + if [[ -f "$fpath" ]]; then + echo "File $fname already exists. Skipping download." + else + # Download the file using wget + + echo "File $fname not found. Downloading..." + + wget -O "$fpath" "$url" + if [[ $? -eq 0 ]]; then + echo "$fname download complete" + else + echo "ERROR:$fname download Fail." + fi + fi + done +} + +installPackages() +{ + echo "@@@@@@@@@@@@@@@@@@ Installing packages ... @@@@@@@@@@@@@@@@@@@@" + if [[ -r /etc/os-release ]]; then + OS_NAME=$(grep -w "NAME" /etc/os-release | cut -d= -f2 | tr -d '"') + OS_VERSION_ID=$(grep -w "VERSION_ID" /etc/os-release | cut -d= -f2 | tr -d '"') + if [[ "${OS_NAME}" == "Ubuntu" ]]; then + echo "OS: ${OS_NAME} ${OS_VERSION_ID}" + else + echo "Error: This script is only for ubuntu" + exit 1 + fi + if [[ "${OS_VERSION_ID}" != "22.04" && "${OS_VERSION_ID}" != "20.04" ]];then + echo "Warning: OS: ${OS_NAME} ${OS_VERSION_ID}" + echo "Warning: Ubuntu 20.04 or 22.04 is recommended" + fi + else + echo "Error: OS information detection failed" + exit 1 + fi + + sudo apt-get update + sudo apt-get install -y $LIST_OF_APPS + + # required to get optimised result from Snort3 + downlaodPackages + mkdir -p ${ROOT_DIR}/snort3 + tar -xzf 3.3.5.0.tar.gz --directory ${ROOT_DIR}/snort3 --strip-components=1 + echo "@@@@@@@@@@@@@@@@@@ Installing Snort3 Dependencies ... @@@@@@@@@@@@@@@@@@@@" + mkdir -p ${SNORT_DIR} + mkdir -p $SNORT_DIR/pcre + tar -xvf pcre-8.45.tar.gz --directory $SNORT_DIR/pcre --strip-components=1 + #vector scan + mkdir -p $SNORT_DIR/vectorscan + tar -xzvf 5.4.11.tar.gz --directory $SNORT_DIR/vectorscan --strip-components=1 + + #libdaq + mkdir -p $SNORT_DIR/libdaq + tar -xvzf v3.0.16.tar.gz --directory $SNORT_DIR/libdaq --strip-components=1 + + #required to get optimized result from vectorscan + mkdir -p $SNORT_DIR/boost + tar -xvf boost_1_86_0.tar.gz -C $SNORT_DIR/boost --strip-components=1 + + #safeclib + mkdir -p $SNORT_DIR/safeclib + tar -xzvf safeclib-3.8.1.tar.gz --directory $SNORT_DIR/safeclib --strip-components=1 + + #gperftools + mkdir -p $SNORT_DIR/gperftools + tar -xzvf gperftools-2.13.tar.gz --directory $SNORT_DIR/gperftools --strip-components=1 + + echo "@@@@@@@@@@@@@@@@@@ Packages installed @@@@@@@@@@@@@@@@@@@@" +} + +buildInstall() +{ + echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Start @@@@@@@@@@@@@@@@@@@@" + cd $SNORT_DIR/libdaq + mkdir -p ${SNORT_DIR}/libdaq/install + ./bootstrap + ./configure + make -j${NUM_JOBS} + sudo make install + + cd ${SNORT_DIR}/safeclib + ./configure + make -j${NUM_JOBS} + sudo make -j${NUM_JOBS} install + + cd $SNORT_DIR/gperftools + ./configure --with-tcmalloc-pagesize=64 + make -j${NUM_JOBS} + + cd $SNORT_DIR/pcre + ./configure + make -j${NUM_JOBS} + + cd ${SNORT_DIR}/vectorscan + cmake -DBOOST_ROOT=$(SNORT_DIR)/boost -DCMAKE_BUILD_TYPE=Release . + make -j${NUM_JOBS} + + cd ${ROOT_DIR}/snort3 + ./configure_cmake.sh --prefix=/usr/local --build-type=Release --with-daq-includes=/usr/local/include/ --with-daq-libraries=/usr/local/lib/ --enable-unit-tests --enable-tcmalloc + cd ${ROOT_DIR}/snort3/build + make -j$NUM_JOBS + sudo make -j$NUM_JOBS install + echo "@@@@@@@@@@@@@@@@@@ Build & Installation ... Done @@@@@@@@@@@@@@@@@@@@" + +} + +#------ Execution Start ----------# +# provide nproc count to the scripts , it will be used as -j for make +if [[ $# -ne 2 ]]; then + echo "Usage: $0 " + exit 1 +fi + +ROOT_DIR=$(pwd)/"$1" +NUM_JOBS="$2" +SNORT_DIR=${ROOT_DIR}/snort3/dependencies +set -e + +LIST_OF_APPS="sudo net-tools build-essential manpages-dev libnuma-dev python3 + python3-venv cmake meson pkg-config python3-pyelftools lshw + util-linux iperf3 nginx libboost-all-dev ragel libsqlite3-dev + libpcap-dev libdumbnet-dev libluajit-5.1-dev zlib1g-dev + libhwloc-dev liblzma-dev libssl-dev libgoogle-perftools-dev + libpcre++-dev flex openssl libunwind-dev autotools-dev + libhugetlbfs-bin autoconf libmnl-dev bats wget unzip iproute2 + git pkg-config cpputest libtool bison libcmocka-dev + libnetfilter-queue-dev ethtool" + +# nprc should be a positive integer) +if ! [[ "$NUM_JOBS" =~ ^[0-9]+$ ]] || [[ "$NUM_JOBS" -le 0 ]]; then + echo "Error: nprc should be a positive integer." + exit 1 +fi + +mkdir -p ${ROOT_DIR} +cd ${ROOT_DIR} +installPackages +buildInstall + +echo 'export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"' >> $HOME/.bashrc +echo 'make sure to source ~/.bashrc or set LD_LIBRARY_PATH using:"' +echo ' export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"' +``` + +The script takes 2 arguments: +- the directory used to build Snort3 and its dependencies +- the number of processors to use for the build + +To build in a new directory named `build` with the number of processors in your system, run the script: + +```bash +bash ./install-snort.sh build `nproc` +``` + +You don't need to run the script as `root` but it assumes you are on Ubuntu 20.04 or 22.04 and have sudo permission. + +When the build completes you have the snort3 directory with all compiled software, and the `snort` executable is located in `/usr/local/bin`. + +To verify the installation is complete, run the command below and see the version printed: + +```bash { output_lines = "2-20" } + snort -V +,,_ -*> Snort++ <*- + o" )~ Version 3.3.5.0 + '''' By Martin Roesch & The Snort Team + http://snort.org/contact#team + Copyright (C) 2014-2024 Cisco and/or its affiliates. All rights reserved. + Copyright (C) 1998-2013 Sourcefire, Inc., et al. + Using DAQ version 3.0.16 + Using Hyperscan version 5.4.11 2024-09-12 + Using libpcap version 1.10.1 (with TPACKET_V3) + Using LuaJIT version 2.1.0-beta3 + Using LZMA version 5.2.5 + Using OpenSSL 3.0.2 15 Mar 2022 + Using PCRE version 8.45 2021-06-15 + Using ZLIB version 1.2.11 + +``` + +Don't delete the `build` directory as it will be used in the next step. + +Proceed to learn how to test Snort3 multithreading. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md index 5af472e368..42d73c45de 100644 --- a/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md +++ b/content/learning-paths/servers-and-cloud-computing/snort3-multithreading/usecase.md @@ -1,200 +1,311 @@ ---- -title: Test Snort3 Multithreading -weight: 3 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- -Before we begin testing, follow the steps below to ensure Snort3 is properly configured - -1. Configure Grub settings -2. Set up the Snort3 Rule Set -3. Download the PCAP files -4. Adjust Lua configurations - -## 1. Configure Grub settings -To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the /etc/default/grub file: -```bash -CMDLINE="cma=128" -HUGEPAGES="default_hugepagesz=1G hugepagesz=1G hugepages=300" -MAXCPUS="" -ISOLCPUS="isolcpus=nohz,domain," -IRQAFFINITY="irqaffinity=" -NOHZ="nohz_full=" -RCU="rcu_nocbs=" -IOMMU="iommu.passthrough=1" -THP="transparent_hugepage=madvise" -GRUB_CMDLINE_LINUX="${CMDLINE} ${HUGEPAGES} ${ISOLCPUS} ${IRQAFFINITY} ${NOHZ} ${RCU} ${MAXCPUS} ${IOMMU} ${THP}" -``` -After making this change, execute update-grub to apply the configuration, and then reboot the system to activate the settings. - -## 2. Set up the Snort3 Rule Set -Download the rule set from https://www.snort.org/ and extract it into your working directory. It should be noted that access to the rule set requires a subscription. -For testing, I used the file https://www.snort.org/downloads/registered/snortrules-snapshot-3110.tar.gz. -### 2.1. Download and unzip the rule set -```bash -mkdir -p Test/snortrules-3110 -tar -xzvf snortrules-snapshot-3110.tar.gz -C Test/snortrules-3110 -``` -### 2.2. Copy the "lua" folder from "snort3" source directory into the rules directory -```bash - cp -r snort3/lua/ Test/snortrules-3110/ -``` -## 3. Download the PCAP files -Feel free to use any pcap files that are relevant to your test scenario. For reference, I’m using this one for my testing: -https://www.netresec.com/?page=MACCDC -```bash -gunzip maccdc2012_00000.pcap.gz -mv maccdc2012_00000.pcap Test/Pcap/ -``` - -## 4. Adjust Lua configurations -First, assign each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration. Next, modify the Lua configuration files to enable the desired ruleset and profiling settings. -### 4.1. Pin snort threads to unique cpu core -Create a file named 'common.lua' -```bash -------------------------------------------------------------------------------- ----- common: shared configuration included at the end of other configs -------------------------------------------------------------------------------- ----- change these mappings so that the first N tests use unique cores -threads = -{ - { thread = 0, cpuset = '2' }, - { thread = 1, cpuset = '3' }, - { thread = 2, cpuset = '4' }, - { thread = 3, cpuset = '5' }, - { thread = 4, cpuset = '6' }, - { thread = 5, cpuset = '7' }, - { thread = 6, cpuset = '8' }, - { thread = 7, cpuset = '9' }, - { thread = 8, cpuset = '10' }, - { thread = 9, cpuset = '11' }, - { thread = 10, cpuset = '12' } -} -process = { threads = threads } -search_engine = { } -snort_whitelist_append("threads") - -``` - -Include the above file in snort.lua, to do so edit the snort.lua file add below line at the end of the file - ``` bash - include('common.lua') - ``` -### 4.2. Tweak Snort.lua file to enable "rules" and profiling - Enable all the rules by adding below line in "ips" block - ``` bash -enable_builtin_rules = true, -rules = [[ - include ../rules/includes.rules -]], -``` -Uncomment "profiler" and "latency" to enable profiling and packet statistics - - -## Snort Parameters -### 1. - -tweaks to select IPS policy - -Snort additionally allows you to fine-tune setups with the --tweaks parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. - -Snort 3 includes four preset policy files: max_detect, security, balanced, and connectivity. The max_detect policy favors maximum security, whereas the connectivity policy focuses on performance and uptime, which may come at the expense of security. - -### 2. - -daq to specify Data Acquisition Module -Snort supports DAQ modules which serves as an abstraction layer for interfacing with data source such as network interface. - -To see list of DAQ modules supported by snort use "--daq-list" command. - -``` bash -./snort3_src/snort3/build/src/snort --daq-dir ./snort3_src/snort3/dependencies/libdaq/install/lib/daq --daq-list -Available DAQ modules: -afpacket(v7): live inline multi unpriv - Variables: - buffer_size_mb - Packet buffer space to allocate in megabytes - debug - Enable debugging output to stdout - fanout_type - Fanout loadbalancing method - fanout_flag - Fanout loadbalancing option - use_tx_ring - Use memory-mapped TX ring - -bpf(v1): inline unpriv wrapper - -dump(v5): inline unpriv wrapper - Variables: - file - PCAP filename to output transmitted packets to (default: inline-out.pcap) - output - Set to none to prevent output from being written to file (deprecated) - dump-rx [arg] - Also dump received packets to their own PCAP file (default: inline-in.pcap) - -fst(v1): unpriv wrapper - Variables: - no_binding_verdicts - Disables enforcement of binding verdicts - enable_meta_ack - Enables support for filtering bare TCP acks - ignore_checksums - Ignore bad checksums while decoding - -gwlb(v1): inline unpriv wrapper - -nfq(v8): live inline multi - Variables: - debug - Enable debugging output to stdout - fail_open - Allow the kernel to bypass the netfilter queue when it is full - queue_maxlen - Maximum queue length (default: 1024) - -pcap(v4): readback live multi unpriv - Variables: - buffer_size - Packet buffer space to allocate in bytes - no_promiscuous - Disables opening the interface in promiscuous mode - no_immediate - Disables immediate mode for traffic capture (may cause unbounded blocking) - readback_timeout - Return timeout receive status in file readback mode - -savefile(v1): readback multi unpriv - -trace(v1): inline unpriv wrapper - Variables: - file - Filename to write text traces to (default: inline-out.txt) -``` - -For testing, we will use "dump" to analyse pcap files. - -## Spawn Snort3 process with multithreading -The following example shows how to use multiple Snort threads to analyze Pcap file. - -``` bash -MPSE=hyperscan POLICY=./snortrules-3110/lua/snort.lua TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/test ../snort3_src/snort3/build/src/snort -c ./snortrules-3110/lua/snort.lua --lua detection.allow_missing_so_rules=true --pcap-filter maccdc2012_00000.pcap --pcap-loop 10 --snaplen 0 --max-packet-threads 10 --daq dump --daq-dir ../snort3_src/snort3/dependencies/libdaq/install/lib/daq --daq-var output=none -H --pcap-dir /root/snort3_learning_path/Test/Pcap -Q --warn-conf-strict --tweaks security -``` ---pcap-loop: to loop pcap files for number specified ---max-packet-threads: to specify threads, which are 10 in this example. - -To confirm that the Snort process spans many threads, use the "mpstat" command to evaluate the CPU utilization. - -``` bash -mpstat -P 2-14 1 - -22:52:26 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle -22:52:28 2 98.50 0.00 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 3 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 4 98.50 0.00 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 5 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 6 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 7 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 8 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 9 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 10 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 11 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -22:52:28 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 -22:52:28 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 -22:52:28 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 -``` -## Usecase : Snort3 multi-threading to process single pcap file -This use case demonstrates how multithreading increases the number of packets processed per second. - -Pcap File Description - -| Name | Total Packets | -|------------------------|---------------| -| maccdc2012_0000.pcap | 86359430 | - -Result -| Threads | Packets Per Second | Runtime in Sec | -|---------|--------------------|----------------| -| 1 | 940960 | 91.777964 | -| 10 | 9406134 | 9.181182 | - -The results above illustrates that increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times. \ No newline at end of file +--- +title: Test Snort3 multithreading +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +Before testing multithreading performance, perform the following steps to configure your system: + +1. Configure Grub settings +2. Set up the Snort3 rule set +3. Download the PCAP files +4. Adjust Lua configurations + +## Configure Grub settings + +To enable Transparent HugePages (THP) and configure CPU isolation and affinity, append the following line to the /etc/default/grub file: + +```bash +CMDLINE="cma=128" +HUGEPAGES="default_hugepagesz=1G hugepagesz=1G hugepages=300" +MAXCPUS="" +ISOLCPUS="isolcpus=nohz,domain,2-12" +IRQAFFINITY="irqaffinity=2-12" +NOHZ="nohz_full=2-12" +RCU="rcu_nocbs=2-12" +IOMMU="iommu.passthrough=1" +THP="transparent_hugepage=madvise" +GRUB_CMDLINE_LINUX="${CMDLINE} ${HUGEPAGES} ${ISOLCPUS} ${IRQAFFINITY} ${NOHZ} ${RCU} ${MAXCPUS} ${IOMMU} ${THP}" +``` + +After making this change, execute update-grub to apply the configuration: + +```bash +sudo update-grub +``` + +Reboot the system to activate the settings. + +```bash +sudo reboot +``` + +Confirm the new command line was used for the last boot: + +```bash +cat /proc/cmdline +``` + +The output shows the additions to the kernel command line. + +It is similar to: + +```output +BOOT_IMAGE=/boot/vmlinuz-6.8.0-1019-aws root=PARTUUID=20d0887f-2302-4e77-9c05-b78f1f0ad30e ro default_hugepagesz=1G hugepagesz=1G hugepages=300 isolcpus=nohz,domain,2-12 irqaffinity=2-12 nohz_full=2-12 rcu_nocbs=2-12 iommu.passthrough=1 transparent_hugepage=madvise console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1 +``` + +You can also confirm the isolated processors: + +```bash +cat /sys/devices/system/cpu/isolated +``` + +The output shows the isolated processors: + +```output +2-12 +``` + +## Set up the Snort3 rule set + +Download the rule set from https://www.snort.org/ and extract it into your working directory. You should start in the `build` directory you used to build snort. + +```bash +cd $HOME/build +``` + +For testing, you can use the file https://www.snort.org/downloads/registered/snortrules-snapshot-3110.tar.gz. + +Download and unzip the rule set: + +```bash +wget https://www.snort.org/downloads/community/snort3-community-rules.tar.gz +mkdir -p Test/snortrules +tar -xzvf snort3-community-rules.tar.gz -C Test/snortrules +``` + +Copy the `lua` folder from the `snort3` source directory into the rules directory: + +```bash +cp -r snort3/lua/ Test/snortrules/ +``` + +## Download the packet capture (PCAP) files + +You can use any PCAP files that are relevant to your test scenario. + +One place to get PCAP files is: +https://www.netresec.com/?page=MACCDC + +Visit https://share.netresec.com/s/wC4mqF2HNso4Ten and download a PCAP file. + +Copy the file to your working directory and extract it, adjust the file name as needed if you downloaded a different PCAP file. + +```bash +gunzip maccdc2010_00000_20100310205651.pcap.gz +mkdir Test/Pcap +cp maccdc2010_00000_20100310205651.pcap Test/Pcap/ +``` + +## Adjust Lua configurations + +There are two modifications to the Lau configurations: +- Pin each Snort thread to a unique core, ensuring that the cores match those isolated in the GRUB configuration +- Enable the desired ruleset and enabling profiling + +### Pin snort threads to unique cpu core + +Navigate to the `Test/snortrules/lua` directory. + +```bash +cd Test/snortrules/lua +```` + +Use an editor to create a file named `common.lua` with the contents below. + +```bash +------------------------------------------------------------------------------- +---- common: shared configuration included at the end of other configs +------------------------------------------------------------------------------- +---- change these mappings so that the first N tests use unique cores +threads = +{ + { thread = 0, cpuset = '2' }, + { thread = 1, cpuset = '3' }, + { thread = 2, cpuset = '4' }, + { thread = 3, cpuset = '5' }, + { thread = 4, cpuset = '6' }, + { thread = 5, cpuset = '7' }, + { thread = 6, cpuset = '8' }, + { thread = 7, cpuset = '9' }, + { thread = 8, cpuset = '10' }, + { thread = 9, cpuset = '11' }, + { thread = 10, cpuset = '12' } +} +process = { threads = threads } +search_engine = { } +snort_whitelist_append("threads") +``` + +Include the above file in `snort.lua` by editing the file and adding the line below to the end of the file. + + ``` bash + include('common.lua') + ``` + +### Modify the snort.lua file to enable rules and profiling + +Use an editor to modify the `snort.lua` file. + +Enable all the rules by uncommenting the `enable_builtin_rules` line and adding the rule search directory as shown below: + +```bash +enable_builtin_rules = true, +rules = [[ + include ../rules/includes.rules +]], +``` + +Continue to edit `snort.lua` and comment out the `profiler` and `latency` lines to enable profiling and packet statistics. + +## Review the Snort parameters + +### Modify the IPS policy + +Snort3 allows you to fine-tune setups with the `--tweaks` parameter. This feature allows you to use one of Snort's policy files to enhance the detection engine for improved performance or increased security. + +Snort3 includes four preset policy files: max_detect, security, balanced, and connectivity. + +The max_detect policy favors maximum security, whereas the connectivity policy focuses on performance and uptime, which may come at the expense of security. + +### Specify the data acquisition module + +Snort supports DAQ modules which serves as an abstraction layer for interfacing with data source such as network interface. + +To see list of DAQ modules supported by snort use `--daq-list` command. + +Return to the `build` directory: + +```bash +cd $HOME/build +``` + +Run using the command: + +``` bash +snort --daq-dir ./snort3/dependencies/libdaq/install/lib/daq --daq-list +``` + +The output is: + +```output +Available DAQ modules: +afpacket(v7): live inline multi unpriv + Variables: + buffer_size_mb - Packet buffer space to allocate in megabytes + debug - Enable debugging output to stdout + fanout_type - Fanout loadbalancing method + fanout_flag - Fanout loadbalancing option + use_tx_ring - Use memory-mapped TX ring + +bpf(v1): inline unpriv wrapper + +dump(v5): inline unpriv wrapper + Variables: + file - PCAP filename to output transmitted packets to (default: inline-out.pcap) + output - Set to none to prevent output from being written to file (deprecated) + dump-rx [arg] - Also dump received packets to their own PCAP file (default: inline-in.pcap) + +fst(v1): unpriv wrapper + Variables: + no_binding_verdicts - Disables enforcement of binding verdicts + enable_meta_ack - Enables support for filtering bare TCP acks + ignore_checksums - Ignore bad checksums while decoding + +gwlb(v1): inline unpriv wrapper + +nfq(v8): live inline multi + Variables: + debug - Enable debugging output to stdout + fail_open - Allow the kernel to bypass the netfilter queue when it is full + queue_maxlen - Maximum queue length (default: 1024) + +pcap(v4): readback live multi unpriv + Variables: + buffer_size - Packet buffer space to allocate in bytes + no_promiscuous - Disables opening the interface in promiscuous mode + no_immediate - Disables immediate mode for traffic capture (may cause unbounded blocking) + readback_timeout - Return timeout receive status in file readback mode + +savefile(v1): readback multi unpriv + +trace(v1): inline unpriv wrapper + Variables: + file - Filename to write text traces to (default: inline-out.txt) +``` + +For testing, you can use `--daq dump` to analyze PCAP files. + +## Spawn Snort3 process with multithreading + +To run Snort3 with multithread start from the `Test` directory. + +```bash +cd $HOME/build/Test +``` + +The following example shows how to use multiple Snort threads to analyze PCAP files. + +``` bash +MPSE=hyperscan POLICY=./snortrules/lua/snort.lua TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/test snort -c ./snortrules/lua/snort.lua --lua detection.allow_missing_so_rules=true --pcap-filter maccdc2010_00000_20100310205651.pcap --pcap-loop 10 --snaplen 0 --max-packet-threads 10 --daq dump --daq-dir /usr/local/lib/daq --daq-var output=none -H --pcap-dir Pcap -Q --warn-conf-strict --tweaks security +``` + +Use `--pcap-loop` to loop PCAP files a number of time, 10 in this example. + +Use `--max-packet-threads` to specify the number of threads, 10 in this example. + +To confirm that the Snort process spans many threads, use the `mpstat` command to evaluate the CPU utilization. + +```bash +mpstat -P 2-14 1 +``` + +The output is similar to: + +```output +22:52:26 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle +22:52:28 2 98.50 0.00 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 3 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 4 98.50 0.00 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 5 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 6 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 7 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 8 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 9 99.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 10 98.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 11 97.50 0.00 2.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 +22:52:28 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 +22:52:28 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 +22:52:28 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 +``` + +## Test Snort3 multi-threading to process single pcap file + +The example usage demonstrates how multithreading increases the number of packets processed per second. + +PCAP File Description + +| Name | Total Packets | +|------------------------|---------------| +| maccdc2012_0000.pcap | 86359430 | + +Performance results + +| Threads | Packets Per Second | Runtime in Sec | +|---------|--------------------|----------------| +| 1 | 940960 | 91.777964 | +| 10 | 9406134 | 9.181182 | + +The results demonstrate how increasing the thread count by ten times results in a ten times increase in packets processed per second, while reducing the execution time by ten times. \ No newline at end of file