banip: reduce fork overhead in monitor and feeds

[Coralesoft]

Under brute-force traffic, the banIP log monitor can fall behind the
log stream on a 4-core ARM router due to multiple fork/exec calls for
IP extraction, hit counting, logging, and nft set refresh.

This patch reduces that overhead by:

• replacing 24 printf | grep membership checks with shell case
  matching in f_conf, f_getdl, f_getdev, f_getup,
  f_nftinit, f_down, f_rmset, and banip-service.sh
• moving awk-based IP extraction in f_monitor into the pipeline as
  a persistent filter with fflush()
• replacing the per-hit logread | grep counter with shell variables,
  resetting after 10,000 unique IPs
• replacing the post-block nft list set re-dump with a string append
• consolidating two f_log calls into one
• parallelising country and ASN feed downloads in banip-service.sh
  using the existing ban_cores throttle

No new dependencies. All changes are POSIX case / busybox ash
compatible.

Semantic changes

Two monitor-path changes are not strictly equivalent to the previous
behaviour:

1. Hit counters are now session-scoped and reset on monitor restart.
2. The blocked-IP cache is monitor-local, so IPs added externally
   after startup are not reflected until the next restart.

These trade-offs remove repeated per-event process and set refresh
overhead. Full state is still rebuilt on restart.

Benchmarks (GL-MT6000, automated test suite)

Change	Before	After	Improvement
IP hit counter (5000 IPs)	10,700 ms	20 ms	535x
case vs printf|grep (500 iter)	840 ms	10 ms	84x
Persistent awk (1K log lines)	5,470 ms	80 ms	68x
Post-block set re-dump	5 ms/block	0 ms/block	eliminated
Logger forks per event	2	1	50%

---

📦 Package Details

Maintainer: @dibdot

Description:
Shell-level optimisations to banIP's log monitor and feed processing to reduce fork/exec overhead under sustained load.

---

🧪 Run Testing Details

• OpenWrt Version: 25.12.1
• OpenWrt Target/Subtarget: mediatek/filogic
• OpenWrt Device: GL-iNet GL-MT6000

---

✅ Formalities

• I have reviewed the CONTRIBUTING.md file for detailed contributing guidelines.

[Copilot]

Pull request overview

This PR optimizes banIP’s shell-based log monitor and feed download processing to reduce fork/exec overhead under sustained brute-force traffic, targeting better throughput on low-core OpenWrt routers.

Changes:

• Replace repeated printf | grep membership checks with shell case matching in multiple code paths.
• Rework the monitor pipeline to use a persistent awk filter for IP extraction and maintain in-process hit counters/caches.
• Parallelize country/ASN feed downloads in banip-service.sh using the existing ban_cores throttle approach.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
net/banip/files/banip-service.sh	Adds case-based allowlist-only skipping and backgrounds country/ASN downloads with a ban_cores throttle.
net/banip/files/banip-functions.sh	Replaces grep-based membership checks, refactors monitor IP extraction/counting, and adjusts feed/set handling logic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dibdot]

@Coralesoft on a first sight this PR looks heavily AI generated.
Please provide your benchmark to reproduce your results. Thanks.

[Coralesoft]

@dibdot, Here is my test script, I've combined then into a single script to make it easier to run on the router
This was run on a GL-MT6000 running OpenWrt 25.12.1 with banIP active and country split enabled.

#!/bin/ash
# banIP test helper
# run on the router

NFTCMD="$(command -v nft)"
GREPCMD="$(command -v grep)"
AWKCMD="$(command -v gawk || command -v awk)"
LOGCMD="$(command -v logger)"
JSONFILTERCMD="$(command -v jsonfilter)"
UCICMD="$(command -v uci)"
TMPDIR="$(mktemp -d)"
PASS=0
FAIL=0
ITERATIONS=500

cleanup() {
    rm -rf "${TMPDIR}"
}

now_ms() {
    awk '{print int($1 * 1000)}' /proc/uptime 2>/dev/null | head -1 || date +%s000
}

pass() { PASS=$((PASS + 1)); printf "  ok: %s\n" "$1"; }
fail() { FAIL=$((FAIL + 1)); printf "  fail: %s\n" "$1"; }

need_cmd() {
    name="$1"
    path="$2"
    if [ -z "${path}" ]; then
        fail "missing command: ${name}"
        cleanup
        exit 1
    fi
}

trap cleanup EXIT INT TERM

echo
echo "banIP test script"
echo "================="
printf "host: %s\n" "$(uname -a)"
echo

need_cmd nft "${NFTCMD}"
need_cmd grep "${GREPCMD}"
need_cmd awk "${AWKCMD}"
need_cmd logger "${LOGCMD}"
need_cmd jsonfilter "${JSONFILTERCMD}"
need_cmd uci "${UCICMD}"

if [ ! -x /etc/init.d/banip ]; then
    fail "/etc/init.d/banip not found"
    printf "run this on the OpenWrt router where banIP is installed\n"
    exit 1
fi

echo "[1] basic checks"
echo

printf "service status\n"
if /etc/init.d/banip running 2>/dev/null; then
    pass "banIP service is running"
else
    fail "banIP service is not running"
fi

printf "nft table\n"
if ${NFTCMD} list table inet banIP >/dev/null 2>&1; then
    sets=$(${NFTCMD} -j list table inet banIP 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[@.set.family="inet"].set.name' | wc -l)
    pass "banIP table loaded (${sets} sets)"
else
    fail "banIP table not found"
fi

printf "feed count\n"
elem=$(/etc/init.d/banip status 2>/dev/null | ${AWKCMD} '/element_count/{split($0,a,":"); gsub(/[^0-9]/,"",a[2]); print a[2]}')
if [ -n "${elem}" ] && [ "${elem}" != "0" ]; then
    pass "${elem} elements loaded"
else
    fail "no elements loaded"
fi

printf "monitor pid\n"
pidfile="/var/run/banip.pid"
if [ -s "${pidfile}" ]; then
    ppid=$(cat "${pidfile}")
    if kill -0 "${ppid}" 2>/dev/null; then
        pass "monitor running (PID ${ppid})"
    else
        fail "monitor PID ${ppid} not running"
    fi
else
    fail "monitor PID file empty or missing"
fi

printf "blocklist access\n"
if ${NFTCMD} list set inet banIP blocklist.v4 >/dev/null 2>&1; then
    bl_size=$(${NFTCMD} -j list set inet banIP blocklist.v4 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[*].set.elem[*]' 2>/dev/null | wc -l)
    pass "blocklist.v4 accessible (${bl_size} elements)"
else
    fail "blocklist.v4 set not available"
fi

printf "live monitor check\n"
test_ip="198.51.100.99"
${NFTCMD} delete element inet banIP blocklist.v4 "{ ${test_ip} }" 2>/dev/null
${LOGCMD} -p auth.info -t "sshd" "Exit before auth from ${test_ip} port 22"
${LOGCMD} -p auth.info -t "dropbear" "Exit before auth from ${test_ip}"
sleep 5

if logread -l 30 2>/dev/null | ${GREPCMD} -q "suspicious IP '${test_ip}'"; then
    pass "monitor detected injected IP ${test_ip}"
    ${NFTCMD} delete element inet banIP blocklist.v4 "{ ${test_ip} }" 2>/dev/null
elif ${NFTCMD} get element inet banIP blocklist.v4 "{ ${test_ip} }" >/dev/null 2>&1; then
    pass "monitor detected and blocked injected IP ${test_ip}"
    ${NFTCMD} delete element inet banIP blocklist.v4 "{ ${test_ip} }" 2>/dev/null
else
    fail "monitor did not detect injected IP"
fi

echo
echo "[2] IP counter test"
echo "shell vars vs grep over log"
echo

SYSLOG="${TMPDIR}/syslog"
: > "${SYSLOG}"
i=0
while [ "$i" -lt 4900 ]; do
    case $((i % 5)) in
        0) printf "kernel: [%d] banIP/pre-ct/drop: SRC=1.2.3.%d\n" "$i" "$((i%256))" ;;
        1) printf "dnsmasq: query A from 192.168.1.%d\n" "$((i%256))" ;;
        2) printf "odhcpd: sending RA on br-lan\n" ;;
        3) printf "kernel: [%d] nf_conntrack: table full\n" "$i" ;;
        4) printf "dropbear: password auth for root\n" ;;
    esac >> "${SYSLOG}"
    i=$((i + 1))
done

SEARCH_IP="10.50.0.0"

echo "Detected IPs | OLD | NEW | Speedup"

for DETECTIONS in 10 100 1000 5000; do
    LOG_FILE="${TMPDIR}/syslog.${DETECTIONS}"
    cp "${SYSLOG}" "${LOG_FILE}"

    d=0
    while [ "$d" -lt "${DETECTIONS}" ]; do
        printf "banIP: suspicious IP '10.%d.%d.%d'\n" "$((d%256))" "$((d/256%256))" "0" >> "${LOG_FILE}"
        d=$((d + 1))
    done
    printf "banIP: suspicious IP '%s'\n" "${SEARCH_IP}" >> "${LOG_FILE}"

    ip_var="${SEARCH_IP//./_}"
    eval "_cnt_${ip_var}=${DETECTIONS}"

    start=$(now_ms)
    i=0
    while [ "$i" -lt "${ITERATIONS}" ]; do
        cat "${LOG_FILE}" | ${GREPCMD} -Fc "suspicious IP '${SEARCH_IP}'" >/dev/null 2>&1
        i=$((i + 1))
    done
    old_ms=$(( $(now_ms) - start ))

    start=$(now_ms)
    i=0
    while [ "$i" -lt "${ITERATIONS}" ]; do
        eval "_cnt_${ip_var}=\$((\${_cnt_${ip_var}:-0} + 1))"
        eval "log_count=\${_cnt_${ip_var}}"
        i=$((i + 1))
    done
    new_ms=$(( $(now_ms) - start ))

    if [ "${new_ms}" -gt 0 ]; then
        speedup="$((old_ms / new_ms))x"
    elif [ "${old_ms}" -gt 0 ]; then
        speedup=">>${old_ms}x"
    else
        speedup="~"
    fi

    printf "%s -> old %s ms, new %s ms, %s\n" "${DETECTIONS}" "${old_ms}" "${new_ms}" "${speedup}"
done

echo
echo "[3] IP extraction test"
echo "persistent awk vs awk per line"
echo

TEST_LOG="${TMPDIR}/testlog"
: > "${TEST_LOG}"
i=0
while [ "$i" -lt 1000 ]; do
    printf "Mar 28 12:00:00 router sshd[%d]: Exit before auth from 10.%d.%d.%d port 22\n" \
        "$((i + 1000))" "$((i%256))" "$((i/256%256))" "0" >> "${TEST_LOG}"
    i=$((i + 1))
done

AWK_PROG='{
    gsub(/[<>[\]]/, "", $0)
    sub(/%.*/, "", $0)
    sub(/:[0-9]+([ >]|$)/, "\\1", $0)
    if (match($0, /([0-9]{1,3}\.){3}[0-9]{1,3}/, m4)) {
        if (m4[0] !~ /^127\./ && m4[0] !~ /^0\./) {
            print m4[0] " .v4"
            fflush()
            next
        }
    }
}'

echo "processing 1000 lines..."

start=$(now_ms)
while IFS= read -r line; do
    ip_proto=$(printf "%s" "${line}" | ${AWKCMD} "${AWK_PROG}")
done < "${TEST_LOG}"
old_ms=$(( $(now_ms) - start ))

start=$(now_ms)
cat "${TEST_LOG}" | ${AWKCMD} "${AWK_PROG}" | while read -r ip proto; do
    :
done
new_ms=$(( $(now_ms) - start ))

if [ "${new_ms}" -gt 0 ]; then
    speedup="$((old_ms / new_ms))x"
elif [ "${old_ms}" -gt 0 ]; then
    speedup=">>${old_ms}x"
else
    speedup="~"
fi

echo "per-line awk: ${old_ms} ms"
echo "persistent awk: ${new_ms} ms"
echo "speedup: ${speedup}"

echo
echo "[4] set dump cost"
echo

printf "measuring blocklist dump cost (20 iterations)...\n"

start=$(now_ms)
i=0
dump_iters=20
while [ "$i" -lt "${dump_iters}" ]; do
    cached_v4="$(${NFTCMD} list set inet banIP blocklist.v4 2>/dev/null)"
    i=$((i + 1))
done
dump_ms=$(( $(now_ms) - start ))
dump_per=$((dump_ms / dump_iters))
cached_size=$(printf "%s" "${cached_v4}" | wc -c)

local_blocked=""
start=$(now_ms)
i=0
while [ "$i" -lt "${ITERATIONS}" ]; do
    local_blocked="${local_blocked} 10.$((i%256)).$((i/256%256)).0 "
    i=$((i + 1))
done
append_ms=$(( $(now_ms) - start ))

printf "dump per call:  %s ms\n" "${dump_per}"
printf "dump size:      %s bytes\n" "${cached_size}"
printf "append loop:    %s ms (%s appends)\n" "${append_ms}" "${ITERATIONS}"
printf "10/min:         %s ms\n" "$((dump_per * 10))"
printf "100/min:        %s ms\n" "$((dump_per * 100))"

echo
echo "[5] case vs grep"
echo

test_list="cinsscore drop dshield firehol1 ipthreat country debl doh threat threatview tor feodo greensnow bruteforceblock "
search_val="ipthreat"

start=$(now_ms)
i=0
while [ "$i" -lt "${ITERATIONS}" ]; do
    printf "%s" "${test_list}" | ${GREPCMD} -q "${search_val}"
    i=$((i + 1))
done
old_ms=$(( $(now_ms) - start ))

start=$(now_ms)
i=0
while [ "$i" -lt "${ITERATIONS}" ]; do
    case " ${test_list}" in
        *" ${search_val} "*) ;;
    esac
    i=$((i + 1))
done
new_ms=$(( $(now_ms) - start ))

if [ "${new_ms}" -gt 0 ]; then
    speedup="$((old_ms / new_ms))x"
elif [ "${old_ms}" -gt 0 ]; then
    speedup=">>${old_ms}x"
else
    speedup="~"
fi

echo "printf|grep: ${old_ms} ms"
echo "case: ${new_ms} ms"
echo "speedup: ${speedup}"
echo "saved over ~60 checks: $(( (old_ms - new_ms) * 60 / ITERATIONS )) ms"

echo
echo "[6] reload stress test"
echo

RELOAD_ROUNDS=3

printf "capturing baseline...\n"
baseline_sets="${TMPDIR}/baseline_sets"
baseline_elems="${TMPDIR}/baseline_elems"

${NFTCMD} -j list table inet banIP 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[@.set.family="inet"].set.name' | sort > "${baseline_sets}" 2>/dev/null
baseline_set_count=$(wc -l < "${baseline_sets}")

while read -r setname; do
    [ -z "${setname}" ] && continue
    cnt=$(${NFTCMD} -j list set inet banIP "${setname}" 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[*].set.elem[*]' 2>/dev/null | wc -l)
    printf "%s:%s\n" "${setname}" "${cnt}"
done < "${baseline_sets}" > "${baseline_elems}"
baseline_total=$(${AWKCMD} -F: '{s+=$2}END{print s+0}' "${baseline_elems}")

if [ "${baseline_set_count}" -gt 0 ]; then
    pass "baseline captured: ${baseline_set_count} sets, ${baseline_total} total elements"
else
    fail "could not capture baseline"
fi

country_split="$(${UCICMD} -q get banip.global.ban_countrysplit)"
asn_split="$(${UCICMD} -q get banip.global.ban_asnsplit)"
country_list="$(${UCICMD} -q get banip.global.ban_country)"
asn_list="$(${UCICMD} -q get banip.global.ban_asn)"

if [ "${country_split}" = "1" ] && [ -n "${country_list}" ]; then
    country_count=0
    for c in ${country_list}; do
        country_count=$((country_count + 1))
    done
    printf "country split active: %d countries\n" "${country_count}"
elif [ "${asn_split}" = "1" ] && [ -n "${asn_list}" ]; then
    asn_count=0
    for a in ${asn_list}; do
        asn_count=$((asn_count + 1))
    done
    printf "ASN split active: %d ASNs\n" "${asn_count}"
else
    printf "note: country/ASN split not active, still running general reload test\n"
fi

BANIP_RT="/var/run/banip_runtime.json"
MAX_WAIT=300

wait_active() {
    waited=0
    while [ "${waited}" -lt "${MAX_WAIT}" ]; do
        if [ -f "${BANIP_RT}" ]; then
            st=$(${JSONFILTERCMD} -qi "${BANIP_RT}" -e '@.status' 2>/dev/null)
            [ "${st}" = "active" ] && return 0
        fi
        sleep 2
        waited=$((waited + 2))
    done
    return 1
}

printf "\nrunning %d reloads...\n" "${RELOAD_ROUNDS}"
reload_fail=0
r=1
while [ "${r}" -le "${RELOAD_ROUNDS}" ]; do
    if [ -d "/var/run/banip.lock" ]; then
        printf "waiting for prior reload..."
        if ! wait_active; then
            printf " timed out\n"
            fail "reload ${r}: prior reload did not finish"
            reload_fail=$((reload_fail + 1))
            r=$((r + 1))
            continue
        fi

        lock_wait=0
        while [ -d "/var/run/banip.lock" ] && [ "${lock_wait}" -lt 15 ]; do
            sleep 1
            lock_wait=$((lock_wait + 1))
        done
        printf " done\n"
    fi

    printf "\nreload %d/%d\n" "${r}" "${RELOAD_ROUNDS}"
    log_marker="banip_stress_test_round_${r}_$(date +%s)"
    ${LOGCMD} -t "banip-test" "${log_marker}"

    reload_start=$(now_ms)
    /etc/init.d/banip reload >/dev/null 2>&1
    rc=$?

    if [ "${rc}" -ne 0 ]; then
        fail "reload ${r}: init script exit code ${rc}"
        reload_fail=$((reload_fail + 1))
        r=$((r + 1))
        continue
    fi

    wait_start=0
    while [ "${wait_start}" -lt 30 ]; do
        sleep 1
        wait_start=$((wait_start + 1))
        if [ -f "${BANIP_RT}" ]; then
            st=$(${JSONFILTERCMD} -qi "${BANIP_RT}" -e '@.status' 2>/dev/null)
            [ "${st}" != "active" ] && break
        fi
    done

    if ! wait_active; then
        reload_ms=$(( $(now_ms) - reload_start ))
        fail "reload ${r}: did not return to active within ${MAX_WAIT}s"
        reload_fail=$((reload_fail + 1))
        r=$((r + 1))
        continue
    fi
    reload_ms=$(( $(now_ms) - reload_start ))
    echo "reload completed in ${reload_ms} ms ($((reload_ms / 1000)) s)"

    if ! ${NFTCMD} list table inet banIP >/dev/null 2>&1; then
        fail "reload ${r}: banIP nft table missing"
        reload_fail=$((reload_fail + 1))
        r=$((r + 1))
        continue
    fi

    post_sets="${TMPDIR}/post_sets_${r}"
    ${NFTCMD} -j list table inet banIP 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[@.set.family="inet"].set.name' | sort > "${post_sets}" 2>/dev/null
    post_set_count=$(wc -l < "${post_sets}")

    if [ "${post_set_count}" -eq "${baseline_set_count}" ]; then
        printf "set count matches baseline: %d\n" "${post_set_count}"
    else
        fail "reload ${r}: set count ${post_set_count} != baseline ${baseline_set_count}"
        diff_out=$(diff "${baseline_sets}" "${post_sets}" 2>/dev/null)
        [ -n "${diff_out}" ] && printf "%s\n" "${diff_out}"
        reload_fail=$((reload_fail + 1))
    fi

    post_elems="${TMPDIR}/post_elems_${r}"
    while read -r setname; do
        [ -z "${setname}" ] && continue
        cnt=$(${NFTCMD} -j list set inet banIP "${setname}" 2>/dev/null | ${JSONFILTERCMD} -qe '@.nftables[*].set.elem[*]' 2>/dev/null | wc -l)
        printf "%s:%s\n" "${setname}" "${cnt}"
    done < "${post_sets}" > "${post_elems}"
    post_total=$(${AWKCMD} -F: '{s+=$2}END{print s+0}' "${post_elems}")

    empty_after=0
    while IFS=: read -r bname bcnt; do
        [ -z "${bname}" ] && continue
        [ "${bcnt}" -eq 0 ] && continue
        pcnt=$(${GREPCMD} -F "${bname}:" "${post_elems}" | head -1 | cut -d: -f2)
        if [ "${pcnt:-0}" -eq 0 ]; then
            printf "warning: set '%s' had %s elements, now empty\n" "${bname}" "${bcnt}"
            empty_after=$((empty_after + 1))
        fi
    done < "${baseline_elems}"

    if [ "${empty_after}" -gt 0 ]; then
        fail "reload ${r}: ${empty_after} sets went empty"
        reload_fail=$((reload_fail + 1))
    fi

    if [ "${baseline_total}" -gt 0 ]; then
        drift=$(( (post_total - baseline_total) * 100 / baseline_total ))
        if [ "${drift}" -lt 0 ]; then
            drift=$(( -drift ))
        fi
        printf "elements: %d (baseline %d, drift %d%%)\n" "${post_total}" "${baseline_total}" "${drift}"
        if [ "${drift}" -gt 5 ]; then
            printf "warning: element count drifted >5%%\n"
        fi
    fi

    error_count=$(logread 2>/dev/null | ${AWKCMD} -v marker="${log_marker}" '
        found{if(/banIP.*error|banIP.*fail/){c++}}
        $0~marker{found=1}
        END{print c+0}')

    if [ "${error_count}" -gt 0 ]; then
        printf "warning: %d error/fail messages after reload\n" "${error_count}"
        logread 2>/dev/null | ${AWKCMD} -v marker="${log_marker}" '
            found{if(/banIP.*error|banIP.*fail/){print "  " $0}}
            $0~marker{found=1}' | head -5
    else
        printf "no banIP errors in log\n"
    fi

    r=$((r + 1))
done

echo
echo "summary"
if [ "${reload_fail}" -eq 0 ]; then
    pass "all ${RELOAD_ROUNDS} reloads completed cleanly"
else
    fail "${reload_fail} issue(s) across ${RELOAD_ROUNDS} reloads"
fi

printf "\n=================\n"
printf "results: %d passed, %d failed\n" "${PASS}" "${FAIL}"
printf "=================\n"

the results

=================
host: Linux xxxxxx 6.12.74 #0 SMP Wed Mar 25 20:09:53 2026 aarch64 GNU/Linux

[1] basic checks

service status
ok: banIP service is running
nft table
ok: banIP table loaded (62 sets)
feed count
ok: 88256 elements loaded
monitor pid
ok: monitor running (PID 8643)
blocklist access
ok: blocklist.v4 accessible (0 elements)
live monitor check
ok: monitor detected and blocked injected IP 198.51.100.99

[2] IP counter test
shell vars vs grep over log

Detected IPs | OLD | NEW | Speedup
10 -> old 5210 ms, new 20 ms, 260x
100 -> old 5220 ms, new 20 ms, 261x
1000 -> old 6080 ms, new 20 ms, 304x
5000 -> old 10610 ms, new 20 ms, 530x

[3] IP extraction test
persistent awk vs awk per line

processing 1000 lines...
per-line awk: 6070 ms
persistent awk: 80 ms
speedup: 75x

[4] set dump cost

measuring blocklist dump cost (20 iterations)...
dump per call: 5 ms
dump size: 114 bytes
append loop: 60 ms (500 appends)
10/min: 50 ms
100/min: 500 ms

[5] case vs grep

printf|grep: 1050 ms
case: 10 ms
speedup: 105x
saved over ~60 checks: 124 ms

[6] reload stress test

capturing baseline...
ok: baseline captured: 62 sets, 88256 total elements
country split active: 17 countries

running 3 reloads...

reload 1/3
reload completed in 105600 ms (105 s)
set count matches baseline: 62
elements: 88293 (baseline 88256, drift 0%)
no banIP errors in log

reload 2/3
reload completed in 119570 ms (119 s)
set count matches baseline: 62
elements: 88278 (baseline 88256, drift 0%)
no banIP errors in log

reload 3/3
reload completed in 103540 ms (103 s)
set count matches baseline: 62
elements: 88293 (baseline 88256, drift 0%)
no banIP errors in log

summary
ok: all 3 reloads completed cleanly

results: 8 passed, 0 failed
root@xxxxxx:~#

[Coralesoft]

@dibdot Here is a breakdown for why it works:

For the printf | grep, the old code checks if a value is in a list by piping it through two external programs, every time it does this the shell has to start 2 new processes and hand data between them and then wait for them to finish. A case statement does the same check inside entirely inside the shell, no new processes needed.
This happens ~60 time per reload cycle, it adds up .

The in memory IP hit counter
the old monitor counted how many times it had seen an IP by re-reading the system log and counting matches with grep, this means every time a new log line arrives it forks an external processes to scan through potentially 1000's of log lines. The new code just keeps a simple counter in a shell variable. This is like keeping a tally on paper instead of re-reading the entire book each time to see how many times a word appears.

For the persistent awk in the monitor pipeline, the old monitor extracted the IP address from each log line by forking a new awk process for every single line. The new code runs one awk process that stays alive and filters all lines as they stream through. The old code was like restarting a car engine for every metre travelled versus just keeping it running for the whole trip.

Removing the post-block set re-dump, After blocking an IP the old code re-reads the entire nftables blocklist set to update its local cache, the new code just appends the IP to a local string instead, it allready knows what its just added so there is no need to ask the nftables to repeat it back

Parallel country/ASN feed downloads, the main gain is that those feeds no longer have to be fetched one after another when multiple split feeds are enabled.

[dibdot]

@Coralesoft Thanks for your post. It’ll be a while before I can give you my feedback, as I’m on holiday at the moment.

[Coralesoft]

@dibdot, all good mate, I'm about to do the same :)

[dibdot]

@Coralesoft
I have incorporated some of your suggested changes regarding forking and parallelisation into banIP 1.8.5 and added you as a co-author (see 9c3470a).

I have completely overhauled the monitor, and the new version should work significantly better than the version you proposed. (and my original version).

Thank you for your contribution.