Skip to content

v1.1 — payload dedup, memory caps, diagnostics

Choose a tag to compare

View all tags
@hett-patell hett-patell released this 25 May 06:24
· 36 commits to main since this release
v1.1
74e3b51

Maintenance release. Drop-in upgrade from v1 — no config changes, no schema breaks (migration v3 runs automatically on first start).

Payload library: deduplicated by SHA-256

The payload library previously rendered one row per artifact INSERT, so a single binary delivered via N rotated URLs from M source IPs appeared as N visually identical rows. In a 7-day window the table was 200 mostly-duplicate rows for ~80 unique payloads (the top SHA, an Outlaw SSH key, alone accounted for 462 of 893 captures).

  • store: new ListArtifactsAggregatedSince returns one row per unique sha with occurrences, urlCount, ipCount, actorCount, sessionCount, plus the most-recent capture metadata as Last{URL,SrcIP,Actor,Session,TS}. Portable SQL: CTE + correlated subqueries, no window functions.
  • /api/intel/payloads: response keeps url/actor/session/srcIp/ts keys (now last-seen) for backward compatibility and adds firstTs, occurrences, urlCount, ipCount, actorCount, sessionCount.
  • /api/intel/payloads?limit=>1000 now clamps to 1000 instead of silently falling back to 200.
  • /api/intel/sessions accepts ?limit= (default 200, ceiling 2000) instead of being hard-coded at 200 — the session panel was losing ~367 of 567 command-bearing sessions in a 7-day window.
  • Dashboard: payload table gains a delivery column showing Nx · Mip; the meta line reads N unique · M captures · Wh window so the dedup story is visible at a glance.

Memory: bound the live collector + geo cache

Across ~2.5 days of uptime, RSS peaked at 1.6 GB. The cause was two unbounded in-memory structures: the live SSH journal collector accumulated per-IP user maps across thousands of attacker IPs × hundreds of distinct usernames each, and the web geo-resolver cache never evicted.

  • actor.liveJournalCollector replaces the unbounded collector:
    • LRU cap of 4096 distinct source IPs.
    • Per-IP user-map cap of 256 keys; overflow rolls into an _overflow_ synthetic bucket so aggregate counts stay accurate.
    • Idle-TTL sweep at 12h (opportunistic, ≤4 evictions per insert).
    • DB hydration on re-entry: when an evicted IP returns, store.LoadJournalIPStats(ip) loads persisted counters before the next upsert, so the running total is preserved across in-memory eviction cycles.
  • web.geoResolver.cache becomes an LRU capped at 4096 entries (override via SHARDLURE_GEO_CACHE_MAX), with promotion on hit and opportunistic expiry sweep on insert.

Expected ceiling at saturation is now in the low tens of MB rather than unbounded. Verified on the Oracle ARM honeypot: sys=29.8 MB, heapAlloc=2–3 MB, 15 goroutines, flat across sampling.

Diagnostics endpoints

All gated behind the existing dashboard token (SHARDLURE_DASH_TOKEN); same auth as the rest of /api/intel/*.

  • /debug/pprof/{,cmdline,profile,symbol,trace} from net/http/pprof.
  • /debug/runtime returns a JSON snapshot of runtime.MemStats plus bounded-cache sizes (liveJournalCollectorIPs, geoCacheEntries, etc.).
  • HTTP WriteTimeout raised from 20s to 60s so /debug/pprof/profile?seconds=30 completes.

Other

Carried forward from the live-capture branch since v1:

  • web: TTY transcripts decoded and surfaced in the intel session modal; shell-session panel populated from live capture; command-column scrub for noisy CRLF.
  • store: migration v3 normalizes command to '' instead of NULL so downstream JOINs/aggregations don't drop rows.
  • scripts/install.sh: auto-detects architecture, downloads the matching binary from this release, and provisions the systemd unit. --tag overrides the auto-resolved latest.

Install

curl -fsSL https://github.com/hett-patell/ShardLure/releases/download/v1.1/install.sh | sudo bash

Or pin explicitly:

sudo TAG=v1.1 bash -c "$(curl -fsSL https://raw.githubusercontent.com/hett-patell/ShardLure/v1.1/scripts/install.sh)"

Upgrade from v1

Stop the service, replace the binary, restart:

sudo systemctl stop shardlure-live
sudo install -m 0755 shardlure-linux-$(uname -m | sed s/x86_64/amd64/;s/aarch64/arm64/) /usr/local/bin/shardlure
sudo systemctl start shardlure-live

Migration v3 runs automatically. No config changes required.

Full changelog

f8890db actor/web: bound live collector + geo cache, gate pprof behind dash token
4166e1c web/store: aggregate payload library by sha256
658bf67 web: surface decoded TTY transcripts in the intel session modal
6001fbe store: migration v3 sets command='' instead of NULL
1e17e5b live capture: TTY transcripts, shell-session panel, command-column scrub
2ec42ef foundation: add CONTRIBUTING, SECURITY, COC + README badges and ecosystem footer
95056ef add install.sh: auto-detect arch, download binary from release, setup systemd

Assets 6
Loading