Userspace-assisted DLP sensor: on Linux, attach eBPF uprobes to OpenSSL SSL_write, copy outbound TLS plaintext chunks to a ring buffer, and scan them with Aho–Corasick substring matching. When a policy pattern matches, the tool logs an alert and a redacted copy of the buffer—useful for catching secrets and PII before they reach AI APIs (or any TLS peer using libssl).

The policy engine, rolling-buffer aggregation, allowlist, dedupe, and alert pipeline live in internal/sensorcore. On Linux, cleartext from eBPF is the primary path; optional pluggable feeds (localhost ingest, stdin JSONL, hookwire socket) can still call HandleChunk / HandleChunkWithMeta for tests and auxiliary capture—see Platforms.

Scope: Linux only for spectral-mesh: full observation and logging with redaction in Go; optional kernel-side blocking (-enable-bpf-blocking) can wipe matching plaintext in the outbound buffer before it is sent (keep it off unless you explicitly want that behavior). The separate spectral-edge binary (Network edge) remains plain Go and can be built for multiple OSes for HTTP-body inspection.

• How it works
• Platforms
• Requirements
• Build
• Run
• CLI flags and environment
• Policy patterns
• BPF blocking and the verifier
• Logging and metrics
• Try it
• Docker
• Network edge (spectral-edge)
• Repository layout
• Testing, CI, and load benchmarks
• HTTPS flood script (no API key)
• OpenAI flood script (live API)
• Simulation scripts
• Tests layout

1. Discovery — Scan $HOST_PROC (default /proc) for */maps entries listing mapped libssl.so (any 1.1 / 3.x), resolve paths under each process root, and attach a uprobe on SSL_write once per library inode. Discovery repeats every 10 seconds to pick up new processes.
2. Capture — The BPF program reserves a ringbuf slot, copies up to 512 bytes from the user buffer, records the chunk length, and submits PID and comm. If ring buffer reservation fails, a per-CPU kernel counter is incremented (exposed as a Prometheus gauge).
3. Match — Go appends each chunk to a per-PID rolling buffer (last ~4 KiB) and runs Aho–Corasick on that buffer. That way a sensitive substring split across multiple SSL_write calls—common with HTTP/2 (small TLS records)—can still match. Match cost scales with rolling-buffer size, not naïvely with pattern count alone. Rolling state for exited PIDs is pruned on the same 10-second tick as discovery. Optional allowlist (-allowlist) skips alerts by process name prefix; dedupe (-alert-dedupe-window) suppresses repeat PID+rule alerts in a time window.
4. Redact — On match, overlapping runs are masked with * in the logged copy. If -enable-bpf-blocking is on and a rule has action: block, the eBPF program may overwrite the user buffer (see docs/PACKAGING.md and BPF blocking below).

Demo (ingest only, no uprobes required for the demo path): Terminal A: spectral-mesh -policy policy.json -capture-ingest-addr 127.0.0.1:9092 — Terminal B: go run ./cmd/spectral-capture-demo -ingest-url http://127.0.0.1:9092/v1/ingest/chunk — expect policy_alert on stdout. Optional -capture-ingest-token / Authorization: Bearer for hardening. (:9092 avoids clashing with Prometheus in monitoring/docker-compose.yml on host :9091.)

-capture-hook-socket unix:… or tcp:… accepts a binary stream documented in internal/capture/hookwire (36-byte little-endian header, max 128 KiB per chunk, then raw TLS plaintext bytes). Use this for lab tests or a custom in-process sender paired with the mesh.

Not covered: TLS stacks without a visible SSL_write uprobe target (e.g. some static builds), BoringSSL inside Chrome, QUIC, or non-OpenSSL TLS implementations without a separate capture path.

For HTTP-body policy testing on any OS without host TLS capture, use spectral-edge (docs/EDGE.md).

• Kernel with BTF available (e.g. /sys/kernel/btf/vmlinux) for CO-RE-style builds.
• Toolchain: clang, llvm, libbpf development headers; bpftool to generate bpf/vmlinux.h.
• Go: version compatible with go.mod (see repo root).
• Privileges: attaching uprobes and loading BPF typically requires appropriate capabilities (often root or CAP_SYS_ADMIN / CAP_BPF depending on the system).

libssl: Probes target the SSL_write symbol in whatever libssl.so processes map (e.g. .so.3 on RHEL 9). Mismatched or statically linked OpenSSL will not be seen.

Generate the kernel BTF header (once per machine or after kernel changes):

bpftool btf dump file /sys/kernel/btf/vmlinux format c > bpf/vmlinux.h

spectral_bpf_generate.go holds //go:generate directives (not behind GOOS tags) so go generate ./... can run from CI or any host with a suitable clang + libbpf toolchain (often Linux or make generate-docker). bpf2go compiles bpf/scrubber.c for amd64 and arm64 BPF targets. Generated artifacts look like spectral_*_bpfel.go and matching .o files (do not edit). The directives pass -no-strip so a host llvm-strip binary is not required.

On Linux (Fedora/RHEL: dnf install clang llvm libbpf-devel; Debian/Ubuntu: apt install clang llvm libbpf-dev):

make generate   # or: BPF2GO_CFLAGS=... go generate ./...
make build      # CGO_ENABLED=1 go build -o spectral-mesh .

Or in one step:

make   # runs generate then build

If your laptop is not a Linux box with libbpf headers, run generation inside Docker (requires Docker Desktop or a compatible engine):

make generate-docker

The Makefile uses an Ubuntu 24.04 image (GO_GEN_IMAGE, overridable) with clang 18 and GOTOOLCHAIN=auto so the Go version matches go.mod. Build the Linux sensor with CGO_ENABLED=1 on a Linux machine (or image) that has libbpf and clang.

Pass -ldflags with Version, GitCommit, and BuildTime the same way as make build when you need embedded build metadata in /version and logs.

rm -f spectral-mesh spectral_*_bpfel.go spectral_*_bpfel.o

sudo ./spectral-mesh

With Prometheus metrics on port 9090 and a policy file:

sudo ./spectral-mesh -metrics-addr :9090 -policy /path/to/policy.json

On startup you should see JSON logs including discover_start and uprobe_attached for hooked libssl paths; leave it running and generate TLS traffic from processes using OpenSSL (e.g. curl, many language HTTP clients).

Policy reload: if -policy points to a file, send SIGHUP to reload patterns without restarting (see docs/RUNBOOK.md).

Build identity: make injects version, git_commit, and build_time into the binary. They appear in the sensor_active log line, on GET /version (JSON, when -metrics-addr is set), and via -version. For what data leaves the host in logs and metrics, see docs/DATA_HANDLING.md.

• Built-in defaults are defined in internal/policy (DefaultPatterns): API keys, PEM markers, Bearer / Authorization:, cloud-style env markers, common token prefixes, PII phrases, and classification text—plus sample strings such as Project Ethos and Internal-Secret.
• External policy — Prefer v2 JSON with schema_version and rules (id, pattern, severity, action: observe or block). Legacy patterns arrays still work.

{
  "schema_version": 2,
  "rules": [
    {"id": "ex", "pattern": "Bearer ", "severity": "high", "action": "observe"}
  ]
}

See policy.example.json. Use -policy /path/to/file.json (no rebuild required). Pattern count and length are capped (internal/policy limits). action: block patterns are limited to 16 bytes and matched in-kernel only in the first ~32 bytes of each captured TLS chunk (verifier-safe bounds); use observe for longer needles (full userspace rolling buffer).

Packaging: systemd example under packaging/systemd/, notes in docs/PACKAGING.md. Helm charts and cloud Terraform are not maintained in this repository; use the Dockerfiles, systemd unit, and runbooks here as a base. Example allowlist: allowlist.example.json.

Linux only. Kernel blocking is off by default (-enable-bpf-blocking). When enabled on Linux, action: block rules are loaded into BPF; on a match, the program zeros the outbound user buffer (in fixed 64-byte chunks and 1-byte tails so the Linux BPF verifier accepts bpf_probe_write_user sizes).

Semantics differ from userspace matching: in-kernel substring search uses short patterns (≤ 16 bytes) and a small scan window in each captured chunk (see bpf/scrubber.c). Userspace still runs the full rolling buffer and longer patterns for observe rules.

Changing bpf/scrubber.c requires go generate ./... and a rebuild; load failures usually mean the verifier rejected the program (bounds, loop complexity, or helper arguments). CI compiles BPF on every push so broken C is caught early.

• Logs are JSON to stdout (log/slog), suitable for shipping to a log stack or SIEM. Alerts use the policy_alert message with schema_version, rule_id, matched_rule_ids, severity, action, enforcement_mode (how this OS applies block rules), bpf_blocking_enabled, kernel_wipe_eligible, kernel_wipe_applied, plus comm, pid, optional redacted_preview / payload_sha256.
• Metrics (when -metrics-addr is set) use Prometheus naming, including:
  • spectral_mesh_ringbuf_events_total — chunks delivered to the policy engine (primarily from the BPF ring buffer; also increments for optional ingest/hook paths that call HandleChunk)
  • spectral_mesh_policy_alerts_total — policy matches actually logged (after rate limiting)
  • spectral_mesh_policy_alerts_rate_limited_total — matches suppressed by rate limits
  • spectral_mesh_ringbuf_reserve_drops_total — Linux: kernel-side bpf_ringbuf_reserve failures; other OSes: always 0
  • spectral_mesh_tls_blocked_writes_kernel_total — Linux: kernel-side plaintext wipes when blocking is enabled; other OSes: always 0
  • spectral_mesh_policy_alert_dedupe_suppressed_total — matches suppressed by the dedupe window
  • spectral_mesh_policy_block_match_observe_only_total — block-rule matches while the kernel wipe path is inactive (e.g. with -enable-bpf-blocking off)
  • spectral_mesh_bpf_blocking_enabled — 1 or 0
  • spectral_mesh_policy_block_rules_configured — number of block rules loaded toward the BPF limit (up to 8)
  • spectral_mesh_uprobe_attach_success_total, spectral_mesh_uprobe_attach_open_errors_total, spectral_mesh_uprobe_attach_probe_errors_total
  • spectral_mesh_roll_buffer_pids_pruned_total, spectral_mesh_ringbuf_read_errors_total
  • spectral_mesh_policy_reloads_total, spectral_mesh_policy_reload_errors_total

With -metrics-addr :9090 (change host/port if you use another address), from the same machine:

# Full Prometheus text exposition
curl -sS http://127.0.0.1:9090/metrics

# Only this binary’s metrics (names prefixed with spectral_mesh_)
curl -sS http://127.0.0.1:9090/metrics | grep '^spectral_mesh'

# Liveness / readiness (HTTP 200 when ready)
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9090/healthz
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9090/readyz

# Build metadata (JSON)
curl -sS http://127.0.0.1:9090/version

curl: (7) Failed to connect means nothing is listening on that port—usually because the process was started without -metrics-addr, or you are curling from another host while the listener is only on the sensor machine (or inside a pod: use kubectl port-forward to 9090). Confirm with ss -tlnp | grep 9090 after starting:

sudo ./spectral-mesh -metrics-addr :9090

Terminal 1

sudo ./spectral-mesh

Terminal 2 — Any HTTPS request whose plaintext (after TLS) contains a forbidden substring should trigger an alert. Example (replace URL and body with something that includes a test pattern from your policy):

curl -sS -X POST https://example.com/ \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Details on Project Ethos"}' >/dev/null

If you see nothing, confirm Terminal 1 logged uprobe_attached for your host’s OpenSSL, and that you rebuilt after BPF changes (go generate + rebuild). For a deterministic single-request test you can force HTTP/1.1 (larger TLS writes): curl --http1.1 ....

curl: (60) SSL certificate problem means OpenSSL could not validate the server chain against your trust store (missing or outdated CA bundle, custom MITM proxy, or minimal container image). Fix the trust store first—on Fedora/RHEL install or refresh CAs: sudo dnf install -y ca-certificates and ensure /etc/pki/tls/certs/ca-bundle.crt exists. Point curl at it explicitly if needed: curl --cacert /etc/pki/tls/certs/ca-bundle.crt .... Lab-only workaround: curl -k skips verification (insecure); TLS still encrypts, but you lose server authentication.

For vendor APIs you need valid credentials and endpoints; the mesh does not depend on a specific provider.

Multi-stage image (UBI 9) builds the binary and ships spectral-mesh under /usr/local/bin/. Building:

docker build -t spectral-mesh .

Running this meaningfully still requires host permissions for BPF and access to host /proc and libraries—typical deployments use a privileged DaemonSet or equivalent, not an isolated default bridge container. Pass flags such as -metrics-addr :9090 via container args if you want metrics inside the pod.

For HTTP(S) after TLS termination (reverse proxy, load balancer, or service mesh), spectral-edge applies the same policy JSON to request bodies, logs policy_alert with sensor_kind: edge, and can reverse-proxy to -upstream after scanning. No BPF and no CGO.

make edge
./spectral-edge -listen 127.0.0.1:8080
# optional: TLS on the listen socket
./spectral-edge -listen :8443 -tls-cert /path/to/cert.pem -tls-key /path/to/key.pem
# optional: shared secret for all routes except /healthz and /readyz
./spectral-edge -api-key "$SPECTRAL_EDGE_API_KEY" ...

Distribution: Dockerfile.edge builds a distroless image for spectral-edge only (the root Dockerfile is spectral-mesh). Pushing a v* tag runs .github/workflows/release-edge.yml (multi-arch tarballs, checksums, go version -m text, GHCR image). Details: docs/EDGE.md.

Highlights: configurable -scan-path (default /v1/scan), -proxy-mount-path to scope which URL prefixes are scanned, -trusted-proxy-cidrs for safe X-Forwarded-For handling, optional -http-ratelimit-rps (per-IP HTTP throttle before auth), SIGHUP policy reload, allowlist user_agent_prefixes, alert dedupe and rate limits, traceparent → trace_id on alerts, Prometheus histograms and extra counters, and OpenAPI served at GET /openapi.yaml (source: cmd/spectral-edge/openapi.yaml).

Policy + mesh: docs/POLICY_MESH_AND_EDGE.md. Edge production runbook: docs/EDGE_PRODUCTION.md.

Reviewers: docs/EDGE_THREAT_MODEL.md, docs/EDGE_VS_MESH.md, docs/EDGE_CAPACITY.md.

Full flag list and operations: docs/EDGE.md.

Tests: go test ./cmd/spectral-edge/... -short (userspace only; no BPF). Includes reverse-proxy integration tests against httptest upstreams, BenchmarkEdgeProxyThroughput, testutil checks on Prometheus counters (dedupe, allowlist, rate limits), custom scan-path, API key auth, /metrics, and shadow-mode cases (block rules alert but default -reject-on-block-rule still forwards traffic—see shadow_edge_test.go). The repository root package spectral-mesh also has tests that require generated BPF artifacts—run make generate (or make generate-docker) before go test ./... if those files are missing. Full map: tests/README.md.

The daemon needs root/CAP_BPF and a suitable kernel to run; automated tests focus on userspace logic that does not load BPF.

spectral-edge (cmd/spectral-edge/) has dedicated tests (mux, auth, client IP, /v1/scan, proxy integration, shadow vs reject-on-block behavior, rate limits, body codec). go test ./... at the repo root compiles the spectral-mesh package for the current GOOS and needs go generate output (spectral_*_bpfel.*) so Linux tests can parse embedded BPF. If you only work on edge, run go test ./cmd/spectral-edge/... ./internal/... -short without generating BPF.

# Same checks as CI (gofmt, vet, test, go generate, CGO build)
make ci

# Unit tests (policy, rollbuf, fleet simulations) — verbose lines per test
make test

# Throughput and allocation profiles for matcher + rolling buffer (~few seconds)
make bench

# Include the heavier prune stress test (~15k ghost PIDs)
go test ./tests/fleet/...

Plain go test ./... only prints package names when everything passes; use make test (adds -v) or go test -v ./... if you want each test name on screen.

CI (.github/workflows/ci.yml): on each push/PR to main or master, linux/amd64 and linux/arm64 jobs each install libbpf, clang, and llvm, then run gofmt, go vet, go test -short, go generate ./... (bpf2go for both architectures), CGO_ENABLED=1 go build (spectral-mesh for Linux), and CGO_ENABLED=0 go build (spectral-edge). Concurrency: newer runs cancel older ones on the same branch.

Releases (spectral-edge): pushing a v* tag runs .github/workflows/release-edge.yml — static tarballs (linux/darwin × amd64/arm64), checksums, go version -m output, GitHub Release assets, and a GHCR image ghcr.io/<owner>/<repo>/spectral-edge:<tag> (see docs/EDGE.md).

Prometheus + Grafana: see monitoring/README.md for Docker Compose (Prometheus on host :9091 scrapes spectral-mesh on :9090 and spectral-edge on :8080 by default; Grafana on :3000 with provisioned Spectral mesh / Spectral edge dashboards). Example edge alert rules: monitoring/prometheus/spectral-edge-rules.yml (merge into your Prometheus config as needed; not mounted by default in Compose).

Real end-to-end proof still requires running spectral-mesh on a host or VM with TLS workload generators; repository tests approximate fleet-scale PID churn and sustained chunk processing on the Go side only. On linux/amd64 and linux/arm64, TestSpectralEmbeddedCollectionSpec verifies the embedded BPF ELF parses (no kernel load).

scripts/flood_https.sh runs parallel curl requests to public HTTPS sites (default GET https://example.com, or POST JSON to https://httpbin.org/post). No API key — useful for general TLS/SSL_write load and metrics checks. Optional TRIGGER_SAMPLE_POLICY=1 with MODE=post puts Project Ethos in the JSON body so built-in policy may emit policy_alert (smoke test).

sudo ./spectral-mesh -metrics-addr :9090   # terminal 1
./scripts/flood_https.sh                 # terminal 2 — default GET flood

MODE=post COUNT=30 ./scripts/flood_https.sh
MODE=post TRIGGER_SAMPLE_POLICY=1 COUNT=10 ./scripts/flood_https.sh

If curl: (60) SSL certificate problem: install OS CA data (sudo dnf install -y ca-certificates on Fedora/RHEL), or set CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt, or for lab-only CURL_INSECURE=1 (same as curl -k).

scripts/flood_openai.sh runs many parallel curl calls to OpenAI’s Chat Completions endpoint (api.openai.com), which drives real SSL_write traffic through OpenSSL (good for exercising the sensor and metrics). This uses your API key and costs money; you may hit HTTP 429 if CONCURRENCY is too high.

# Terminal 1 — sensor + metrics
sudo ./spectral-mesh -metrics-addr :9090

# Terminal 2 — flood (default 50 requests, concurrency 5)
export OPENAI_API_KEY=sk-...
./scripts/flood_openai.sh

# Or: put the key in a file (first line only), then:
# OPENAI_API_KEY_FILE="${HOME}/.openai_api_key" ./scripts/flood_openai.sh

# Heavier load
COUNT=200 CONCURRENCY=15 ./scripts/flood_openai.sh

Tune MODEL and MAX_TOKENS (see script header) to balance cost vs. response size.

Package-level tests live next to code (internal/*, cmd/spectral-edge, root *_test.go). Heavier simulations sit under tests/. For a file-by-file map of spectral-edge tests, shadow-mode coverage, make test vs CI, and BPF prerequisites, see tests/README.md.

Built as a founder-side project for kernel-adjacent security experiments.