Consolidates the three multi-layer fingerprinting capture services behind one gRPC collector that merges every layer of a logical request into a single Postgres row. This is the device-side plane (it fingerprints the visitor being scored); it is distinct from, and a consumer of, the intel data plane.
Retires the old scapy
../tls-tcp-fingerprintservice. That directory is left untouched but is no longer part of this plane — do not deploy it.
┌──────────────── tcpstack/ (l347) ───────────────┐
client ───► │ fp-sniffer (AF_PACKET) ─► /dev/shm ─► FastAPI │ TCP/IP fp
:8443 │ /fingerprintme?request_id=… (kernel terms TLS) │ + JA3/JA4
HTTPS └──────────────────────────────────────────────────┘ + timings
│
┌──────────────── http2fp/ (h2) ───────────┤
client ───► │ asyncio TLS-terminating (ALPN h2) │ Akamai
:8444 │ reads opening frame burst ─► h2 fp │ HTTP/2 fp
HTTPS └───────────────────────────────────────────┤
│ records (dict)
┌──────────────── http3fp/ (h3) ───────────┤ via common/sink.py
client ───► │ aioquic QUIC endpoint (ALPN h3) │
:8443/udp │ H3 SETTINGS + pseudo-header order ─► fp │
└───────────────────────────────────────────┘
│ GrpcSink
▼ grpc://…:50055
┌───────────── collector/ ──────────────┐
│ grpc.aio FingerprintCollector │
│ SendFingerprint UPSERT/MERGE by │
│ request_id across layers │
│ Health -> stored count │
└────────────────┬───────────────────────┘
▼
Postgres fp.fingerprints
(one merged row per request_id)
Each capture service imports make_sink from common/sink.py and emits a
unified record dict. With FP_SINK=grpc://127.0.0.1:50055 (the default) the
record is shipped to the collector; stdout / file: / postgres:// still
work for local debugging.
The three layers are merged on request_id. A client that sets the same
x-request-id (h2/h3 request header) or ?request_id= (l347 query param)
across its connections gets all three layers folded into one row. If absent, a
per-observation key is synthesized from client_ip:client_port:int(ts) so
nothing is dropped (but layers won't merge).
| Service | Bind | Protocol |
|---|---|---|
fp-l347 (tcpstack) |
:8443 TCP |
HTTPS, kernel-terminated; sniffer reads SYN+ClientHello |
fp-sniffer |
(no socket) | AF_PACKET capture → /dev/shm (needs CAP_NET_RAW) |
fp-h2 (http2fp) |
:8444 TCP |
HTTPS, ALPN h2 |
fp-h3 (http3fp) |
:8443 UDP |
QUIC, ALPN h3 |
fp-collector |
:50055 TCP |
gRPC (insecure, localhost) |
l347 and h3 share :8443 (one is TCP, one is UDP — no conflict).
package layerid.fp.v1
service FingerprintCollector {
rpc SendFingerprint(FingerprintRecord) returns (Ack);
rpc Health(HealthRequest) returns (HealthResponse);
}
message FingerprintRecord {
string request_id=1; double ts=2; string layer=3; // "l347"|"h2"|"h3"
string client_ip=4; int32 client_port=5; string doc_json=6;
string ja3=7; string ja4=8; string os_guess=9; // l347
string h2_hash=10; string h3_hash=11; // h2 / h3
}
message Ack { bool ok=1; string id=2; }
message HealthResponse { bool ok=1; int64 stored=2; }doc_json carries the full per-layer record dict; the typed fields are the
key signals surfaced so the collector can MERGE/COALESCE without re-parsing.
Generate the stubs into common/_gen/ (shared by both the sink and the
collector):
bash common/gen_stubs.sh # needs grpcio-toolsCreated on collector boot (collector/db.py):
CREATE SCHEMA IF NOT EXISTS fp;
CREATE TABLE IF NOT EXISTS fp.fingerprints(
request_id text PRIMARY KEY,
first_seen timestamptz DEFAULT now(),
last_seen timestamptz DEFAULT now(),
client_ip inet,
client_port int,
ja3 text, ja4 text, os_guess text, -- from l347
h2_hash text, -- from h2
h3_hash text, -- from h3
layers text[] DEFAULT '{}', -- which layers contributed
doc jsonb DEFAULT '{}' -- { "<layer>": <per-layer record> }
);SendFingerprint UPSERTs by request_id:
- appends
layertolayers(de-duplicated); COALESCEs the key fields — a non-empty incoming value fills a NULL, but an empty incoming value never clobbers a set one (so layer order doesn't matter);- bumps
last_seen; - folds the per-layer doc in as
doc = doc || jsonb_build_object(layer, <doc_json>).
Idempotent Ubuntu host bootstrap. Assumes the repo is synced to /opt/fps.
sudo bash /opt/fps/deploy/install.sh --iface eth0 --vip <endpoint-ip>It installs OS deps, a fps system user, a /opt/fps/.venv with all runtime
deps (fastapi uvicorn hpack aioquic grpcio grpcio-tools asyncpg), runs make
in tcpstack/, generates the gRPC stubs, writes a self-signed cert to
/etc/fps/{cert,key}.pem, provisions the fp Postgres DB owned by fps,
writes /etc/fps/*.env, and installs + enables the five systemd units
(deploy/systemd/):
fp-collector ─ before ─► fp-sniffer ─ before ─► fp-l347
fp-h2
fp-h3
The collector + postgres come up before the emitters; the sniffer comes up
before the l347 endpoint (which is useless without shm). fp-sniffer runs with
AmbientCapabilities=CAP_NET_RAW + NoNewPrivileges=true as the dedicated
fps user; the three python services run from /opt/fps/.venv/bin/python.
The script prints verification curls at the end.
# 1. collector (creates the schema on boot)
cd collector && POSTGRES_DSN=postgresql://localhost/fp ./run.sh
# 2. each capture service (defaults to grpc://127.0.0.1:50055)
cd tcpstack/service && PORT=8443 ./run.sh # needs fp-sniffer + a built tcpstack
cd http2fp && PORT=8444 ./run.sh
cd http3fp && PORT=8443 ./run.sh| Path | Role |
|---|---|
proto/fingerprint.proto |
the gRPC contract (layerid.fp.v1) |
common/sink.py |
shared sink: stdout/file/postgres + GrpcSink |
common/gen_stubs.sh |
protoc → common/_gen/ (with relative-import fix) |
common/_gen/ |
generated fingerprint_pb2{,_grpc}.py |
collector/ |
grpc.aio server + asyncpg store (server.py, db.py) |
tcpstack/ |
L3/L4 sniffer + FastAPI /fingerprintme (l347) |
http2fp/ |
TLS-terminating HTTP/2 fingerprinter (h2) |
http3fp/ |
aioquic HTTP/3 fingerprinter (h3) |
deploy/systemd/ |
the five unit files |
deploy/install.sh |
host bootstrap |