Xtate — a modular all-stack network scanner for Internet-scale surveys

Stateless TCP banners. Hybrid-state TCP/TLS. 83 registered modules.
_{C99 · Linux & Windows · IPv4 & IPv6 · packets first, crawlers never}

God is Stateless. — the Xtate proverb

Xtate is a network scanner for large-scale active measurement. It splits the scanning stack into four pluggable module types — Scan (L3/L4 packets), Probe (L5+ payloads and parsers), Generate (targets) and Output (results) — and ships two original engines:

• ZBanner obtains real TCP responses without keeping per-target connection state.
• HLTCP is a hybrid-state user-space TCP stack that completes TLS handshakes and fetches HTTPS responses when statelessness is not enough.

It is not a crawler, and it does not pretend to be one. It is a packet engine with just enough protocol state to get answers that most fast scanners cannot.

Why · Quick start · Architecture · Modules · Build · CLI · Docs · Papers · License

English · 中文

Most fast scanners stop at "port open"; most service scanners give up at Internet scale. Xtate lives in the overlap.

• ⚡ ZBanner — real TCP responses without a per-target connection table. The peer's TCP stack holds the state; a SipHash cookie proves the reply is yours.
• 🧬 HLTCP — a hybrid-state user-space TCP stack, created only where needed. It completes TLS handshakes and speaks HTTP inside them.
• 🧩 Modules — 11 scan / 60 probe / 7 output / 5 generate modules. Four interfaces, one pipeline, no runtime plugin magic.

# Linux: build from source (dependencies are covered under Build)
git clone https://github.com/babycoff/xtate.git
cd xtate
./build.sh

# A classic SYN scan — no firewall tricks needed
sudo ./bin/xtate -ip 192.168.1.0/24 -p 80,443 -scan tcp-syn -rate 1000

For a real banner-grabbing handshake, stop the Linux kernel from resetting connections it does not know about:

./firewall/add_rules.sh
sudo ./bin/xtate -ip 192.168.1.0/24 -p 80,8080 -scan zbanner -probe http -show info
./firewall/rm_rules.sh

On Windows, install Npcap, build with CMake, and run xtate.exe from an elevated shell. See Build.

# Stateless TLS hello: grab ServerHello hints without a full handshake
sudo ./bin/xtate -ip 10.0.0.0/24 -p 443 -scan zbanner -probe tls-hello -show info

# UDP with a generic echo probe
sudo ./bin/xtate -ip 10.0.0.0/24 -p u:80 -scan udp -probe echo -show info

# LAN ARP sweep
sudo ./bin/xtate -ip 192.168.1.0/24 -scan arp-req -lan

# IPv6 neighbor discovery on a local link
sudo ./bin/xtate -ip fe80::1/120 -scan ndp-ns -src-ip fe80::2 -fake-router-mac

# SCTP INIT scan
sudo ./bin/xtate -ip 10.0.0.0/24 -p s:38412 -scan sctp-init -show info

One Tx thread per shard walks the target space, asks the ScanModule for a packet, and hands it to the fastest available raw path. One Rx thread parses, validates and deduplicates replies, then dispatches by target IP so a given host always lands on the same handle thread.

• Tx: N threads; target generation, rate control, packet construction, raw send.
• Rx: one thread; preprocess, validate, dedup, optional pcap capture, then dispatch.
• Handle: M threads; scan/probe state machines, result formatting, follow-up packets.
• Stateful scans: one TCP connection table per handle thread, so the hot path needs no locks.

Handle threads inject follow-up packets (ACK, RST, next probe) through a lock-free ring back to Tx, which is how a "stateless" Rx path still drives a real TCP conversation.

Under the hood, briefly:

• Packet templates are pre-baked at startup and patched per target; checksums are recomputed.
• The target space is flattened to ip × port and permuted with BlackRock; source IP and port can be randomized too (63-bit scan space).
• Results are an OutItem with three typed key-value chains (scan, probe, output), streamed to whichever OutputModule you chose.
• --pcap, --resume, --shard and a live status bar are built in.

The flagship trick, in one sequence:

No TCB. No timeout table. No retransmission bookkeeping. Just arithmetic that the peer's TCP stack helps you verify. When that is not enough, tcp-state takes over and HLTCP keeps the minimum state needed for real TLS/HTTP conversations.

Counts below assume a full build with all optional dependencies enabled; modules whose libraries are missing simply disappear.

• General: http (+ state), hello (+ UDP/state), tls-hello, tls-state, dns, snmp, echo, jarm.
• Fingerprinting: recog (+ UDP/state, Recog XML), nmap (+ UDP, nmap-service-probes), lzr (38 protocol handshakes behind one aggregator).
• Scriptable: lua-tcp, lua-udp — drop in a Lua script and you have a probe.
• Plumbing: get-state, close-state, null.

• Output: text, csv, ndjson, bson, mongodb, list, null.
• Generate: blackrock (default, stateless permutation), ip-list, addr-list, ip-stream, addr-stream.

Writing your own is intentionally boring: each module is a struct plus one line in a NULL-terminated registry, and the same metadata drives --help-*, parameter parsing, fuzzy search and interactive completion. Look for REGISTER YOUR ... MODULE HERE in src/scan-modules/scan-modules.c and friends.

The README is the landing page. The manual lives in docs/:

• Getting started — build, permissions, first scan, firewall
• Targets and ports — IPv4/IPv6 input, files, ASN, excludes, sharding
• Scanning modes — stateless vs stateful, scan/probe pairings, recipes
• Output — result levels, formats, MongoDB
• Modules — the four module types and what ships with Xtate
• Architecture · ZBanner · HLTCP
• Troubleshooting · Known limitations

中文文档：docs/zh-CN/

sudo apt install build-essential cmake pkg-config libpcap-dev \
  libssl-dev libpcre2-dev libxml2-dev libbson-dev libmongoc-dev
./build.sh                 # release build → bin/xtate
./build.sh debug clang     # debug build with Clang

Ubuntu 22.04/24.04 are the reference environments. On Fedora/Arch, install the equivalent development packages (libpcap, openssl, pcre2, libxml2, libbson, mongo-c-driver).

PCRE2 discovery uses CMake's package config when available and falls back to pkg-config on Debian/Ubuntu, so pkg-config is required alongside libpcre2-dev there.

Recommended compilers: GCC, Clang, MSVC and MinGW-w64. Xtate requires a 64-bit little-endian host; CMake stops early otherwise.

scoop install openssl-lts lua54

cmake -S . -B build-msvc -G "Visual Studio 17 2022" -A x64 `
  -DOPENSSL_ROOT_DIR="$env:USERPROFILE\scoop\apps\openssl-lts\current"
cmake --build build-msvc --config Release -j 8

cmake -S . -B build-mingw -G "MinGW Makefiles" `
  -DCMAKE_C_COMPILER=gcc -DCMAKE_BUILD_TYPE=Release
cmake --build build-mingw -j 8

xtate --usage                 # short usage and examples
xtate --help                  # every global parameter
xtate --intro                 # the architecture tour
xtate --list-scan             # 11 scan modules
xtate --list-probe            # 60 probe modules
xtate --list-out              # 7 output modules
xtate --list-gen              # 5 generate modules
xtate --help-scan zbanner     # module-specific parameters
xtate --interact              # interactive shell with tab completion

Xtate's two engines come from our research:

• ZBanner: Fast Stateless Scanning Capable of Obtaining Responses over TCP — arXiv:2405.07409 · IPCCC 2024
• Efficient Application-Layer Scanning with Hybrid-State Lightweight TCP Stack for TLS-based Service Monitoring — ICCT 2024
• Master's thesis: Research on Technologies of Network Protocol and Device Identification Based on Application-Layer Scanning (NUDT, 2024)

@misc{chen2024zbanner,
  title         = {ZBanner: Fast Stateless Scanning Capable of Obtaining Responses over TCP},
  author        = {Chen, Chiyu and Lu, Yuliang and Yang, Guozheng and Xie, Yi and Guo, Shasha},
  year          = {2024},
  eprint        = {2405.07409},
  archivePrefix = {arXiv},
  primaryClass  = {cs.NI}
}

@inproceedings{chen2024hltcp,
  title     = {Efficient Application-Layer Scanning with Hybrid-State Lightweight TCP Stack for TLS-based Service Monitoring},
  author    = {Chen, Chiyu and Lu, Yuliang and Yang, Guozheng and Xie, Yi and Guo, Shasha},
  booktitle = {2024 IEEE 24th International Conference on Communication Technology (ICCT)},
  pages     = {2067--2073},
  year      = {2024},
  doi       = {10.1109/ICCT62411.2024.10946306}
}

If Xtate helps your research, a citation is appreciated more than a star.

AGPL-3.0. See LICENSE. If you run a modified Xtate as a network service, the AGPL's network-use clause applies.

Xtate started from Masscan's codebase; thanks to Robert Graham and every upstream project noted in the source. Xtate is a member of 404Starlink.

Created by Alvin Chen (陈驰昱) · shineccy@aliyun.com. The original code was written during his master's studies at the National University of Defense Technology and the Anhui Province Key Laboratory of Cyberspace Security Situation Awareness and Evaluation.