Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ bytes = "1"
clap = { version = "4", features = ["derive"] }
chrono = { version = "0.4", default-features = false, features = ["clock"] }
regex = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"

# --- TUI (`--tui`) ---
# ratatui re-exports crossterm (its default backend), so we use that for events.
Expand Down Expand Up @@ -50,3 +52,4 @@ rcgen = "0.14"
# is dev-only: it never reaches a normal `cargo build`/release/`cargo install`.
[dev-dependencies]
clap_mangen = "0.2"
tempfile = "3"
32 changes: 32 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tapgres -p 5432 -i eth0 # capture a specific interface
tapgres --mode mitm \ # decode an encrypted session via the proxy
--listen 127.0.0.1:15432 --upstream 127.0.0.1:5432
tapgres --tui -Y 'message.type == "Query"' # interactive view, filtered
tapgres --save session.jsonl # capture and tee every record to disk
tapgres --replay session.jsonl --tui # reopen it without live capture
```

For making a client trust the mitm proxy's auto-generated CA, see
Expand Down Expand Up @@ -60,12 +62,42 @@ sudo setcap cap_net_raw+ep $(which tapgres)
| `w` / `r` | wrap / rich display |
| `c` | clear |
| `y` | edit the display filter |
| `/` / `:` | command bar (`:save FILE`, `:open FILE`) |
| `Esc` | clear the display filter |

Display filters (`-Y` / `--display-filter`) use a small typed expression
language with fields like `message.type`, `message.text`, `client.ip`, and
`client.port`. See `man tapgres` for the full field and operator reference.

## Save and replay

`--save FILE` continuously writes every output record to versioned JSONL while
stdout or the TUI continues normally. Recording happens before display
filtering and before the TUI's 50,000-record history cap, so hidden or evicted
live records are still saved. An existing destination is replaced.

`--replay FILE` uses a saved session instead of pcap/mitm capture. Replay is
instant, preserves the original timestamps and structured rich-view data, and
passes through the same display filters and renderers as live traffic:

```sh
tapgres --replay session.jsonl
tapgres --replay session.jsonl --tui --tui-rich
tapgres --replay session.jsonl -Y 'message.type == "Query"'
```

In the TUI, `/` or `:` opens the command bar. `:save FILE` (also `:w`) writes
the currently retained events and then continuously records future traffic.
If older events have already left the TUI history, the footer reports the
omission. `:open FILE` (also `:o`) validates the complete file, replaces the
current view with its newest 50,000 records, and switches the session to replay
mode. It closes any active recorder, and subsequent live-source records are
discarded so live and replayed timelines never mix.

Schema version 1 and its compatibility rules are defined in
[`docs/session-format.md`](docs/session-format.md). Unsupported schema versions
and malformed records are refused with a file and line-numbered error.

## Installation

**Prebuilt binary** (Linux x86_64, from
Expand Down
82 changes: 82 additions & 0 deletions docs/session-format.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Tapgres saved-session format

Tapgres saves sessions as UTF-8 JSON Lines (JSONL): one complete JSON object per
line. Version 1 is designed to round-trip every `decode::Output` variant through
the same display-filter and rendering pipeline used by live capture.

The format is local and stream-oriented. `--save` and `:save` replace an
existing destination, then record events before display filtering or TUI
history eviction. Files may contain sensitive SQL, credentials, errors, and
returned row values and should be protected accordingly.

## Common fields

Every line contains:

| Field | Type | Meaning |
| --- | --- | --- |
| `schema_version` | unsigned integer | Version of this record shape; currently `1`. |
| `timestamp` | RFC 3339 string | Original message capture time, or record time for operational lines/status. |
| `record_type` | string | `message`, `line`, or `status`. |

Blank lines are ignored. A malformed record aborts replay with the file path and
line number.

## Message records

A decoded PostgreSQL message contains the structured values required by display
filters and rich rendering:

```json
{"schema_version":1,"timestamp":"2026-07-17T12:34:56.789+01:00","record_type":"message","direction":"f2b","message_type":"Query","text":"SELECT * FROM orders","rendered":"[12:34:56.789] [F→B] Query: SELECT * FROM orders","client":"127.0.0.1:40005"}
```

| Field | Type | Meaning |
| --- | --- | --- |
| `direction` | string | `f2b` for frontend/client to backend/server, or `b2f` for the reverse. |
| `message_type` | string | Decoded pgwire message name, such as `Query` or `DataRow`. |
| `text` | string | Decoded message body used by `message.text` filters. |
| `rendered` | string | Stable flat terminal representation, including the original display timestamp. |
| `client` | socket-address string | Owning connection's client IP and port. |
| `detail` | object, optional | Structured rich-rendering payload described below. |

### Rich detail

`RowDescription` preserves field names, PostgreSQL type OIDs, and format codes:

```json
{"detail_type":"row_description","columns":[{"name":"id","type_oid":23,"format_code":0}]}
```

`DataRow` preserves its already-decoded display value alongside the cached
column name and type OID:

```json
{"detail_type":"data_row","columns":[{"name":"id","type_oid":23,"value":"'1'"}]}
```

The detail object is nested under the message record's `detail` field. Keeping
both forms allows replay to reproduce rich tables without re-decoding pgwire
bytes while retaining the stable flat transcript.

## Operational records

Connection/capture lines and status records retain their original text:

```json
{"schema_version":1,"timestamp":"2026-07-17T12:34:56.789+01:00","record_type":"line","text":"=== new connection 127.0.0.1:40005 -> 127.0.0.1:5432 ==="}
{"schema_version":1,"timestamp":"2026-07-17T12:34:56.790+01:00","record_type":"status","text":"tapgres: capturing on 'lo'"}
```

They remain outside display filtering, matching live behavior.

## Compatibility policy

- Tapgres writes only the current schema version.
- Version 1 readers require `schema_version: 1` on every non-blank line.
- Unknown older or newer versions are refused; there is no silent best-effort
conversion.
- A future incompatible shape must increment `schema_version` and provide an
explicit migration path if backward compatibility is desired.
- Replay preserves recorded `rendered` output rather than reformatting it, while
filters and rich mode use the structured fields.
8 changes: 4 additions & 4 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@

# Source cleaning: keep cargo's own selection (crane's
# commonCargoSources — every .rs/.toml/Cargo.lock across the workspace),
# plus the committed `man/sections.md` that the gen_manpage example
# embeds with include_str!. cleanCargoSource alone strips it (cargo
# doesn't track it), which breaks the example's build; fileset.toSource
# makes the extra include explicit and unambiguous.
# plus committed non-Rust inputs used by builds and tests. Cargo does
# not track these files, so commonCargoSources strips them unless the
# fileset includes them explicitly.
src = pkgs.lib.fileset.toSource {
root = ./.;
fileset = pkgs.lib.fileset.unions [
(craneLib.fileset.commonCargoSources ./.)
./man/sections.md
./tests/fixtures/session-v1.jsonl
];
};

Expand Down
29 changes: 29 additions & 0 deletions man/sections.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,32 @@
# SAVED SESSIONS

`--save FILE` continuously writes every output record as versioned JSONL while
normal stdout or TUI rendering continues. Recording occurs before display
filtering and before the TUI history cap, so hidden and evicted live records
remain in the saved session. Operational line/status records are saved along
with decoded PostgreSQL messages. An existing destination file is replaced.

`--replay FILE` reads a saved session instead of starting pcap or mitm. Records
are replayed immediately through the same stdout/TUI renderer and display
filter as live traffic. Original capture timestamps, client address, direction,
message type/text, and rich RowDescription/DataRow details are preserved. A
replay can be copied to another file with `--save`; the input and output paths
must differ.

The TUI command bar opens with `/` or `:`. `:save FILE` (`:w FILE`) writes the
currently retained history and continuously records future events. If earlier
events have left the 50,000-record TUI history, a footer warning reports the
omission. `:open FILE` (`:o FILE`) validates the complete file before replacing
the view, retains its newest 50,000 records, switches the UI to replay mode, and
closes any active recorder. Subsequent live-source display records are discarded
so timelines do not mix.

The current on-disk schema version is 1. Each JSONL record carries its own
`schema_version` and RFC 3339 timestamp. Unknown versions and malformed records
are refused with the file path and line number; tapgres does not guess at an
incompatible shape. The full format is documented in `docs/session-format.md`
in the source repository.

# DISPLAY FILTER EXPRESSIONS

The `-Y` / `--display-filter` option limits decoded PostgreSQL messages in
Expand Down
64 changes: 62 additions & 2 deletions src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,16 @@ use crate::state;
with the pgwire protocol layer. Use --mode pcap (the default) to passively \
capture a local port with libpcap (cleartext only), or --mode mitm to run a \
local TLS-terminating proxy that decrypts encrypted sessions. Add --tui to \
either source for an interactive, scrollable, filterable view.",
either source for an interactive, scrollable, filterable view. Use --save to \
record versioned JSONL or --replay to open a saved session without capture.",
before_help = crate::tui::BANNER
)]
pub struct Args {
/// Traffic source.
#[arg(long, value_enum, default_value_t = Mode::Pcap)]
pub mode: Mode,

/// Interactive TUI instead of line-oriented stdout (works with any --mode).
/// Interactive TUI instead of line-oriented stdout (works with live or replay sources).
#[arg(long, default_value_t = false)]
pub tui: bool,

Expand All @@ -45,6 +46,34 @@ pub struct Args {
#[arg(short = 'Y', long = "display-filter")]
pub display_filter: Option<DisplayFilter>,

/// Save every live or replayed output record as versioned JSONL while
/// continuing to render normally. Recording happens before display
/// filtering and before the TUI history cap is applied. An existing file
/// is replaced.
#[arg(long, value_name = "FILE")]
pub save: Option<PathBuf>,

/// Read a saved JSONL session instead of starting pcap or mitm capture.
/// Replay is loaded at full speed and preserves original timestamps.
#[arg(
long,
value_name = "FILE",
conflicts_with_all = [
"mode",
"port",
"interface",
"no_promisc",
"snaplen",
"listen",
"upstream",
"tls_dir",
"tls_cert",
"tls_key",
"no_upstream_tls"
]
)]
pub replay: Option<PathBuf>,

/// Maximum retained open + recently-closed connection records.
/// Open connections are never evicted.
#[arg(long, default_value_t = state::DEFAULT_CONNECTION_CAP)]
Expand Down Expand Up @@ -120,3 +149,34 @@ pub enum Mode {
pub fn command() -> clap::Command {
Args::command()
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn parses_save_and_replay_as_file_source_options() {
let args = Args::try_parse_from([
"tapgres",
"--replay",
"capture.jsonl",
"--save",
"copy.jsonl",
"--tui",
])
.unwrap();

assert_eq!(args.replay, Some(PathBuf::from("capture.jsonl")));
assert_eq!(args.save, Some(PathBuf::from("copy.jsonl")));
assert!(args.tui);
}

#[test]
fn replay_rejects_live_source_options() {
let error =
Args::try_parse_from(["tapgres", "--replay", "capture.jsonl", "--mode", "mitm"])
.unwrap_err();

assert_eq!(error.kind(), clap::error::ErrorKind::ArgumentConflict);
}
}
Loading