Jailbreak

A C++ library that reads PostgreSQL heap files and MySQL InnoDB pages directly from disk — bypassing the server, wire protocol, and network entirely — and exposes each table as an Apache Arrow RecordBatch via the Arrow C Data Interface (zero-copy hand-off to Python, DuckDB, DataFusion, Spark, cuDF).

Benchmarked on TPC-H SF=1 lineitem (6M rows, 0.86 GB) on AWS EC2 g4dn.xlarge (T4).

---

Key results

Loading: PostgreSQL → Arrow (lineitem, 6M rows)

Method	Time	Mrows/s	GB/s	C++ speedup
C++ Arrow (this)	0.81s	7.4	1.06	—
C++ Arrow + DuckDB	0.81s	7.4	1.06	~1×
C++ Arrow + DataFusion	0.81s	7.4	1.06	~1×
DuckDB postgres_scanner	4.07s	1.47	0.21	5×
ConnectorX	5.19s	1.16	0.17	6.4×
Spark JDBC	14.08s	0.43	0.06	17×
pandas (psycopg2)	16.72s	0.36	0.05	20×

C++, DuckDB, and DataFusion are statistically identical — con.register() and register_record_batches() are zero-copy Arrow handoffs with negligible overhead.

BRIN-pruned scan (l_orderkey BETWEEN 1 AND 1,500,000 — 25% of table)

Method	Time	Mrows/s	C++ BRIN speedup
C++ BRIN-pruned	0.20s	7.5	—
Spark BRIN (C++ → Spark)	0.81s	1.86	4×
C++ full scan + filter	0.88s	1.71	4.4×
DuckDB pg + WHERE	1.05s	1.43	5.2×
Spark JDBC + WHERE	3.49s	0.43	17×

75% of heap pages are skipped via fseek — the BRIN index (40 KB) is read directly from disk, no server involved.

ETL: PostgreSQL → Parquet (lineitem, 6M rows, end-to-end)

Engine	C++ reader	Wire protocol	C++ speedup
Spark RAPIDS	3.57s	28.56s (JDBC)	8.0×
cuDF	3.60s	19.27s (ConnectorX)	5.4×
DataFusion	5.88s	21.63s (ConnectorX)	3.7×
pyarrow	6.03s	21.25s (ConnectorX)	3.5×
Spark	12.13s	29.86s (JDBC)	2.5×
DuckDB	4.96s	8.62s (postgres_scanner)	1.7×

ETL: MySQL InnoDB → Parquet (lineitem, 6M rows, end-to-end)

Engine	C++ reader	Wire protocol	C++ speedup
pyarrow	7.35s	209s (pymysql)	28.4×
Spark RAPIDS	4.35s	32.29s (JDBC)	7.4×
DuckDB	6.45s	39.42s (mysql_scanner)	6.1×
Spark	11.37s	34.35s (JDBC)	3.0×
cuDF	5.18s	14.10s (ConnectorX)	2.7×
DataFusion	7.25s	15.92s (ConnectorX)	2.2×

cuDF analytics (warm data already on GPU)

Query	cuDF (GPU)	DuckDB (CPU)	GPU speedup
Q1 — full-table groupby	0.113s	0.294s	2.6×
Q6 — selective scan	0.012s	0.087s	7.2×

C++ Arrow ingestion into cuDF (2.77s) is 6.65× faster than ConnectorX (18.4s).

TPC-H queries (7 queries, best of 5 runs, data pre-loaded)

Query	C++ + DataFusion	C++ + DuckDB	vs DuckDB scanner	vs Spark JDBC
Q1 — full-scan aggregate	0.024s	0.029s	68–82×	187–224×
Q3 — 3-table join	0.007s	0.040s	28–157×	39–219×
Q5 — 6-table join	0.048s	0.060s	25–31×	46–57×
Q6 — selective scan	0.005s	0.009s	18–34×	23–44×
Q10 — 4-table join	0.022s	0.048s	14–31×	57–124×
Q12 — 2-table + CASE	0.036s	0.022s	21–29×	47–65×
Q14 — lineitem ⋈ part	0.007s	0.012s	14–25×	23–40×

All 7 queries verified correct across all methods (21/21 checks PASS).

---

How it works

PostgreSQL reader (src/pg_arrow.cpp)

PostgreSQL stores tables in fixed-size 8 KB pages in $PGDATA/base/<db_oid>/<rel_oid>. The C++ reader:

1. Reads the heap file in batches via fread (no server, no socket).
2. Walks each page's ItemIdData slot array, dereferences live HeapTuple records.
3. Applies MVCC visibility checks (infomask bits, xmax).
4. Decodes each column inline: bool, int2/4/8, float4/8, numeric, date, timestamp, bpchar, varchar, text.
5. Returns an ArrowSchema + ArrowArray pair via the Arrow C Data Interface.

On the Python side, pa.RecordBatch._import_from_c() wraps these structs with zero copy.

Optional BRIN-pruned variant (pg_to_arrow_brin): reads the BRIN index file, builds a page mask, and uses fseek to skip pruned page ranges entirely.

MySQL InnoDB reader (src/mysql_arrow.cpp)

Reads InnoDB .ibd files (COMPACT/DYNAMIC row format) directly:

1. Scans 16 KB INDEX pages (type 0x45BF, PAGE_LEVEL=0 for leaf pages).
2. Walks the page's record chain via signed int16 next_record offsets.
3. Skips hidden columns (TRX_ID + ROLL_PTR, 13 bytes).
4. Decodes typed columns: int2/4/8 (big-endian XOR sign), float4/8, date (3-byte XOR 0x80), decimal:M:D (binary groups), char:N/varchar:N/text, raw:N (skip).
5. Returns the same Arrow C Data Interface output as the PostgreSQL reader.

Range scan (mysql_to_arrow_range) enables partitioned parallel reads via Spark.

---

Build

make          # pg_arrow.so     — CPU PostgreSQL reader
make mysql    # _mysql_arrow.so — MySQL InnoDB reader
make clean

---

Requirements

Dependency	Version	Notes
g++	≥ 9	C++17 required
Python	≥ 3.9
PostgreSQL	12–16	Heap files must be readable by the current OS user
MySQL	8.0	.ibd files need o+rx on parent directories
Java	≥ 11	Required for Spark/JDBC benchmarks

Python packages: pip install -r requirements.txt

---

Quick start

# 1. Generate TPC-H SF=1 data and load into PostgreSQL
./scripts/setup_tpch.sh          # optional: ./scripts/setup_tpch.sh 10  (SF=10)

# 2. Build the C++ shared library
make

# 3. Run benchmarks
python benchmarks/loading.py         # full table scan throughput
python benchmarks/brin.py            # BRIN-pruned scan
python benchmarks/etl_benchmark.py   # ETL → Parquet (--db postgres|mysql)
python benchmarks/tpch_multi.py      # TPC-H Q1/Q3/Q5/Q6/Q10/Q12/Q14
python benchmarks/cudf_analytics.py  # cuDF GPU analytics (requires cuDF/RAPIDS)

---

Project structure

.
├── src/
│   ├── pg_arrow.cpp           # PostgreSQL CPU reader (Arrow C Data Interface)
│   └── mysql_arrow.cpp        # MySQL InnoDB reader
├── benchmarks/
│   ├── loading.py             # Full table loading benchmark
│   ├── brin.py                # BRIN-pruned scan benchmark
│   ├── etl_benchmark.py       # ETL → Parquet (postgres + mysql)
│   ├── tpch_multi.py          # TPC-H Q1/Q3/Q5/Q6/Q10/Q12/Q14
│   └── cudf_analytics.py      # cuDF GPU analytics (Q1, Q6)
├── results/
│   ├── REPORT_LOADING.md      # Loading benchmark results + correctness
│   ├── REPORT_BRIN.md         # BRIN pruning results
│   ├── REPORT_ETL_PARQUET.md  # ETL benchmark results (PostgreSQL)
│   ├── REPORT_ETL_MYSQL.md    # ETL benchmark results (MySQL)
│   ├── REPORT_CUDF.md         # cuDF analytics results
│   └── REPORT_TPCH_MULTI.md   # TPC-H multi-query results
├── jailbreak-agentic/         # LLM pipeline to auto-generate C++ readers
├── examples/                  # Usage examples
├── tests/                     # Test suite
├── drivers/                   # JDBC jars (postgresql, mysql-connector, RAPIDS)
├── mysql_arrow.py             # Python ctypes wrapper — MySQL reader
├── spark_pg_reader.py         # Spark integration helper (PostgreSQL)
├── spark_mysql_reader.py      # Partitioned Spark reader (MySQL page ranges)
├── config.py                  # PG connection settings; heap_path/index_path helpers
├── Makefile
└── requirements.txt

---

Environment variables

Variable	Default	Description
TPCH_DSN	dbname=tpch_sf1 host=localhost	psycopg2 connection string
TPCH_JDBC_URL	jdbc:postgresql://localhost/tpch_sf1	JDBC URL for Spark
PGUSER	current OS user	PostgreSQL username
JDBC_JAR	drivers/postgresql-jdbc.jar	Path to JDBC driver jar

---

Platform

Benchmarks run on AWS EC2 g4dn.xlarge — 4 vCPUs, 16 GB RAM, NVIDIA Tesla T4 (16 GB VRAM), EBS gp2 storage. PostgreSQL 16, MySQL 8.0, Python 3.10, cuDF 26.02 (RAPIDS), DuckDB 1.x, DataFusion 46+, PySpark 3.5.