v0.1.0

What's Changed

• Bump rows_emitted only after EndOfStream ack by @heavycrystal in #1
• Walsender: emit idle keepalives; drop redundant per-push 'k' by @heavycrystal in #5
• Atomic stats counters; wire Prometheus emitter metrics by @heavycrystal in #4
• Add simple benchmark by @harshil-goel in #3
• implement pinned_ddl_baseline.md by @serprex in #6
• wal_page.rs by @serprex in #7
• optimize wal rewrite to use less copies by @serprex in #8
• raise coverage by @serprex in #9
• clickhouse-c-rs async by @serprex in #10
• Add a way to bring up instances on ec2, benchmark them, and get profiles by @harshil-goel in #11
• Fix dockerfile to contain bench by @harshil-goel in #14
• _is_deleted by @serprex in #13
• remove REPLICA IDENTITY FULL requirement by @serprex in #17
• TOAST by @serprex in #18
• Fix LSN format for start replication by @harshil-goel in #20
• use walross format_pg_lsn more by @serprex in #22
• Add fatal error msging by @harshil-goel in #25
• Fix shadow pg wal replay by @harshil-goel in #26
• replace pgwalrs submodule with crates.io by @serprex in #27
• Add Traces to walshadow by @harshil-goel in #12
• wal-rus 0.2 by @serprex in #32
• Add tests for types, composoite pkey, tables names, ddl by @harshil-goel in #31
• Reduce memory usage in decode pool by @harshil-goel in #33
• Remove fsync from hot path by @harshil-goel in #34
• Remove catalog.lock().await inside the hot path by @harshil-goel in #43
• clickhouse-c-rs: support global allocator, use with mimalloc by @serprex in #46
• replace mpmc with async_channel by @serprex in #47
• Fix leaking memory on wire_buf by @harshil-goel in #48
• Re-establish clickhouse connection if it's idle for too long by @harshil-goel in #49
• Fix epoch race by @serprex in #51
• port bench/ to terraform by @serprex in #52
• ConfigResolver, SIGHUP handling, add table by @serprex in #45
• Fix ec2 stack after config changes by @harshil-goel in #54
• Fix ReplIdent::Full table to have primary key in the LSN OrderBy by @harshil-goel in #53
• Bound queuing thread with hard limit instead of soft by @harshil-goel in #50
• Stream commit drain by @serprex in #56
• fix new lint warnings with rust 1.97 by @serprex in #59
• clickhouse-c-rs: improve safety by @serprex in #58
• Avoid advancing lsn past minimum in-flight transaction's lsn by @serprex in #60
• update clickhouse-c by @serprex in #61
• Fix table creation and backfill with TOML by @harshil-goel in #57
• AGPLv3 by @serprex in #62
• TOAST v2 by @serprex in #55
• Encapsulation by @serprex in #63
• update dependencies by @serprex in #68
• Added tests for auto create table in clickhouse by @harshil-goel in #64
• docker/: fix config with new table syntax by @serprex in #69
• register SIGHUP handler right away, be fatal if it fails by @serprex in #70
• Manage shadow catalog more directly by @serprex in #71
• Refactor restart state by @serprex in #72
• Control walshadow using unix sockets by @harshil-goel in #65
• At walshadow restart, recover from last state or s3 by @harshil-goel in #44
• Make catalog decoding consistent across DDL & restarts by @serprex in #73
• Use pg background worker, get rid of CREATE EXTENSION for oracle by @serprex in #74
• Track database id of wal, avoid processing for non-target database by @serprex in #77
• use cargo nextest to parallelize CI by @serprex in #78
• Ignore zeroed pages during page walk by @serprex in #79
• use ahash, 4x faster lookup, 2x faster insert by @serprex in #80
• Optimize vacuum/analyze by @serprex in #81
• Fix restore command for shadow pg by @harshil-goel in #86
• Update defaults and tuning knobs and other configuration bugs by @harshil-goel in #84
• Switchover mechanism, for use when primary itself is doing switchover by @serprex in #83
• Spool object-store parts to scratch before decoding by @harshil-goel in #88
• Make WAL position and progress tracking safer by @serprex in #91
• Updated default configurations by @harshil-goel in #90
• fix CI after bad merge by @serprex in #94
• Add name matching capabilities by @serprex in #95
• Fixed stashed toast heap record crash during ddl + toast transaction by @harshil-goel in #98
• quickstart by @serprex in #96
• Replace CopyBinaryParser with tokio_postgres::binary_copy::BinaryCopyOutStream by @serprex in #99
• use serde for toml by @serprex in #100
• clickhouse-c-rs moved to its own crate by @serprex in #101
• Docs by @serprex in #102
• Moved creating tables in clickhouse over pg incomplete marker by @harshil-goel in #103
• Configure on CH meta names, ORDER BY, PRIMARY KEY by @serprex in #93
• Support more native datatypes in Walshadow by @harshil-goel in #92
• Add pg oracle during bootstrap to decode complex types by @harshil-goel in #108
• Honor initial mode config with direct bootstrap by @harshil-goel in #112
• Remove toast config by @harshil-goel in #113
• update astral-tokio-tar, wal-rus, & clickhouse-c-rs by @serprex in #117
• oracle: respond with native blocks by @serprex in #111
• fix readme saying walshadow doesn't create databases by @serprex in #118
• consolidate line coverage from PG16/PG17/PG18 ci by @serprex in #119
• avoid connection failure to avoid 5s wait, raise max_wal_senders in CI by @serprex in #122
• Split quickstart and getting started by @serprex in #127
• Deprecate shadow pg dbname by @harshil-goel in #116
• Bootstrap consistency by @serprex in #125
• Filter extensions during bootstrap by @harshil-goel in #128
• add benchmark for initial load by @serprex in #130
• Remove XLOG_FPI and XLOG_FPI_FOR_HINT from the logs sent to shadow pg by @harshil-goel in #131
• fix ci flake by @serprex in #137
• reduce plans/ by @serprex in #136
• more coverage by @serprex in #138
• more backon by @serprex in #139
• Update status to contain hostname and tables by @harshil-goel in #135
• Fix deadlock when ddl gets triggered during autovacuum by @harshil-goel in #126
• add github action to publish ghcr.io images by @serprex in #140
• Optimizations by @serprex in #121
• Fixed crash due to different history in object store by @harshil-goel in #141
• handle timeline switch while replaying WAL by @serprex in #143
• Fix pool size defaults by @harshil-goel in #142
• update dependencies, particularly wal-rus 0.3.3 by @serprex in #144
• Fix compressed-external TOAST detoasting by @harshil-goel in #145
• Fix ADD COLUMN DEFAULT with tier 3 types by @serprex in #147
• use prometheus-client by @serprex in #148
• Implement JSONB as tier 2 type by @serprex in #153
• Resolve backup rows without COPY repair by @serprex in #151
• map json/jsonb to String, not JSON by @serprex in #154
• Added metrics during backfil of object storage by @harshil-goel in #149
• Restart after interuppupted object store bootstrap by @harshil-goel in #155
• Use AsciiBuf for formatting time/uuid outside decoder. Remove chrono by @serprex in #156
• use roaring bitmap for PgXactPatch by @serprex in #158
• Batch deferred bootstrap TOAST fetches, 50x faster by @serprex in #157
• Added more stages in resume by @harshil-goel in #159
• PG19 by @serprex in #150
• Batch TOAST store reads and writes during WAL replay by @serprex in #161
• update clickhouse-c-rs & pg-clickhouse-c by @serprex in #160
• Optimize bloom filter settings by @harshil-goel in #162
• Add a mode to let toast stay in shadow pg by @harshil-goel in #163
• Structure ctl status tables by @harshil-goel in #164
• Publish linux x86_64/aarch64 release binaries, add --version by @iskakaushik in #167
• Replicate multiple PostgreSQL databases from one WAL stream by @serprex in #165
• Harden WAL recovery, catalog tracking and backfill consistency by @serprex in #166

New Contributors

• @heavycrystal made their first contribution in #1
• @serprex made their first contribution in #6
• @iskakaushik made their first contribution in #167

Full Changelog: https://github.com/ClickHouse/walshadow/commits/v0.1.0