feat: add audit_log and binlog system tables

[suxiaogang223]

Summary

• Add table$audit_log and table$binlog on top of the system table framework.
• Route catalog, scan, and read paths through context-aware SystemTable interfaces.
• Output rowkind, optional _SEQUENCE_NUMBER, and batch-mode binlog singleton-list columns.
• Reuse the primary-key table read path with delete records preserved; non-primary-key tables return a clear unsupported status.
• Remove the legacy SystemTableRead fallback so system tables use the same scan/read context path.

Part of #141.

Scope

• This PR focuses on batch read correctness for audit_log and binlog.
• Streaming before/after pairing and full predicate pushdown are intentionally left for follow-up work.

Validation

• Fedora: cmake --build build-codex-pr2 --target paimon-core-test paimon-read-inte-test -j4
• Fedora: ./build-codex-pr2/release/paimon-core-test --gtest_filter="FileSystemCatalogTest.TestAuditLogAndBinlogSystemTableCatalog:CoreOptionsTest.*"
• Fedora: ./build-codex-pr2/release/paimon-read-inte-test --gtest_filter="SystemTableReadInteTest.*"

[suxiaogang223]

Hi @lxy-9602 , could you please help review this PR when you have time?

This PR adds the next batch of system table support on top of the merged system table framework: table$audit_log and table$binlog. It includes catalog routing/schema exposure, scan/read integration, rowkind output, optional _SEQUENCE_NUMBER, and related tests for catalog and read paths.

The next PR will continue this work by adding the remaining metadata system tables. :)

[lxy-9602]

Thanks for the submission! It's great to see not just new features, but also thoughtful cleanups and small refactorings you've made🚀.
Looking forward to your adjustments and continued contributions!

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds new audit_log and binlog system tables and refactors system-table scan/read to use the same context-aware scan/read pipeline as data tables.

Changes:

• Introduce AuditLogSystemTable and BinlogSystemTable with changelog-oriented schemas (rowkind, optional sequence number, and binlog list-wrapping).
• Route TableScan::Create / TableRead::Create through SystemTable::NewScan(NewRead) using ScanContext / ReadContext and remove the legacy SystemTableRead wrapper.
• Extend KV read path to optionally preserve deletes and to project _SEQUENCE_NUMBER from KeyValue metadata.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/inte/read_inte_test.cpp	Adds integration tests/fixtures for reading audit_log / binlog system tables (including branch + sequence number).
src/paimon/core/utils/special_field_ids.cpp	Adjusts SpecialFieldIds definitions to match the updated header initialization strategy.
src/paimon/core/table/system/system_table_read.h	Replaces legacy SystemTableRead header content (now declares BinlogSystemTable).
src/paimon/core/table/system/system_table_read.cpp	Removes legacy SystemTableRead implementation.
src/paimon/core/table/system/system_table.h	Updates SystemTable interface to context-aware NewScan/NewRead; loader now accepts dynamic options.
src/paimon/core/table/system/system_table.cpp	Registers/loads audit_log and binlog; merges dynamic options (including branch).
src/paimon/core/table/system/options_system_table.h	Updates options system table to new NewScan/NewRead interface.
src/paimon/core/table/system/options_system_table.cpp	Implements OptionsTableRead to fit the new read pipeline.
src/paimon/core/table/system/binlog_system_table.cpp	Implements binlog schema/read via list-wrapping converter on top of audit_log.
src/paimon/core/table/system/audit_log_system_table.h	Adds audit_log system table + changelog conversion interfaces.
src/paimon/core/table/system/audit_log_system_table.cpp	Implements audit_log scan/read using KV read path with delete preservation and rowkind derivation.
src/paimon/core/table/source/table_scan.cpp	Routes system-table scans through SystemTable::NewScan(context) and makes scans branch-aware.
src/paimon/core/table/source/table_read.cpp	Routes system-table reads through SystemTable::NewRead(context); refactors data-table read creation.
src/paimon/core/table/source/key_value_table_read.h	Exposes ForceKeepDelete to preserve delete records when required (system tables).
src/paimon/core/table/source/key_value_table_read.cpp	Implements ForceKeepDelete propagation into split reads.
src/paimon/core/operation/merge_file_split_read.h	Adds runtime flag to keep/delete delete records.
src/paimon/core/operation/merge_file_split_read.cpp	Adds KeyValue metadata sequence-number projection and uses drop_delete = !force_keep_delete_.
src/paimon/core/operation/internal_read_context.cpp	Allows _SEQUENCE_NUMBER in read schema when KV sequence-number option is enabled.
src/paimon/core/io/key_value_projection_consumer.h	Introduces sentinel projection id for sequence-number-from-metadata.
src/paimon/core/io/key_value_projection_consumer.cpp	Implements sequence-number injection into the projected Arrow batch.
src/paimon/core/core_options_test.cpp	Adds unit coverage for new sequence-number options defaults and parsing.
src/paimon/core/core_options.h	Adds getters for table-read and key-value sequence-number switches.
src/paimon/core/core_options.cpp	Parses and stores new sequence-number option flags.
src/paimon/core/catalog/file_system_catalog_test.cpp	Adds catalog tests for audit_log and binlog system tables.
src/paimon/core/catalog/file_system_catalog.cpp	Passes branch as dynamic option when loading system table schemas.
src/paimon/common/defs.cpp	Defines new option keys for sequence-number switches.
src/paimon/CMakeLists.txt	Builds new system table sources; removes legacy system_table_read.cpp.
include/paimon/utils/special_field_ids.h	Moves special field ids to in-class initialized constants.
include/paimon/defs.h	Documents newly added sequence-number option keys.
cmake_modules/SetupCxxFlags.cmake	Removes -Wno-deprecated-builtins suppression at CHECKIN warning level.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lxy-9602]

+1

[lxy-9602]

+1