Skip to content

[python] Add paimon tail CLI for streaming table reads#7351

Draft
tub wants to merge 13 commits intoapache:masterfrom
tub:python-streaming-3-cli-v2
Draft

[python] Add paimon tail CLI for streaming table reads#7351
tub wants to merge 13 commits intoapache:masterfrom
tub:python-streaming-3-cli-v2

Conversation

@tub
Copy link
Contributor

@tub tub commented Mar 5, 2026

Summary

  • Add paimon CLI entry point with tail subcommand
  • paimon tail streams real-time table changes to terminal with formatted output
  • Add CLI utilities for table formatting, column sizing, and output rendering
  • Add CLI documentation to python-api.md
  • Add click dependency to setup.py

Stacked PR series

This is PR 3/5 in the Python streaming read series:

  • PR 1a: Caching infrastructure + utilities
  • PR 1b: Scanners, sharding, row kind
  • PR 1c: Consumer management
  • PR 2: Core streaming (AsyncStreamingTableScan)
  • PR 3 (this): CLI (paimon tail) (~1389 lines)

Incremental diff (vs PR 2): python-streaming-2-core-v2...tub:paimon:python-streaming-3-cli-v2

Test plan

  • flake8 passes on all changed files
  • python -m pytest passes (630/630)
  • New tests: cli_utils_test.py

tub and others added 5 commits March 5, 2026 14:43
@tub @claude
[python] Add caching infrastructure and utilities
f010f45 
- Add backtick quoting to Identifier for SQL-safe formatting
- Add ChangelogProducer enum to core_options
- Add exists_batch() for bulk file existence checks
- Add LRU caching to ManifestFileManager and ManifestListManager
- Add snapshot caching and traversal helpers to SnapshotManager
- Add cachetools dependency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Add scanners, sharding, and row kind support
3498b54 
- Add FollowUpScanner hierarchy (base, delta, changelog)
- Add IncrementalDiffScanner for diff-based streaming reads
- Add sharding support to FileScanner
- Add row kind support to TableRead for changelog streams

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Add consumer management for streaming progress
fa8b9d4 
- Add Consumer dataclass for tracking consumption progress
- Add ConsumerManager for persisting/loading/expiring consumers
- Add unit tests for consumer operations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Add core streaming read with AsyncStreamingTableScan
f64e2f4 
- Add StreamReadBuilder for configuring streaming reads
- Add AsyncStreamingTableScan with consumer registration and sharding
- Add streaming support to FileStoreTable, Table, FormatTable, IcebergTable
- Add acceptance test for incremental diff reads
- Add streaming documentation to python-api.md
- Add oss/lance extras to setup.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Add CLI with paimon tail for streaming reads
9cd45b2 
- Add paimon CLI entry point with tail subcommand
- Add tail command for real-time streaming table consumption
- Add CLI utilities for table formatting and output
- Add CLI documentation to python-api.md
- Add click dependency to setup.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 5, 2026
Open
Closed
Open
tub and others added 8 commits March 5, 2026 18:09
@tub
Making examples more generic
eb63866
@tub @claude
[python] Reduce test boilerplate in manifest_cache_test and fix asser…
a49852f 
…tEquals

Extract shared base class for ManifestFileCacheTest and ManifestListCacheTest,
add _make_snapshot() helper, and fix deprecated assertEquals (removed in 3.12).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Address PR apache#7347 review feedback: use @cachedmethod, t…
9e162b9 
…rim docs, remove ChangelogProducer

- Upgrade cachetools to >=7,<8 for cachedmethod(info=True) support
- Remove ChangelogProducer enum (belongs in apache#7348 scanners branch)
- Replace manual cache hit/miss counters with @cachedmethod(info=True)
  decorator on ManifestFileManager, ManifestListManager, SnapshotManager
- Trim verbose docstrings across identifier, file_io, pyarrow_file_io,
  manifest_list_manager, and snapshot_manager
- Update cache tests to use cache_info() instead of manual counters

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Simplify manifest cache tests: deduplicate via mixin, drop d…
8673c6a 
…efault-size tests

- Move shared cache-behaviour tests (second_read, disabled_when_zero)
  into _CacheBehaviourMixin so they run for both manager types without
  duplication
- Extract _EMPTY_ROW / _EMPTY_STATS module constants to reduce
  DataFileMeta boilerplate
- Remove test_default_cache_size tests (just assert constructor defaults)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Fix trailing blank line in snapshot_manager.py (flake8 W391)
6b7eff1 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Downgrade cachetools to 5.x for Python 3.6 compat
9f4a1da 
cachetools 7.x requires Python >=3.10 but the project supports 3.6+.
Drop info=True and explicit key= from @cachedmethod (both 7.x-only
features) while keeping the decorator itself (available since 4.x).

Replace cache_info()-based test assertions with unittest.mock spies on
file_io.new_input_stream, testing the actual caching effect without any
production code counters.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Address PR apache#7348 review: trim verbose docstrings, cons…
4265498 
…olidate tests, fix parallelism

- Collapse repetitive module/class/method docstrings to one-liners in all
  scanner files (follow_up_scanner, delta, changelog, incremental_diff)
- Remove TDD process commentary from test docstrings
- Consolidate DeltaFollowUpScanner false-case tests into one parameterized test
- Remove misleading commit_kind from ChangelogFollowUpScanner test mocks
- Extract duplicated mock helpers to module-level functions
- Fix max(8, ...) parallelism bug: respect user-configured parallelism
- Remove obvious/redundant inline comments
- Standardize license headers to comment style, merge double docstrings
- Add clarifying docstring to ManifestListManager.read_all

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tub @claude
[python] Trim consumer docstrings, add validation tests, fix path con…
4c31fd7 
…sistency

Align consumer module with the one-liner docstring style used across the rest of the streaming PR stack. Replace os.path.join with f-string path construction for consistency with paimon-python conventions. Add tests for _validate_consumer_id rejection cases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
JingsongLi
JingsongLi reviewed Mar 7, 2026
View reviewed changes
Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you can create a separate PR to enable CLI functionality and perform a simple data read, such as placing the catalog options in a separate yaml configuration file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JingsongLi JingsongLi JingsongLi left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tub @JingsongLi