columnar: fix BIGINT clustered PK at INT64_MAX missing on TiFlash read

[JaySon-Huang]

What problem does this PR solve?

Issue Number: close #10852

Problem Summary:

On next-gen / disaggregated columnar read path, queries succeed but omit the row whose clustered BIGINT PK is 9223372036854775807 (i64::MAX). For example, SELECT * FROM t WHERE a > INT64_MIN returns only the 0 row on TiFlash instead of both 0 and INT64_MAX. Full table scan and COUNT(*) also report 2 rows instead of 3.

Root cause is in cloud-storage-engine / kvengine: when building columnar handle index, next_handle + 1 overflows at i64::MAX, so no sentinel is appended and the last pack cannot be read via seek.

What is changed and how it works?

columnar-hub: bump cloud-storage-engine to a9d93252f and add AGENTS.md
columnar-hub: report cloud-storage-engine version at startup

• Bump cloud-storage-engine (kvengine and related crates) to revision a9d93252f, which fixes handle-index sentinel handling when max clustered PK is i64::MAX.
• Add contrib/tiflash-columnar-hub/AGENTS.md documenting how to maintain the columnar-hub dependency.
• Embed kvengine git rev from Cargo.lock in version output and log it at startup for easier on-cluster debugging.

After deploying, rebuild/restart TiFlash and rebuild or re-compact columnar files for affected tables before re-running SQL verification.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Fix next-gen columnar read missing rows when clustered BIGINT primary key is INT64_MAX.

Summary by CodeRabbit

Release Notes

• Documentation

  • Added comprehensive guide for the columnar hub architecture and operational workflows.
  • Added contribution guidelines documenting fork and pull request procedures.

• Improvements

  • Enhanced startup logging to display version information, now including cloud-storage-engine Git hash.
  • Version details are now logged at startup for better operational visibility.

[pantheon-ai]

@JaySon-Huang I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

_{ℹ️ Learn more details on Pantheon AI.}

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces build-time capture of cloud-storage-engine git revision from Cargo.lock, refactors version reporting to use that captured hash, adds startup logging of version details, and documents the columnar hub architecture and dependency management workflow.

Changes

Columnar Hub Dependency Versioning

Layer / File(s)	Summary
Build-time hash extraction from Cargo.lock contrib/tiflash-columnar-hub/hub-runtime/build.rs	Build script reads Cargo.lock to locate the kvengine package source field, extracts git revision hash (from #hash fragment or ?rev= parameter), and emits CLOUD_STORAGE_ENGINE_GIT_HASH for compiler; includes unit tests for both hash formats.
Version info refactored to use captured hash contrib/tiflash-columnar-hub/hub-runtime/src/lib.rs	Made proxy_version_info() pub(crate) and updated output to report CLOUD_STORAGE_ENGINE_GIT_HASH instead of separate git hash/branch fields; removed "Git Commit Branch" line while retaining Rust version and profile fields.
Startup logging of version information contrib/tiflash-columnar-hub/hub-runtime/src/run.rs	Added log_columnar_hub_info() helper that emits welcome message and version details at proxy startup, invoked immediately after logger initialization and before environment/configuration processing.

Documentation

Layer / File(s)	Summary
Columnar hub architecture and dependency management guide contrib/tiflash-columnar-hub/AGENTS.md	Comprehensive guide documenting the Rust FFI proxy (libtiflash_proxy.so), its role in columnar-only next-gen builds, module structure and responsibilities, and step-by-step procedures for managing cloud-storage-engine dependency alignment: pinning revisions, tracking branches, local overrides, rebuilding TiFlash, and integration-test compatibility.
Contribution workflow documentation AGENTS.md	Added "Issues & Pull Requests" section documenting repository issue/PR template locations and standard fork-based contribution workflow.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

The PR involves a new build script with parsing logic and multiple interconnected changes across build, runtime, and runtime initialization code. The hash extraction logic requires careful review of Cargo.lock format parsing and edge cases, while version info integration touches startup flow. Documentation is substantial but straightforward. Mixed complexity across distinct files warrants moderate review effort.

Possibly related PRs

• pingcap/tiflash#10863: Modifies run_proxy startup flow to adjust PD store registration/heartbeat timing relative to engine-store status transition, potentially overlapping with this PR's integration of version logging into the startup sequence.

Suggested reviewers

• yongman
• CalvinNeo
• windtalker

---

🐰 A columnar hub springs to life,
Dependencies pinned through build-script strife,
Hash captured at compile, logged at dawn—
Your versioning troubles are swiftly gone! ✨

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title describes a columnar bug fix for BIGINT PKs at INT64_MAX, but the changeset consists of dependency bump documentation and version logging—changes only tangentially related to the core bug fix.	Update the title to reflect that this PR bumps cloud-storage-engine and adds documentation/logging, not a direct fix. E.g., 'columnar-hub: bump cloud-storage-engine and add version logging' would be more accurate.
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR bumps cloud-storage-engine to a9d93252f (which implements the handle-index sentinel fix for i64::MAX) [#10852], adds AGENTS.md documentation [#10852], and embeds version reporting [#10852]. These directly address the fix requirement and operational needs.
Out of Scope Changes check	✅ Passed	All changes—AGENTS.md documentation, build.rs git hash extraction, lib.rs version output changes, and run.rs startup logging—are scoped to supporting the cloud-storage-engine bump and communicating the kvengine version at runtime.
Description check	✅ Passed	The pull request description follows the required template and includes all essential sections with clear problem statement, solution approach, and testing confirmation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@contrib/tiflash-columnar-hub/hub-runtime/build.rs`:
- Around line 38-41: The current parsing logic in build.rs only checks for
"?rev=" and returns Unknown for branch-tracking entries; update the extraction
logic around source.find("?rev=")/rev_part to also detect "?branch=" and, when a
branch param is present, parse the string after the '#' character (the resolved
commit hash) instead of the branch name; modify the code that computes
rev_part/hash (used where rev_start and rev_part are referenced) to check for
"?branch=" if "?rev=" is absent, split on '#' and return the post-# token
trimmed of quotes as the commit hash.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a482af76-59aa-41ef-a106-d7c8f12bbb3e

📥 Commits

Reviewing files that changed from the base of the PR and between 2e721b3 and 15bb0b3.

⛔ Files ignored due to path filters (1)

• contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (5)

• contrib/tiflash-columnar-hub/AGENTS.md
• contrib/tiflash-columnar-hub/Cargo.toml
• contrib/tiflash-columnar-hub/hub-runtime/build.rs
• contrib/tiflash-columnar-hub/hub-runtime/src/lib.rs
• contrib/tiflash-columnar-hub/hub-runtime/src/run.rs

[ti-chi-bot]

@JaySon-Huang: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-tsan	580dcd3	link	false	/test pull-sanitizer-tsan

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JinheLin, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JinheLin,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-05-26 06:09:53.111137472 +0000 UTC m=+332463.081302535: ☑️ agreed by yongman.
• 2026-05-26 07:00:23.139215337 +0000 UTC m=+335493.109380398: ☑️ agreed by JinheLin.