Skip to content

columnar: fix CD compile error#10887

Merged
ti-chi-bot[bot] merged 8 commits into
pingcap:masterfrom
JaySon-Huang:fix_release_compile
Jun 4, 2026
Merged

columnar: fix CD compile error#10887
ti-chi-bot[bot] merged 8 commits into
pingcap:masterfrom
JaySon-Huang:fix_release_compile

Conversation

@JaySon-Huang
Copy link
Copy Markdown
Contributor

@JaySon-Huang JaySon-Huang commented Jun 3, 2026

What problem does this PR solve?

Issue Number: ref #10844

Problem Summary:

What is changed and how it works?

Currently the pingcap internal CD pipeline does not support checkout private repo at compile time. So ‎contrib/tiflash-columnar-hub compile fails.
Add cloud-storage-engine as a submodule so that the private repo can be checkouted at submodule init.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

  • Chores
    • Simplified and unified external dependency declarations in manifests to streamline resolution and local overrides.
    • Added and updated a cloud-storage submodule pointer for the cloud storage integration.
    • Reordered build/install steps and consolidated build configuration in release build scripts for more predictable builds.
  • Style
    • Minor import cleanup in profiling code for consistency.

Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 3, 2026
@pantheon-ai
Copy link
Copy Markdown

pantheon-ai Bot commented Jun 3, 2026

@JaySon-Huang I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

ℹ️ Learn more details on Pantheon AI.

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 3, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jun 3, 2026

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Convert kvproto and tipb git sources from SSH to HTTPS and centralize their crate patches in the root [patch.crates-io]; update hub-runtime Cargo.toml to match; remove a pinned pprof patch; move a Pin import in profile.rs; relocate the non-columnar TiFlash build/install block and remove a global git URL rewrite; add/update cloud-storage-engine submodule and enable local path patches.

Changes

Dependency & Build Script Updates

Layer / File(s) Summary
Root patch configuration restructuring
contrib/tiflash-columnar-hub/Cargo.toml
Adds kvproto and tipb under [patch.crates-io] as HTTPS git overrides, removes a pinned pprof patch entry, and enables multiple tidbcloud/cloud-storage-engine local path = ../... overrides.
Runtime dependency source updates
contrib/tiflash-columnar-hub/hub-runtime/Cargo.toml
Switches kvproto and tipb git dependency URLs from SSH to HTTPS and removes pinned rev commits.
Profile import adjustment
contrib/tiflash-columnar-hub/hub-runtime/src/profile.rs
Moves Pin to a dedicated use std::pin::Pin; import and removes it from the grouped use std::{ ... } under the jemalloc cfg.
Release script build/install relocation
release-linux-llvm/scripts/build-tiflash-release.sh
Moves computation of ENABLE_FAILPOINTS/JEMALLOC_NARENAS earlier, removes the global git URL rewrite, and relocates the non-columnar TiFlash build/install block to run after the columnar branch, reusing envs and printing diagnostics.
Add/update cloud-storage-engine submodule and pointer
.gitmodules, contrib/cloud-storage-engine
Adds a .gitmodules entry for contrib/cloud-storage-engine pointing to the cloud-engine branch and advances the submodule record to commit a9d93252f2ad0cba95eec51a857cd867cd5e6567.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • pingcap/tiflash#10878: Related edits to the TiFlash release build script flow and columnar build handling.
  • pingcap/tiflash#10869: Related changes around contrib/cloud-storage-engine revision/patching used by tiflash-columnar-hub.

Suggested labels

size/L

Suggested reviewers

  • CalvinNeo
  • JinheLin
  • yongman
  • wuhuizuo
  • Lloyd-Pottiger

Poem

🐰 I hopped from ssh to https bright,
crates now fetched without the old bite.
A tidy import, builds rearranged,
submodule added, paths unchained.
Nibble, compile, and sleep tonight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is missing critical information: the 'Problem Summary' and 'What is changed and how it works?' sections are empty, and the commit-message block lacks detail. Complete the 'Problem Summary' section with a clear explanation of the CD pipeline issue, and fill the 'What is changed and how it works?' section with detailed commit message content explaining the submodule addition and related Cargo.toml changes.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title 'columnar: fix CD compile error' directly addresses the PR's stated intent to fix CD compile issues. It is concise, specific, and clearly summarizes the main purpose of the changeset.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@JaySon-Huang

This comment was marked as resolved.

Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Copy link
Copy Markdown
Contributor Author

/test ?

@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Jun 3, 2026

@JaySon-Huang: The following commands are available to trigger required jobs:

/test pull-integration-next-gen
/test pull-integration-test
/test pull-unit-next-gen
/test pull-unit-test

The following commands are available to trigger optional jobs:

/test pull-error-log-review
/test pull-integration-next-gen-columnar
/test pull-sanitizer-asan
/test pull-sanitizer-tsan

Use /test all to run the following jobs that were automatically triggered:

pingcap/tiflash/pull_integration_next_gen
pingcap/tiflash/pull_integration_test
pingcap/tiflash/pull_unit_next_gen
pingcap/tiflash/pull_unit_test
pull-sanitizer-asan
pull-sanitizer-tsan
Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@JaySon-Huang
Copy link
Copy Markdown
Contributor Author

/test pull-integration-next-gen-columnar

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@contrib/tiflash-columnar-hub/hub-runtime/Cargo.toml`:
- Line 32: The git dependencies for kvproto and tipb are currently floating;
update the Cargo.toml entries that declare kvproto = { git =
"https://github.com/pingcap/kvproto.git" } and tipb = { git =
"https://github.com/pingcap/tipb.git" } to include pinned rev fields from the
Cargo.lock: set kvproto rev = "1e6ff1c80df24a9ead2b4408133af23c09273718" and
tipb rev = "a4d204a193b4f9aa776343ac75281cf8442343ba" so the protobuf-generated
APIs used by run.rs and columnar_impls.rs remain stable; update each Cargo.toml
where those git deps are declared to include the corresponding rev string.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d51e9a5e-d92d-4317-a91c-fbcff7e743d5

📥 Commits

Reviewing files that changed from the base of the PR and between 252c474 and b85b654.

⛔ Files ignored due to path filters (1)
  • contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • contrib/tiflash-columnar-hub/Cargo.toml
  • contrib/tiflash-columnar-hub/hub-runtime/Cargo.toml
  • contrib/tiflash-columnar-hub/hub-runtime/src/profile.rs
✅ Files skipped from review due to trivial changes (1)
  • contrib/tiflash-columnar-hub/hub-runtime/src/profile.rs

Comment thread contrib/tiflash-columnar-hub/hub-runtime/Cargo.toml
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
release-linux-llvm/scripts/build-tiflash-release.sh (1)

113-120: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not unset LD_LIBRARY_PATH for the remainder of the script.

Line 114 now runs before the non-columnar configure/build block, so the relocated flow strips LD_LIBRARY_PATH from the later cmake and cmake --build steps as well. That changes the build environment, not just the post-install verification.

Suggested fix
-  unset LD_LIBRARY_PATH
-  readelf -d "${INSTALL_DIR}/tiflash"
-  ldd "${INSTALL_DIR}/tiflash"
+  env -u LD_LIBRARY_PATH readelf -d "${INSTALL_DIR}/tiflash"
+  env -u LD_LIBRARY_PATH ldd "${INSTALL_DIR}/tiflash"
   
   # show version
-  ${INSTALL_DIR}/tiflash version
+  env -u LD_LIBRARY_PATH "${INSTALL_DIR}/tiflash" version
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@release-linux-llvm/scripts/build-tiflash-release.sh` around lines 113 - 120,
The script currently unsets LD_LIBRARY_PATH globally (unset LD_LIBRARY_PATH)
before running readelf/ldd/${INSTALL_DIR}/tiflash version which removes needed
library paths for subsequent cmake and build steps; instead limit this change to
the verification step by either (a) running readelf/ldd and
${INSTALL_DIR}/tiflash version inside a subshell where LD_LIBRARY_PATH is unset,
or (b) save the original LD_LIBRARY_PATH into a temp var and restore it
immediately after the readelf/ldd/version checks so that later cmake and cmake
--build steps see the original LD_LIBRARY_PATH; locate the unset invocation and
the surrounding verification commands (readelf, ldd, ${INSTALL_DIR}/tiflash
version) and implement one of these scoped approaches to avoid wiping
LD_LIBRARY_PATH for the remainder of the script.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@release-linux-llvm/scripts/build-tiflash-release.sh`:
- Around line 63-69: Change the unconditional else branch to apply the
failpoint/jemalloc overrides only for sanitizer builds: keep the RELWITHDEBINFO
branch setting ENABLE_FAILPOINTS="OFF" and JEMALLOC_NARENAS="-1", add an elif
that checks if CMAKE_BUILD_TYPE equals "ASAN" or "TSAN" and only in that case
set ENABLE_FAILPOINTS="ON" and JEMALLOC_NARENAS="40"; for other build types
leave ENABLE_FAILPOINTS and JEMALLOC_NARENAS unset so CMake defaults
(ENABLE_TESTS/jemalloc defaults) remain in effect.

---

Outside diff comments:
In `@release-linux-llvm/scripts/build-tiflash-release.sh`:
- Around line 113-120: The script currently unsets LD_LIBRARY_PATH globally
(unset LD_LIBRARY_PATH) before running readelf/ldd/${INSTALL_DIR}/tiflash
version which removes needed library paths for subsequent cmake and build steps;
instead limit this change to the verification step by either (a) running
readelf/ldd and ${INSTALL_DIR}/tiflash version inside a subshell where
LD_LIBRARY_PATH is unset, or (b) save the original LD_LIBRARY_PATH into a temp
var and restore it immediately after the readelf/ldd/version checks so that
later cmake and cmake --build steps see the original LD_LIBRARY_PATH; locate the
unset invocation and the surrounding verification commands (readelf, ldd,
${INSTALL_DIR}/tiflash version) and implement one of these scoped approaches to
avoid wiping LD_LIBRARY_PATH for the remainder of the script.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 7911bb3e-1cba-4480-bd0a-8a0debf3bc57

📥 Commits

Reviewing files that changed from the base of the PR and between 6256bea and 71f6d22.

⛔ Files ignored due to path filters (1)
  • contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • contrib/tiflash-columnar-hub/Cargo.toml
  • contrib/tiflash-columnar-hub/hub-runtime/Cargo.toml
  • release-linux-llvm/scripts/build-tiflash-release.sh

Comment thread release-linux-llvm/scripts/build-tiflash-release.sh
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 4, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
contrib/tiflash-columnar-hub/Cargo.toml (1)

27-28: ⚡ Quick win

Reproducibility for kvproto/tipb git patches (Cargo.toml): contrib/tiflash-columnar-hub/Cargo.toml points kvproto/tipb at git without rev/tag, but contrib/tiflash-columnar-hub/Cargo.lock already pins them (kvproto1e6ff1c80df24a9ead2b4408133af23c09273718, tipba4d204a193b4f9aa776343ac75281cf8442343ba), and the Makefile builds with cargo build --locked, so builds stay reproducible as long as Cargo.lock is kept in sync when changing these deps. Optional: consider adding rev in Cargo.toml for clarity.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@contrib/tiflash-columnar-hub/Cargo.toml` around lines 27 - 28, The Cargo.toml
currently specifies kvproto and tipb as git dependencies without pinned revs
which can be unclear; ensure reproducible builds by either (A) keeping
Cargo.lock updated whenever those git dependencies change (verify
contrib/tiflash-columnar-hub/Cargo.lock contains the commits
kvproto=1e6ff1c80df24a9ead2b4408133af23c09273718 and
tipb=a4d204a193b4f9aa776343ac75281cf8442343ba and maintain that lock in the repo
because the Makefile invokes cargo build --locked), or (B) explicitly add rev
entries to the kvproto and tipb git specifications in
contrib/tiflash-columnar-hub/Cargo.toml to pin those commits for clarity and
reproducibility; pick one approach and update either Cargo.lock or Cargo.toml
accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@contrib/tiflash-columnar-hub/Cargo.toml`:
- Around line 27-28: The Cargo.toml currently specifies kvproto and tipb as git
dependencies without pinned revs which can be unclear; ensure reproducible
builds by either (A) keeping Cargo.lock updated whenever those git dependencies
change (verify contrib/tiflash-columnar-hub/Cargo.lock contains the commits
kvproto=1e6ff1c80df24a9ead2b4408133af23c09273718 and
tipb=a4d204a193b4f9aa776343ac75281cf8442343ba and maintain that lock in the repo
because the Makefile invokes cargo build --locked), or (B) explicitly add rev
entries to the kvproto and tipb git specifications in
contrib/tiflash-columnar-hub/Cargo.toml to pin those commits for clarity and
reproducibility; pick one approach and update either Cargo.lock or Cargo.toml
accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 9b9b9c5d-3346-4f9b-a754-f600b2511adc

📥 Commits

Reviewing files that changed from the base of the PR and between 71f6d22 and 17576cb.

⛔ Files ignored due to path filters (1)
  • contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • .gitmodules
  • contrib/cloud-storage-engine
  • contrib/tiflash-columnar-hub/Cargo.toml
✅ Files skipped from review due to trivial changes (1)
  • .gitmodules

@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Jun 4, 2026

@JaySon-Huang: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-integration-next-gen-columnar b85b654 link false /test pull-integration-next-gen-columnar
pull-sanitizer-tsan 17576cb link false /test pull-sanitizer-tsan
pull-sanitizer-asan 17576cb link false /test pull-sanitizer-asan

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@JaySon-Huang JaySon-Huang changed the title [WIP] fix CD compile columnar: fix CD compile error Jun 4, 2026
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 4, 2026
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jun 4, 2026
@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Jun 4, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Lloyd-Pottiger, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [Lloyd-Pottiger,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 4, 2026
@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Jun 4, 2026

[LGTM Timeline notifier]

Timeline:

  • 2026-06-04 10:33:08.508103103 +0000 UTC m=+437689.578420532: ☑️ agreed by Lloyd-Pottiger.
  • 2026-06-04 10:40:01.048927354 +0000 UTC m=+438102.119244734: ☑️ agreed by yongman.

@ti-chi-bot ti-chi-bot Bot merged commit 3e8c8ec into pingcap:master Jun 4, 2026
8 of 10 checks passed
@JaySon-Huang JaySon-Huang deleted the fix_release_compile branch June 4, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants