[improvement](build) Fully exclude OBS/COS deps via --exclude-{obs,cos}-dependencies - #66564
Open
zhiqiang-hhhh wants to merge 1 commit into
Open
[improvement](build) Fully exclude OBS/COS deps via --exclude-{obs,cos}-dependencies#66564zhiqiang-hhhh wants to merge 1 commit into
zhiqiang-hhhh wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
zhiqiang-hhhh
force-pushed
the
feat/disable-huaweicloud
branch
from
August 7, 2026 02:02
1c700d5 to
52d3bc4
Compare
zhiqiang-hhhh
changed the title
zhiqiang-hhhh
marked this pull request as ready for review
August 7, 2026 02:04
zhiqiang-hhhh
requested review from
CalvinKirs,
gavinchou and
morningman
as code owners
August 7, 2026 02:04
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28675 ms |
Contributor
TPC-DS: Total hot run time: 166517 ms |
Contributor
ClickBench: Total hot run time: 23.61 s |
yiguolei
previously approved these changes
Aug 7, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
CalvinKirs
reviewed
Aug 7, 2026
CalvinKirs
left a comment
CalvinKirs
left a comment
Member
There was a problem hiding this comment.
The current implementation does not fully exclude the provider-specific dependencies yet:
-
fe/fe-connector/fe-connector-paimon/pom.xmlstill unconditionally declarescom.huaweicloud:hadoop-huaweicloudand the Huawei Maven repository. Since the Paimon connector is
included in the default FE reactor,--exclude-obs-dependenciesmay still resolve Huawei artifacts. -
fe/be-java-extensions/preload-extensions/pom.xmlstill unconditionally declarescom.qcloud.cos:hadoop-cos.--feenables BE Java extensions and includespreload- extensions, so--exclude-cos-dependenciesmay still resolve Tencent COS artifacts.
Could you move these dependencies, together with the Huawei repository, into the corresponding obs/cos profiles?
Defining the profiles only in fe/pom.xml would not be sufficient because these dependencies and nested modules are declared in their respective child POMs. The module-specific
profiles and the build.sh reactor/module list need to remain consistent.
It would also be good to verify the full dependency tree, rather than only checking active profiles, and confirm that no com.huaweicloud or com.qcloud.cos artifacts are present
when the corresponding exclusion flag is enabled.
…s}-dependencies Turn the existing `--exclude-obs-dependencies` / `--exclude-cos-dependencies` flags from a scope=provided downgrade (which still resolved the jars from their remote repositories, only skipping the bundling step) into a full exclusion: the dependencies are no longer resolved, compiled, or bundled. This is driven by active-by-default Maven profiles (`obs` / `cos`, deactivated via -Ddisable.obs=true / -Ddisable.cos=true) that wrap every touch point: - fe-core: the fe-filesystem-obs / fe-filesystem-cos test couplings. - hadoop-deps: the hadoop-huaweicloud dependency and the Huawei OBS repository. - fe-filesystem: the fe-filesystem-obs / fe-filesystem-cos modules. - fe-connector-paimon: the runtime hadoop-huaweicloud dependency (OBSFileSystem for obs:// warehouses) and the Huawei OBS repository. - preload-extensions: the com.qcloud.cos:hadoop-cos dependency (BE CosN access). build.sh maps the flags to -Ddisable.obs=true / -Ddisable.cos=true and drops the corresponding fe-filesystem provider from both the -pl reactor list and the plugin packaging loop, so the build stays consistent when a provider is excluded. Default builds are unchanged (profiles active unless the flag is passed). Verified with `mvn dependency:tree` across fe-core, fe-connector-paimon, preload-extensions and hadoop-deps: no com.huaweicloud / com.qcloud.cos artifact resolves anywhere in the reactor when the corresponding flag is enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zhiqiang-hhhh
force-pushed
the
feat/disable-huaweicloud
branch
from
August 7, 2026 09:10
52d3bc4 to
a3536a5
Compare
Contributor
Author
|
run buildall |
github-actions
Bot
removed
the
approved
Contributor
TPC-H: Total hot run time: 28767 ms |
Contributor
TPC-DS: Total hot run time: 158754 ms |
Contributor
ClickBench: Total hot run time: 23.72 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Turn the existing
--exclude-obs-dependenciesand--exclude-cos-dependenciesbuild flags into a full exclusion of the respective cloud provider (Huawei OBS / Tencent COS): when passed, nothing from that provider is resolved, compiled, or bundled into the FE binary.Why
The two flags were introduced in #58071 as a
scope=provideddowngrade. That only kept the jars out of the shipped artifact — Maven still resolved and downloaded them at build time. In particularcom.huaweicloud:hadoop-huaweicloud:3.1.1-hw-46is published only on Huawei's own Maven repo (repo.huaweicloud.com). Environments that cannot or must not reach it (corporate proxies blocking the host, or compliance constraints forbidding Huawei/Tencent artifacts) still could not build the FE, because resolution was unconditional. The flags now do what their names imply.How
Active-by-default Maven profiles
obs/cos, deactivated via-Ddisable.obs=true/-Ddisable.cos=true, wrap every touch point:fe-filesystem-obs/fe-filesystem-costest couplings.hadoop-huaweiclouddependency and the Huawei OBS Maven repository.fe-filesystem-obs/fe-filesystem-cosmodules (which transitively pull the Huawei / Tencent SDKs).build.shmaps--exclude-obs-dependencies→-Ddisable.obs=trueand--exclude-cos-dependencies→-Ddisable.cos=true(replacing the old*.dependency.scope=providedmapping), and drops the corresponding provider from both the-plreactor list and the filesystem plugin packaging loop, so the reactor and the dist layout stay consistent when a provider is excluded.Behaviour
sh build.sh --fe --exclude-obs-dependenciesbuilds the FE with no Huawei artifact resolved and no OBS code compiled or bundled; same for--exclude-cos-dependenciesand Tencent COS.Testing
mvn help:active-profilesonfe-core,fe-filesystem, andhadoop-deps:obs/cosactive by default, and absent with-Ddisable.obs=true/-Ddisable.cos=true.compileandtest-compilestay green when the profiles are off.xmllinton all changed poms andbash -n build.shpass.🤖 Generated with Claude Code