branch-4.1: [opt](jindofs) update jindofs to 6.10.4 and add DlfFileIO (#60856) [feat](iceberg) support aliyun dlf iceberg rest catalog (#60796)#61341
Closed
morningman wants to merge 16 commits intoapache:branch-4.1from
Conversation
…hrough) add ForcePassthrough SessionVariable" (apache#61275) ### What problem does this PR solve? apache#59366 apache#60671 Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Pxl <xl@selectdb.com>
…#59446 apache#58728 (apache#61282) ### What problem does this PR solve? cherry pick the opt code apache#60137 apache#59492 apache#59446 apache#58728 Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
… key distribution apache#61248 (apache#61297) picked from apache#61248
…eAggregation (apache#59116) (apache#61301) picked from apache#59116 apache#60045 apache#59876 apache#60091 apache#60811 --------- Co-authored-by: yujun <yujun@selectdb.com>
…ewrite simple agg to constant (apache#61306) ### What problem does this PR solve? pick#61183 Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…pache#61303) pick changes from PR apache#61104 and PR apache#60941
…mLiteralRewrite (apache#61224) (apache#61308) pick apache#61224 ### What problem does this PR solve? SumLiteralRewrite transforms SUM(expr +/- literal) into SUM(expr) +/- literal * COUNT(expr). When type coercion has introduced an implicit widening cast (e.g. CAST(smallint_col AS INT)), the rewritten SUM/COUNT still operates on the wider type, forcing unnecessary wider data reads. This is redundant because SUM always returns BIGINT for any integer input (TINYINT/SMALLINT/INT/BIGINT). Strip implicit widening integer casts in extractSumLiteral() so the aggregate operates on the original narrow column directly. This benefits ClickBench Q29-style queries where SUM(col), SUM(col+1), SUM(col+2) share a narrow integer column — after stripping the cast, SUM(col+1) and SUM(col+2) reuse the existing SUM(col). ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…61321) pick apache#59591 Co-authored-by: starocean999 <lichi@selectdb.com>
…#59572) (apache#61309) ### What problem does this PR solve? pick apache#59572 Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…e version for MUST_NOT to handle NULL rows apache#61200 (apache#61323) Cherry-pick of apache#61200 to branch-4.1. ### What problem does this PR solve? Issue Number: close #DORIS-24681 `search('NOT msg:omega')` incorrectly includes NULL rows in the result set, while `NOT search('msg:omega')` correctly excludes them. **Root cause:** `ExcludeScorer` does not implement `has_null_bitmap()` / `get_null_bitmap()`, inheriting the `Scorer` base class defaults that always return `false` / `nullptr`. This caused NULL documents to be treated as TRUE (matching) rather than NULL, violating SQL three-valued logic where `NOT(NULL) = NULL`. **Fix:** Enhance `ExcludeScorer` with null bitmap support: - Accept a pre-collected `exclude_null` bitmap (read from index at O(1) cost) - Check it during `advance()`/`seek()` before the original lazy `is_within()` exclusion - Implement `has_null_bitmap()`/`get_null_bitmap()` to propagate null info - Plumb `binding_keys` from `function_search.cpp` through OccurBooleanQueryBuilder → OccurBooleanQuery → OccurBooleanWeight for proper null bitmap resolution ### Release note Fix search('NOT field:value') incorrectly including NULL rows by using null-bitmap-aware ExcludeScorer. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
depends on apache#60856 Support Aliyun DLF (Data Lake Formation) as an Iceberg REST catalog. Main changes: - Preserve case-sensitive REST signing names by removing toLowerCase()on `rest.signing-name`, so names like "DlfNext" work correctly. - Prioritize non-S3 storage properties when multiple S3-compatible properties are returned from iceberg rest catalog, to avoid incorrectly using S3 credentials for non-S3 storage (e.g. OSS). - Remove redundant resetToUninitialized() override in HMSExternalCatalog that only delegates to super. - Add IcebergDlfRestCatalogTest, IcebergRestPropertiesTest and StoragePropertiesTest for DLF REST catalog integration and property resolution testing. ``` CREATE CATALOG ice PROPERTIES ( 'type' = 'iceberg', 'warehouse' = 'new_dlf_iceberg_catalog', 'iceberg.catalog.type' = 'rest', 'iceberg.rest.uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com/iceberg', 'iceberg.rest.sigv4-enabled' = 'true', 'iceberg.rest.signing-name' = 'DlfNext', 'iceberg.rest.access-key-id' = 'xx', 'iceberg.rest.secret-access-key' = 'xx', 'iceberg.rest.signing-region' = 'cn-beijing', 'iceberg.rest.vended-credentials-enabled' = 'true', 'io-impl' = 'org.apache.iceberg.rest.DlfFileIO', 'fs.oss.support' = 'true' ); ```
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #60856 and #60796 to branch-4.1