Skip to content

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Dec 11, 2025


When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance. To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list.

Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues:

The auto-loaded default HDFS entry could appear after other storage providers.

Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters.

Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs.

Previously, the catalog used:

catalogProperty.getStoragePropertiesMap().values()

to obtain the list of StorageProperties.
However, getStoragePropertiesMap()
stores the properties in a Map, and Map iteration order is not guaranteed.
This caused the resolved storage configurations to be non-deterministic.

Problem

Some Hadoop/S3 parameters—such as:
fs.s3a.aws.credentials.provider

have default values.
When multiple StorageProperties instances were merged in unpredictable order, these defaults could override user-configured settings.

This created actual production issues:

When using IAM Role (Assume Role), the correct provider must be AwsAssumedRoleCredentialProvider.

Because the list order was unstable, sometimes the default provider overrode the intended Assume-Role provider.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…ading default HDFS

When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance.
To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list.

Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues:

The auto-loaded default HDFS entry could appear after other storage providers.

Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters.

Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs CalvinKirs changed the title [Fix](Hdfs)Ensure StorageProperties list remains ordered when auto-loading default HDFS [Fix](Iam-role)Ensure StorageProperties list remains ordered when auto-loading default HDFS Dec 11, 2025
@CalvinKirs
Copy link
Member Author

run buildall

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 90.91% (10/11) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (11/11) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@CalvinKirs CalvinKirs merged commit 992a5d1 into apache:master Dec 16, 2025
32 of 33 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 16, 2025
…o-loading default HDFS (#58968)

…
When no explicit HdfsProperties is configured, the system automatically
adds a default HdfsProperties instance. To prevent configuration
conflicts and parameter overrides, it is essential that this default
entry is always inserted at the first position of the storage list.

Previously, the ordering of the StorageProperties list was not
explicitly guaranteed, which could lead to the following issues:

The auto-loaded default HDFS entry could appear after other storage
providers.

Subsequent initialization steps (initNormalizeAndCheckProps,
buildHadoopStorageConfig) might unintentionally override user-defined
parameters.

Behavior of storage resolution became dependent on iteration order,
causing subtle configuration bugs.


Previously, the catalog used:


`catalogProperty.getStoragePropertiesMap().values()`


to obtain the list of StorageProperties.
However, `getStoragePropertiesMap()`
stores the properties in a Map, and Map iteration order is not
guaranteed.
This caused the resolved storage configurations to be non-deterministic.

Problem

Some Hadoop/S3 parameters—such as:
`
fs.s3a.aws.credentials.provider`


have default values.
When multiple StorageProperties instances were merged in unpredictable
order, these defaults could override user-configured settings.

This created actual production issues:

When using IAM Role (Assume Role), the correct provider must be
AwsAssumedRoleCredentialProvider.

Because the list order was unstable, sometimes the default provider
overrode the intended Assume-Role provider.
### What problem does this PR solve?

Issue Number: close #xxx
yiguolei pushed a commit that referenced this pull request Dec 17, 2025
…red when auto-loading default HDFS #58968 (#59072)

Cherry-picked from #58968

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
morningman pushed a commit that referenced this pull request Dec 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants