-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[Fix](Iam-role)Ensure StorageProperties list remains ordered when auto-loading default HDFS #58968
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ading default HDFS When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance. To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list. Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues: The auto-loaded default HDFS entry could appear after other storage providers. Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters. Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
…when auto-loading default HDFS apache#58968
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…o-loading default HDFS (#58968) … When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance. To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list. Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues: The auto-loaded default HDFS entry could appear after other storage providers. Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters. Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs. Previously, the catalog used: `catalogProperty.getStoragePropertiesMap().values()` to obtain the list of StorageProperties. However, `getStoragePropertiesMap()` stores the properties in a Map, and Map iteration order is not guaranteed. This caused the resolved storage configurations to be non-deterministic. Problem Some Hadoop/S3 parameters—such as: ` fs.s3a.aws.credentials.provider` have default values. When multiple StorageProperties instances were merged in unpredictable order, these defaults could override user-configured settings. This created actual production issues: When using IAM Role (Assume Role), the correct provider must be AwsAssumedRoleCredentialProvider. Because the list order was unstable, sometimes the default provider overrode the intended Assume-Role provider. ### What problem does this PR solve? Issue Number: close #xxx
…
When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance. To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list.
Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues:
The auto-loaded default HDFS entry could appear after other storage providers.
Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters.
Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs.
Previously, the catalog used:
catalogProperty.getStoragePropertiesMap().values()to obtain the list of StorageProperties.
However,
getStoragePropertiesMap()stores the properties in a Map, and Map iteration order is not guaranteed.
This caused the resolved storage configurations to be non-deterministic.
Problem
Some Hadoop/S3 parameters—such as:
fs.s3a.aws.credentials.providerhave default values.
When multiple StorageProperties instances were merged in unpredictable order, these defaults could override user-configured settings.
This created actual production issues:
When using IAM Role (Assume Role), the correct provider must be AwsAssumedRoleCredentialProvider.
Because the list order was unstable, sometimes the default provider overrode the intended Assume-Role provider.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)