Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
cpp/velox/utils/ConfigExtractor.cc
Outdated
| std::string_view kSparkHadoopAbfsPrefix = "spark.hadoop.fs.azure."; | ||
| std::string_view kAbfsPrefix = "fs.azure."; | ||
| for (const auto& [key, value] : conf->rawConfigs()) { | ||
| if (key.find(kAbfsPrefix) == 0) { |
There was a problem hiding this comment.
Let's always use prefix spark.hadoop.fs.azure. in Gluten to align with Spark's configuration.
|
@zhli1142015 Can you take a look? would you like to add the abfs configurations to Gluten's doc or reuse the Velox one? Does Velox support all the abfs configurations in Spark now? |
| #ifdef ENABLE_ABFS | ||
| std::string_view kSparkHadoopPrefix = "spark.hadoop."; | ||
| std::string_view kSparkHadoopAbfsPrefix = "spark.hadoop.fs.azure."; | ||
| for (const auto& [key, value] : conf->rawConfigs()) { |
There was a problem hiding this comment.
Please also update getNativeBackendConf
https://github.com/apache/incubator-gluten/blob/5fb0448ae2f5d3347b643fb219aecfeda7322802/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala#L549
There was a problem hiding this comment.
@zhli1142015 done! I missed this. Thanks.
|
Run Gluten Clickhouse CI on x86 |
zhli1142015
left a comment
zhli1142015
left a comment
There was a problem hiding this comment.
Thanks for your change.
…11675) facebookincubator/velox#11623 and #8403 added support for SharedKey, SAS, and OAuth ABFS access mechanisms. Updating the doc to reflect this.
3 participants
What changes were proposed in this pull request?
Velox added new authentication mechanisms for ABFS: facebookincubator/velox#11623
We need to plumb these configs.
How was this patch tested?
ABFS is not enabled in CI. Tested locally on a branch majetideepak@2ed4786