Skip to content

[VL] Support pass unmapped dynamic and static configs to Velox #11229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Yohahaha merged 2 commits into from
Dec 3, 2025
Merged

[VL] Support pass unmapped dynamic and static configs to Velox #11229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad6860a
fix
Yohahaha Dec 1, 2025
bb291bd
rebase
Yohahaha Dec 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cpp/velox/compute/WholeStageResultIterator.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -663,6 +663,8 @@ std::unordered_map<std::string, std::string> WholeStageResultIterator::getQueryC
setIfExists(kQueryTraceMaxBytes, velox::core::QueryConfig::kQueryTraceMaxBytes);
setIfExists(kQueryTraceTaskRegExp, velox::core::QueryConfig::kQueryTraceTaskRegExp);
setIfExists(kOpTraceDirectoryCreateConfig, velox::core::QueryConfig::kOpTraceDirectoryCreateConfig);

overwriteVeloxConf(veloxCfg_.get(), configs, kDynamicBackendConfPrefix);
} catch (const std::invalid_argument& err) {
std::string errDetails = err.what();
throw std::runtime_error("Invalid conf arg: " + errDetails);
Expand Down
3 changes: 3 additions & 0 deletions cpp/velox/config/VeloxConfig.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,4 +193,7 @@ const std::string kCudfEnableTableScan = "spark.gluten.sql.columnar.backend.velo
const bool kCudfEnableTableScanDefault = false;
const std::string kCudfHiveConnectorId = "cudf-hive";

const std::string kStaticBackendConfPrefix = "spark.gluten.velox.";
const std::string kDynamicBackendConfPrefix = "spark.gluten.sql.columnar.backend.velox.";

} // namespace gluten
14 changes: 14 additions & 0 deletions cpp/velox/utils/ConfigExtractor.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,8 @@ std::shared_ptr<facebook::velox::config::ConfigBase> createHiveConnectorSessionC
conf->get<bool>(kParquetUseColumnNames, true) ? "true" : "false";
configs[facebook::velox::connector::hive::HiveConfig::kOrcUseColumnNamesSession] =
conf->get<bool>(kOrcUseColumnNames, true) ? "true" : "false";

overwriteVeloxConf(conf.get(), configs, kDynamicBackendConfPrefix);
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this introduce some redundant config set?

Copy link
Copy Markdown
Contributor Author

@Yohahaha Yohahaha Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only works for unmapped configs, mapped configs use spark defined name.

return std::make_shared<facebook::velox::config::ConfigBase>(std::move(configs));
}

Expand Down Expand Up @@ -299,7 +301,19 @@ std::shared_ptr<facebook::velox::config::ConfigBase> createHiveConnectorConfig(
// read as UTC
hiveConfMap[facebook::velox::connector::hive::HiveConfig::kReadTimestampPartitionValueAsLocalTime] = "false";

overwriteVeloxConf(conf.get(), hiveConfMap, kStaticBackendConfPrefix);
return std::make_shared<facebook::velox::config::ConfigBase>(std::move(hiveConfMap));
}

void overwriteVeloxConf(
const facebook::velox::config::ConfigBase* from,
std::unordered_map<std::string, std::string>& to,
const std::string& prefix) {
for (const auto& [k, v] : from->rawConfigs()) {
if (k.starts_with(prefix)) {
to[k.substr(prefix.size())] = v;
}
}
}

} // namespace gluten
5 changes: 5 additions & 0 deletions cpp/velox/utils/ConfigExtractor.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,9 @@ std::shared_ptr<facebook::velox::config::ConfigBase> createHiveConnectorConfig(
const std::shared_ptr<facebook::velox::config::ConfigBase>& conf,
FileSystemType fsType = FileSystemType::kAll);

void overwriteVeloxConf(
const facebook::velox::config::ConfigBase* from,
std::unordered_map<std::string, std::string>& to,
const std::string& prefix);

} // namespace gluten