Skip to content

[feature](Cloud) Generate one suffix to distinguish different hdfs root path #33411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
platoneko merged 3 commits into from
Apr 9, 2024
Merged

[feature](Cloud) Generate one suffix to distinguish different hdfs root path #33411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2720fdf
generate one suffix to distinguish suffix
ByteYue Apr 9, 2024
f2dace0
use config to control
ByteYue Apr 9, 2024
ab07272
Update config.h
ByteYue Apr 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cloud/src/common/config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,4 +165,6 @@ CONF_String(kerberos_ccache_path, "");
// set krb5.conf path, use "/etc/krb5.conf" by default
CONF_String(kerberos_krb5_conf_path, "/etc/krb5.conf");

CONF_mBool(enable_distinguish_hdfs_path, "true");

} // namespace doris::cloud::config
10 changes: 10 additions & 0 deletions cloud/src/meta-service/meta_service_resource.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
// under the License.

#include <brpc/channel.h>
Comment thread
ByteYue marked this conversation as resolved.
#include <butil/guid.h>
#include <fmt/core.h>
#include <gen_cpp/cloud.pb.h>

Expand Down Expand Up @@ -387,6 +388,15 @@ static int add_hdfs_storage_vault(InstanceInfoPB& instance, Transaction* txn,
msg = fmt::format("invalid prefix: ", *prefix);
return -1;
}
if (config::enable_distinguish_hdfs_path) {
auto uuid_suffix = butil::GenerateGUID();
if (uuid_suffix.empty()) [[unlikely]] {
code = MetaServiceCode::UNDEFINED_ERR;
msg = fmt::format("failed to generate one suffix for hdfs prefix");
return -1;
}
*prefix = fmt::format("{}_{}", *prefix, uuid_suffix);
}

auto* fs_name = hdfs_param.mutable_hdfs_info()->mutable_build_conf()->mutable_fs_name();
if (!normalize_hdfs_fs_name(*fs_name)) {
Expand Down