Skip to content

[enhancement](cloud) add bvar to monitor s3 throughput&QPS #34087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dataroaring merged 2 commits into from
Apr 25, 2024
Merged

[enhancement](cloud) add bvar to monitor s3 throughput&QPS #34087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4bcee1f
[enhancement](cloud) add bvar to monitor s3 throughput&QPS
freemandealer Apr 24, 2024
475752a
Merge branch 'master' into s3bvar
freemandealer Apr 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions be/src/io/fs/hdfs_file_reader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
#include <ostream>
#include <utility>

#include "bvar/latency_recorder.h"
#include "bvar/reducer.h"
#include "common/compiler_util.h" // IWYU pragma: keep
#include "common/logging.h"
#include "common/sync_point.h"
Expand All @@ -33,6 +35,13 @@
#include "util/doris_metrics.h"

namespace doris::io {

bvar::Adder<uint64_t> hdfs_bytes_read_total("hdfs_file_reader", "bytes_read");
bvar::LatencyRecorder hdfs_bytes_per_read("hdfs_file_reader", "bytes_per_read"); // also QPS
bvar::PerSecond<bvar::Adder<uint64_t>> hdfs_read_througthput("hdfs_file_reader",
"hdfs_read_throughput",
&hdfs_bytes_read_total);

namespace {

Result<FileHandleCache::Accessor> get_file(const hdfsFS& fs, const Path& file, int64_t mtime,
Expand Down Expand Up @@ -148,6 +157,8 @@ Status HdfsFileReader::read_at_impl(size_t offset, Slice result, size_t* bytes_r
has_read += loop_read;
}
*bytes_read = has_read;
hdfs_bytes_read_total << *bytes_read;
hdfs_bytes_per_read << *bytes_read;
return Status::OK();
}

Expand Down Expand Up @@ -206,6 +217,8 @@ Status HdfsFileReader::read_at_impl(size_t offset, Slice result, size_t* bytes_r
has_read += loop_read;
}
*bytes_read = has_read;
hdfs_bytes_read_total << *bytes_read;
hdfs_bytes_per_read << *bytes_read;
return Status::OK();
}
#endif
Expand Down
5 changes: 5 additions & 0 deletions be/src/io/fs/s3_file_reader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
#include <aws/s3/S3Errors.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <aws/s3/model/GetObjectResult.h>
#include <bvar/latency_recorder.h>
#include <bvar/reducer.h>
#include <fmt/format.h>
#include <glog/logging.h>
Expand All @@ -43,6 +44,9 @@ bvar::Adder<uint64_t> s3_file_reader_read_counter("s3_file_reader", "read_at");
bvar::Adder<uint64_t> s3_file_reader_total("s3_file_reader", "total_num");
bvar::Adder<uint64_t> s3_bytes_read_total("s3_file_reader", "bytes_read");
bvar::Adder<uint64_t> s3_file_being_read("s3_file_reader", "file_being_read");
bvar::LatencyRecorder s3_bytes_per_read("s3_file_reader", "bytes_per_read"); // also QPS
Copy link
Contributor

@gavinchou gavinchou Apr 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems calculating the rate of s3_bytes_read_total (throughput) and s3_file_reader_read_counter (qps), by grafana, will do the work.
Adding redundant metrics is a burden for the scraping procedure (Prometheus).

Copy link
Contributor Author

@freemandealer freemandealer Apr 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bvar has fine granularity, so it is easier to identify throughput issue (whether caused by long wait or low instantaneous speed). So I insist, but we can delete them when the optimization work is done.

bvar::PerSecond<bvar::Adder<uint64_t>> s3_read_througthput("s3_file_reader", "s3_read_throughput",
&s3_bytes_read_total);

Result<FileReaderSPtr> S3FileReader::create(std::shared_ptr<const S3ClientHolder> client,
std::string bucket, std::string key,
Expand Down Expand Up @@ -125,6 +129,7 @@ Status S3FileReader::read_at_impl(size_t offset, Slice result, size_t* bytes_rea
_path.native(), *bytes_read, bytes_req);
}
s3_bytes_read_total << *bytes_read;
s3_bytes_per_read << *bytes_read;
s3_file_reader_read_counter << 1;
DorisMetrics::instance()->s3_bytes_read_total->increment(*bytes_read);
return Status::OK();
Expand Down