Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expose Peak memory usage in query statistics. #51946

Merged
merged 21 commits into from Jul 25, 2023
Merged

Expose Peak memory usage in query statistics. #51946

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a2c9e26
Add peak memory usage (prototype)
Jun 28, 2023
a41ec12
Http prototype
Jun 28, 2023
b276f45
Remove whilespace
Jun 28, 2023
ffa4f37
Try to fix style
Jun 29, 2023
7e99295
Next trying slyle fix
Jun 29, 2023
6f4d6fd
Remove whitespace
Jun 29, 2023
bdfaffb
Add/fix tests
Jun 29, 2023
54838e9
Remove whitespace
Jun 29, 2023
901089b
Remove comment
Jun 30, 2023
3fcfbb1
Merge remote-tracking branch 'clickhouse/master' into ADQM-940
Jul 7, 2023
09ba975
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 10, 2023
1a7ee40
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 11, 2023
7704469
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 11, 2023
feebad3
Try to fix tests by adding mutex like in onProgress()
Jul 14, 2023
2bc7bb0
Add peak to http progress; not display peak if old server
Jul 17, 2023
1d4d829
Remove spaces
Jul 17, 2023
e24883d
Cut peak memory to fix tests
Jul 17, 2023
68a501a
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 18, 2023
62f9a95
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 24, 2023
810137e
Add new peak_memory_usage to docs
Jul 24, 2023
d4d381d
Merge remote-tracking branch 'origin/master' into ADQM-940
Jul 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/Client/ClientBase.cpp
View file
Open in desktop
Expand Up @@ -1191,6 +1191,8 @@ void ClientBase::onProfileEvents(Block & block)
thread_times[host_name].system_ms = value;
else if (event_name == MemoryTracker::USAGE_EVENT_NAME)
thread_times[host_name].memory_usage = value;
else if (event_name == MemoryTracker::PEAK_USAGE_EVENT_NAME)
thread_times[host_name].peak_memory_usage = value;
}
progress_indication.updateThreadEventData(thread_times);

Expand Down
1 change: 1 addition & 0 deletions src/Common/MemoryTracker.h
View file
Open in desktop
Expand Up @@ -95,6 +95,7 @@ class MemoryTracker
public:

static constexpr auto USAGE_EVENT_NAME = "MemoryTrackerUsage";
static constexpr auto PEAK_USAGE_EVENT_NAME = "MemoryTrackerPeakUsage";

explicit MemoryTracker(VariableContext level_ = VariableContext::Thread);
explicit MemoryTracker(MemoryTracker * parent_, VariableContext level_ = VariableContext::Thread);
Expand Down
7 changes: 5 additions & 2 deletions src/Common/ProgressIndication.cpp
View file
Open in desktop
Expand Up @@ -83,7 +83,7 @@ ProgressIndication::MemoryUsage ProgressIndication::getMemoryUsage() const
[](MemoryUsage const & acc, auto const & host_data)
{
UInt64 host_usage = host_data.second.memory_usage;
return MemoryUsage{.total = acc.total + host_usage, .max = std::max(acc.max, host_usage)};
return MemoryUsage{.total = acc.total + host_usage, .max = std::max(acc.max, host_usage), .peak = std::max(acc.peak, host_data.second.peak_memory_usage)};
});
}

Expand All @@ -101,6 +101,9 @@ void ProgressIndication::writeFinalProgress()
<< formatReadableSizeWithDecimalSuffix(progress.read_bytes * 1000000000.0 / elapsed_ns) << "/s.)";
else
std::cout << ". ";
auto peak_memory_usage = getMemoryUsage().peak;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one concern here how a new client works with an old server. Client would print 0 as peak_memory_usage

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me that this entire line is for information, and it is unlikely that the client will rely on this value.
Try not to display the value for old servers?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would right and heavy. May be just enough not to log it when it zero. Just because memory usage never is zero.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added check (not show if old server). I use '-1' (as default) because memory can be zero (I don't know why it works like this; on small queries always 0).

if (peak_memory_usage >= 0)
std::cout << "\nPeak memory usage (for query) " << formatReadableSizeWithBinarySuffix(peak_memory_usage) << ".";
}

void ProgressIndication::writeProgress(WriteBufferFromFileDescriptor & message)
Expand Down Expand Up @@ -152,7 +155,7 @@ void ProgressIndication::writeProgress(WriteBufferFromFileDescriptor & message)
std::string profiling_msg;

double cpu_usage = getCPUUsage();
auto [memory_usage, max_host_usage] = getMemoryUsage();
auto [memory_usage, max_host_usage, peak_usage] = getMemoryUsage();

if (cpu_usage > 0 || memory_usage > 0)
{
Expand Down
4 changes: 4 additions & 0 deletions src/Common/ProgressIndication.h
View file
Open in desktop
Expand Up @@ -22,6 +22,9 @@ struct ThreadEventData
UInt64 user_ms = 0;
UInt64 system_ms = 0;
UInt64 memory_usage = 0;

// -1 used as flag 'is not show for old servers'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is not show -> is not shown

Int64 peak_memory_usage = -1;
};

using HostToTimesMap = std::unordered_map<String, ThreadEventData>;
Expand Down Expand Up @@ -64,6 +67,7 @@ class ProgressIndication
{
UInt64 total = 0;
UInt64 max = 0;
Int64 peak = -1;
};

MemoryUsage getMemoryUsage() const;
Expand Down
14 changes: 9 additions & 5 deletions src/IO/Progress.cpp
View file
Open in desktop
Expand Up @@ -69,12 +69,14 @@ void ProgressValues::write(WriteBuffer & out, UInt64 client_revision) const
}
}

void ProgressValues::writeJSON(WriteBuffer & out) const
void ProgressValues::writeJSON(WriteBuffer & out, bool add_braces) const
{
/// Numbers are written in double quotes (as strings) to avoid loss of precision
/// of 64-bit integers after interpretation by JavaScript.

writeCString("{\"read_rows\":\"", out);
if (add_braces)
writeCString("{", out);
writeCString("\"read_rows\":\"", out);
writeText(read_rows, out);
writeCString("\",\"read_bytes\":\"", out);
writeText(read_bytes, out);
Expand All @@ -88,7 +90,9 @@ void ProgressValues::writeJSON(WriteBuffer & out) const
writeText(result_rows, out);
writeCString("\",\"result_bytes\":\"", out);
writeText(result_bytes, out);
writeCString("\"}", out);
writeCString("\"", out);
if (add_braces)
writeCString("}", out);
}

bool Progress::incrementPiecewiseAtomically(const Progress & rhs)
Expand Down Expand Up @@ -230,9 +234,9 @@ void Progress::write(WriteBuffer & out, UInt64 client_revision) const
getValues().write(out, client_revision);
}

void Progress::writeJSON(WriteBuffer & out) const
void Progress::writeJSON(WriteBuffer & out, bool add_braces) const
{
getValues().writeJSON(out);
getValues().writeJSON(out, add_braces);
}

}
4 changes: 2 additions & 2 deletions src/IO/Progress.h
View file
Open in desktop
Expand Up @@ -32,7 +32,7 @@ struct ProgressValues

void read(ReadBuffer & in, UInt64 server_revision);
void write(WriteBuffer & out, UInt64 client_revision) const;
void writeJSON(WriteBuffer & out) const;
void writeJSON(WriteBuffer & out, bool add_braces = true) const;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default values are harmful.

};

struct ReadProgress
Expand Down Expand Up @@ -118,7 +118,7 @@ struct Progress
void write(WriteBuffer & out, UInt64 client_revision) const;

/// Progress in JSON format (single line, without whitespaces) is used in HTTP headers.
void writeJSON(WriteBuffer & out) const;
void writeJSON(WriteBuffer & out, bool add_braces = true) const;

/// Each value separately is changed atomically (but not whole object).
bool incrementPiecewiseAtomically(const Progress & rhs);
Expand Down
12 changes: 10 additions & 2 deletions src/Interpreters/ProfileEventsExt.cpp
View file
Open in desktop
Expand Up @@ -86,9 +86,16 @@ static void dumpMemoryTracker(ProfileEventsSnapshot const & snapshot, DB::Mutabl
columns[i++]->insert(static_cast<UInt64>(snapshot.current_time));
columns[i++]->insert(static_cast<UInt64>(snapshot.thread_id));
columns[i++]->insert(Type::GAUGE);

columns[i++]->insertData(MemoryTracker::USAGE_EVENT_NAME, strlen(MemoryTracker::USAGE_EVENT_NAME));
columns[i++]->insert(snapshot.memory_usage);
columns[i]->insert(snapshot.memory_usage);

i = 0;
columns[i++]->insertData(host_name.data(), host_name.size());
columns[i++]->insert(static_cast<UInt64>(snapshot.current_time));
columns[i++]->insert(static_cast<UInt64>(snapshot.thread_id));
columns[i++]->insert(Type::GAUGE);
columns[i++]->insertData(MemoryTracker::PEAK_USAGE_EVENT_NAME, strlen(MemoryTracker::PEAK_USAGE_EVENT_NAME));
columns[i]->insert(snapshot.peak_memory_usage);
}
Comment on lines +90 to 99
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain those lines to me. I do not understand it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data (ProfileEvents and memory events) between the server and the client passes through row in the sql block (DB::Block). I added one more row with my data (peak memory).
Columns:

static const NamesAndTypesList column_names_and_types = {
        {"host_name", std::make_shared<DataTypeString>()},
        {"current_time", std::make_shared<DataTypeDateTime>()},
        {"thread_id", std::make_shared<DataTypeUInt64>()},
        {"type", TypeEnum},
        {"name", std::make_shared<DataTypeString>()},
        {"value", std::make_shared<DataTypeInt64>()},
    };

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is Ok.
The diff itself looks strange.

image

Looks like one write is not enough and you added two :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, my bad.
That is right. You add one more row here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand, what exactly is confusing? The first block write current_memory_usage. Insert() appends new value at the end of column (column's size is increased by 1). i++ in block ends is not necessary, i replaced to i;


void getProfileEvents(
Expand Down Expand Up @@ -121,6 +128,7 @@ void getProfileEvents(
group_snapshot.thread_id = 0;
group_snapshot.current_time = time(nullptr);
group_snapshot.memory_usage = thread_group->memory_tracker.get();
group_snapshot.peak_memory_usage = thread_group->memory_tracker.getPeak();
auto group_counters = thread_group->performance_counters.getPartiallyAtomicSnapshot();
auto prev_group_snapshot = last_sent_snapshots.find(0);
group_snapshot.counters =
Expand Down
1 change: 1 addition & 0 deletions src/Interpreters/ProfileEventsExt.h
View file
Open in desktop
Expand Up @@ -16,6 +16,7 @@ struct ProfileEventsSnapshot
UInt64 thread_id;
CountersIncrement counters;
Int64 memory_usage;
Int64 peak_memory_usage;
time_t current_time;
};

Expand Down
31 changes: 17 additions & 14 deletions src/Server/HTTP/WriteBufferFromHTTPServerResponse.cpp
View file
Open in desktop
Expand Up @@ -3,7 +3,7 @@
#include <IO/HTTPCommon.h>
#include <IO/Progress.h>
#include <IO/WriteBufferFromString.h>

#include <IO/WriteHelpers.h>

namespace DB
{
Expand All @@ -29,28 +29,31 @@ void WriteBufferFromHTTPServerResponse::startSendHeaders()
}
}

void WriteBufferFromHTTPServerResponse::writeHeaderSummary()
void WriteBufferFromHTTPServerResponse::writeHeaderProgressImpl(const char * header_name)
{
if (headers_finished_sending)
return;

WriteBufferFromOwnString progress_string_writer;
accumulated_progress.writeJSON(progress_string_writer);

writeCString("{", progress_string_writer);
accumulated_progress.writeJSON(progress_string_writer, false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like trash.

writeCString(",\"peak_memory_usage\":\"", progress_string_writer);
writeText(peak_memory_usage, progress_string_writer);
writeCString("\"}", progress_string_writer);

if (response_header_ostr)
*response_header_ostr << "X-ClickHouse-Summary: " << progress_string_writer.str() << "\r\n" << std::flush;
*response_header_ostr << header_name << progress_string_writer.str() << "\r\n" << std::flush;
}

void WriteBufferFromHTTPServerResponse::writeHeaderProgress()
void WriteBufferFromHTTPServerResponse::writeHeaderSummary()
{
if (headers_finished_sending)
return;

WriteBufferFromOwnString progress_string_writer;
accumulated_progress.writeJSON(progress_string_writer);
writeHeaderProgressImpl("X-ClickHouse-Summary: ");
}

if (response_header_ostr)
*response_header_ostr << "X-ClickHouse-Progress: " << progress_string_writer.str() << "\r\n" << std::flush;
void WriteBufferFromHTTPServerResponse::writeHeaderProgress()
{
writeHeaderProgressImpl("X-ClickHouse-Progress: ");
}

void WriteBufferFromHTTPServerResponse::writeExceptionCode()
Expand Down Expand Up @@ -149,7 +152,7 @@ WriteBufferFromHTTPServerResponse::WriteBufferFromHTTPServerResponse(
}


void WriteBufferFromHTTPServerResponse::onProgress(const Progress & progress)
void WriteBufferFromHTTPServerResponse::onProgress(const Progress & progress, Int64 peak_memory_usage_)
{
std::lock_guard lock(mutex);

Expand All @@ -158,7 +161,7 @@ void WriteBufferFromHTTPServerResponse::onProgress(const Progress & progress)
return;

accumulated_progress.incrementPiecewiseAtomically(progress);

peak_memory_usage = peak_memory_usage_;
if (send_progress && progress_watch.elapsed() >= send_progress_interval_ms * 1000000)
{
progress_watch.restart();
Expand Down
6 changes: 5 additions & 1 deletion src/Server/HTTP/WriteBufferFromHTTPServerResponse.h
View file
Open in desktop
Expand Up @@ -43,7 +43,7 @@ class WriteBufferFromHTTPServerResponse final : public BufferWithOwnMemory<Write
~WriteBufferFromHTTPServerResponse() override;

/// Writes progress in repeating HTTP headers.
void onProgress(const Progress & progress);
void onProgress(const Progress & progress, Int64 peak_memory_usage_);

/// Turn compression on or off.
/// The setting has any effect only if HTTP headers haven't been sent yet.
Expand Down Expand Up @@ -89,6 +89,8 @@ class WriteBufferFromHTTPServerResponse final : public BufferWithOwnMemory<Write
/// but not finish them with \r\n, allowing to send more headers subsequently.
void startSendHeaders();

// Used for write the header X-ClickHouse-Progress / X-ClickHouse-Summary
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used for write -> Used for writing

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double whitespace.

void writeHeaderProgressImpl(const char * header_name);
// Used for write the header X-ClickHouse-Progress
void writeHeaderProgress();
// Used for write the header X-ClickHouse-Summary
Expand Down Expand Up @@ -126,6 +128,8 @@ class WriteBufferFromHTTPServerResponse final : public BufferWithOwnMemory<Write

int exception_code = 0;

Int64 peak_memory_usage = 0;

std::mutex mutex; /// progress callback could be called from different threads.
};

Expand Down
6 changes: 5 additions & 1 deletion src/Server/HTTPHandler.cpp
View file
Open in desktop
Expand Up @@ -818,7 +818,11 @@ void HTTPHandler::processQuery(

/// While still no data has been sent, we will report about query execution progress by sending HTTP headers.
/// Note that we add it unconditionally so the progress is available for `X-ClickHouse-Summary`
append_callback([&used_output](const Progress & progress) { used_output.out->onProgress(progress); });
append_callback([&used_output](const Progress & progress)
{
const auto& thread_group = CurrentThread::getGroup();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong style: missing whitespace.

used_output.out->onProgress(progress, thread_group->memory_tracker.getPeak());
});

if (settings.readonly > 0 && settings.cancel_http_readonly_queries_on_client_close)
{
Expand Down
6 changes: 3 additions & 3 deletions tests/queries/0_stateless/00416_pocopatch_progress_in_http_headers.sh
View file
Open in desktop
Expand Up @@ -10,7 +10,7 @@ result=""
lines_expected=4
counter=0
while [ $counter -lt $RETRIES ] && [ "$(echo "$result" | wc -l)" != "$lines_expected" ]; do
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=5&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0" -d 'SELECT max(number) FROM numbers(10)' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Progress|^[0-9]')
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=5&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0" -d 'SELECT max(number) FROM numbers(10)' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Progress|^[0-9]' | sed 's/,\"peak_mem[^}]*//')
let counter=counter+1
done
echo "$result"
Expand All @@ -19,7 +19,7 @@ result=""
lines_expected=12
counter=0
while [ $counter -lt $RETRIES ] && [ "$(echo "$result" | wc -l)" != "$lines_expected" ]; do
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=1&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0&output_format_parallel_formatting=0" -d 'SELECT number FROM numbers(10)' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Progress|^[0-9]')
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=1&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0&output_format_parallel_formatting=0" -d 'SELECT number FROM numbers(10)' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Progress|^[0-9]'| sed 's/,\"peak_mem[^}]*//')
let counter=counter+1
done
echo "$result"
Expand All @@ -46,7 +46,7 @@ ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}" -H 'Accept-Encoding: gzip' -d 'CREAT
result=""
counter=0
while [ $counter -lt $RETRIES ] && [ -z "$result" ]; do
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=1&http_headers_progress_interval_ms=0&send_progress_in_http_headers=1" -d 'INSERT INTO insert_number_query (record) SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Summary|^[0-9]')
result=$(${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&max_block_size=1&http_headers_progress_interval_ms=0&send_progress_in_http_headers=1" -d 'INSERT INTO insert_number_query (record) SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep -E 'Content-Encoding|X-ClickHouse-Summary|^[0-9]' | sed 's/,\"peak_mem[^}]*//')
let counter=counter+1
done
echo "$result"
Expand Down
7 changes: 4 additions & 3 deletions tests/queries/0_stateless/01921_test_progress_bar.py
View file
Open in desktop
Expand Up @@ -14,6 +14,7 @@

with client(name="client1>", log=log) as client1:
client1.expect(prompt)
client1.send("SELECT number FROM numbers(100) FORMAT Null")
client1.expect("Progress: 100\.00 rows, 800\.00 B.*" + end_of_block)
client1.expect("0 rows in set. Elapsed: [\\w]{1}\.[\\w]{3} sec." + end_of_block)
client1.send("SELECT number FROM numbers(1000) FORMAT Null")
client1.expect("Progress: 1\.00 thousand rows, 8\.00 KB .*" + end_of_block)
client1.expect("0 rows in set. Elapsed: [\\w]{1}\.[\\w]{3} sec.")
client1.expect("Peak memory usage \(for query\) .*B" + end_of_block)
2 changes: 1 addition & 1 deletion tests/queries/0_stateless/02136_scalar_progress.sh
View file
Open in desktop
Expand Up @@ -4,4 +4,4 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
# shellcheck source=../shell_config.sh
. "$CURDIR"/../shell_config.sh

$CLICKHOUSE_CURL -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0" -d "SELECT (SELECT max(number), count(number) FROM numbers(100000) settings max_block_size=65505);" -v 2>&1 | grep -E "X-ClickHouse-Summary|X-ClickHouse-Progress"
$CLICKHOUSE_CURL -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0" -d "SELECT (SELECT max(number), count(number) FROM numbers(100000) settings max_block_size=65505);" -v 2>&1 | grep -E "X-ClickHouse-Summary|X-ClickHouse-Progress" | sed 's/,\"peak_mem[^}]*//'
2 changes: 1 addition & 1 deletion tests/queries/0_stateless/02373_progress_contain_result.sh
View file
Open in desktop
Expand Up @@ -6,4 +6,4 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

echo 'SELECT 1 FROM numbers(100)' |
${CLICKHOUSE_CURL_COMMAND} -v "${CLICKHOUSE_URL}&wait_end_of_query=1&send_progress_in_http_headers=0" --data-binary @- 2>&1 |
grep 'X-ClickHouse-Summary'
grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
12 changes: 6 additions & 6 deletions tests/queries/0_stateless/02423_insert_summary_behaviour.sh
View file
Open in desktop
Expand Up @@ -11,11 +11,11 @@ $CLICKHOUSE_CLIENT -q "CREATE MATERIALIZED VIEW floats_to_target TO target_1 AS
$CLICKHOUSE_CLIENT -q "CREATE MATERIALIZED VIEW floats_to_target_2 TO target_2 AS SELECT * FROM floats, numbers(2) n"

echo "No materialized views"
${CLICKHOUSE_CURL} "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1" -d "VALUES(1.0)" -v 2>&1 | grep 'X-ClickHouse-Summary'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format Native | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1+FORMAT+Native" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format RowBinary | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1+FORMAT+RowBinary" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary'
${CLICKHOUSE_CURL} "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1" -d "VALUES(1.0)" -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format Native | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1+FORMAT+Native" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format RowBinary | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+target_1+FORMAT+RowBinary" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'

echo "With materialized views"
${CLICKHOUSE_CURL} "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats" -d "VALUES(1.0)" -v 2>&1 | grep 'X-ClickHouse-Summary'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format Native | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats+FORMAT+Native" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format RowBinary | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats+FORMAT+RowBinary" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary'
${CLICKHOUSE_CURL} "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats" -d "VALUES(1.0)" -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format Native | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats+FORMAT+Native" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
$CLICKHOUSE_LOCAL -q "SELECT number::Float64 AS v FROM numbers(10)" --format RowBinary | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&wait_end_of_query=1&query=INSERT+INTO+floats+FORMAT+RowBinary" --data-binary @- -v 2>&1 | grep 'X-ClickHouse-Summary' | sed 's/,\"peak_mem[^}]*//'
2 changes: 1 addition & 1 deletion tests/queries/0_stateless/02457_insert_select_progress_http.sh
View file
Open in desktop
Expand Up @@ -5,5 +5,5 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$CURDIR"/../shell_config.sh

${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&send_progress_in_http_headers=1&http_headers_progress_interval_ms=0" -d @- <<< "insert into function null('_ Int') select * from numbers(5) settings max_block_size=1" -v |& {
grep -F -e X-ClickHouse-Progress: -e X-ClickHouse-Summary:
grep -F -e X-ClickHouse-Progress: -e X-ClickHouse-Summary: | sed 's/,\"peak_mem[^}]*//'
}