Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Delta Lake] Infinite (?) retries in AWSClient #49959

Open
melvynator opened this issue May 17, 2023 · 3 comments
Open

[Delta Lake] Infinite (?) retries in AWSClient #49959

melvynator opened this issue May 17, 2023 · 3 comments
Labels
potential bug To be reviewed by developers and confirmed/rejected.

Comments

@melvynator
Copy link
Member

Query to delta lake is returning timeout.

Even simple queries like: select count(1)

It might be related to: #48867
@tavplubix
Copy link
Member

Provided information is not enough to do anything. Please provide:

  • exact error message with stacktrace
  • output of this query with the id of some query that failed with timeout

@tavplubix tavplubix added st-need-info We need extra data to continue (waiting for response) performance and removed unexpected behaviour labels May 17, 2023
@alifirat
Copy link

Hey @tavplubix
We have generated a dataset with 1776 data files (Parquet format) and we have started to read the data using ClickHouse.

First, the time to create the table takes

clickhouse1 :) SET send_logs_level = 'trace';
clickhouse1:) create table test Engine = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/')
CREATE TABLE test
ENGINE = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/')

Query id: a6342c0c-fa3b-404e-b075-894c8fcd4be4

[clickhouse1] 2023.05.19 07:31:05.766713 [ 9 ] {a6342c0c-fa3b-404e-b075-894c8fcd4be4} <Debug> executeQuery: (from 127.0.0.1:43818) create table test Engine = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/') (stage: Complete)




Timeout exceeded while receiving data from server. Waited for 300 seconds, timeout is 300 seconds.
Cancelling query.

SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5
SETTINGS allow_introspection_functions = 1

Query id: 946ed89f-a7a3-48ef-97a2-bbd72be18faf

┌─count()─┬─sym────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│     146 │ __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse
bssl::ssl_run_handshake(bssl::SSL_HANDSHAKE*, bool*)
    /usr/bin/clickhouse
SSL_do_handshake
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connectSSL(bool)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│     116 │ poll
    /lib/libc.so.6
Poco::Net::SocketImpl::pollImpl(Poco::Timespan&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│     102 │ __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
SSL_read
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int)
    /usr/bin/clickhouse
Poco::Net::HTTPSession::refill()
    /usr/bin/clickhouse
Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::uflow()
    /usr/bin/clickhouse
std::__1::basic_istream<char, std::__1::char_traits<char>>::get()
    /usr/bin/clickhouse
Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char>>&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│      24 │ MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│      24 │ MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
└─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.044 sec. Processed 4.66 thousand rows, 1.48 MB (105.07 thousand rows/s., 33.28 MB/s.)

clickhouse1 :) SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5 SETTINGS allow_introspection_functions = true Format Vertical

SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5
SETTINGS allow_introspection_functions = 1
FORMAT Vertical

Query id: e7682830-ccf7-480c-a245-7d019c032c14

Row 1:
──────
count(): 146
sym:     __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse
bssl::ssl_run_handshake(bssl::SSL_HANDSHAKE*, bool*)
    /usr/bin/clickhouse
SSL_do_handshake
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connectSSL(bool)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 2:
──────
count(): 116
sym:     poll
    /lib/libc.so.6
Poco::Net::SocketImpl::pollImpl(Poco::Timespan&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 3:
──────
count(): 102
sym:     __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
SSL_read
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int)
    /usr/bin/clickhouse
Poco::Net::HTTPSession::refill()
    /usr/bin/clickhouse
Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::uflow()
    /usr/bin/clickhouse
std::__1::basic_istream<char, std::__1::char_traits<char>>::get()
    /usr/bin/clickhouse
Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char>>&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 4:
──────
count(): 24
sym:     MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 5:
──────
count(): 24
sym:     MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Not sure to understand what happens when creating the table, maybe infering the schema from all the data files take to much time ? I now that for Apache Iceberg, we do a head-object request to every data file, maybe it's the same for Delta Lake ?

@alifirat alifirat mentioned this issue May 19, 2023
Open
@tavplubix tavplubix changed the title [Delta Lake] Performance issue [Delta Lake] Infinite (?) retries in AWSClient May 19, 2023
@tavplubix tavplubix added potential bug To be reviewed by developers and confirmed/rejected. and removed performance st-need-info We need extra data to continue (waiting for response) labels May 19, 2023
@alifirat
Copy link

One line in the error stack trace may me think that we have the same issues between Iceberg and Delta Lake.
I see the function getTableStructureFromDataImpl that calls the keys iterator, I guess it's done for the schema inference. While the issue on Iceberg is different, I still think when there is a plenty of data files, it takes too much time so solving this schema inference must solve the issues for both Iceberg / Delta Lake.
I don't know much for Apache Hudi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
potential bug To be reviewed by developers and confirmed/rejected.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@melvynator @alifirat @tavplubix