Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add separate config directive for Buffer profile #19721

Merged
merged 2 commits into from Feb 12, 2021

Conversation

azat
Copy link
Collaborator

@azat azat commented Jan 27, 2021

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add separate config directive for Buffer profile

Detailed description / Documentation draft:
If you push data via Buffer engine then all your queries will be done
from one user, however this is not always desired behavior, since this
will not allow to limit queries with max_concurrent_queries_for_user and
similar.

@robot-clickhouse robot-clickhouse added the pr-improvement Pull request with some product improvements label Jan 27, 2021
@azat
Copy link
Collaborator Author

azat commented Jan 29, 2021

Integration tests (release) — fail: 2, passed: 1087, error: 0
Integration tests (thread) — fail: 2, passed: 1081, error: 0

Failed in upstream too

If you push data via Buffer engine then all your queries will be done
from one user, however this is not always desired behavior, since this
will not allow to limit queries with max_concurrent_queries_for_user and
similar.
@azat azat marked this pull request as draft February 10, 2021 18:59
@azat azat marked this pull request as ready for review February 10, 2021 19:46
@azat
Copy link
Collaborator Author

azat commented Feb 11, 2021

Stress test (thread) — Sanitizer assert (in stderr.log)

Stats: LargeMmapAllocator: allocated 377831 times, remains 4869 (3693112 K) max 6117 M; by size logs: 17:23222; 18:72316; 19:107149; 20:143857; 21:29088; 22:1313; 23:673; 24:147; 25:66; 
==299==Launching Symbolizer process: /usr/bin/llvm-symbolizer-11 --inlining=true --default-arch=x86_64 
==================
WARNING: ThreadSanitizer: thread leak (pid=299)
  Thread T694 'QueryPipelineEx' (tid=21637, finished) created by thread T595 at:
    #0 pthread_create <null> (clickhouse+0x8b08c4b)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:509:10 (clickhouse+0x8bf88a0)
    #2 std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(), void>(void&&, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...) obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:307:16 (clickhouse+0x8bf88a0)
    #3 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:124:35 (clickhouse+0x8bf422a)
    #4 ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:160:5 (clickhouse+0x8bf4957)
    #5 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) obj-x86_64-linux-gnu/../src/Common/ThreadPool.h:162:38 (clickhouse+0x12fdbf7e)
    #6 DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) obj-x86_64-linux-gnu/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:104:24 (clickhouse+0x12fdbf7e)
    #7 DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) obj-x86_64-linux-gnu/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:144:10 (clickhouse+0x12fdc736)
    #8 DB::TCPHandler::processOrdinaryQueryWithProcessors() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:622:25 (clickhouse+0x12f24960)
    #9 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:284:17 (clickhouse+0x12f1ec60)
    #10 DB::TCPHandler::run() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1459:9 (clickhouse+0x12f2cd57)
    #11 Poco::Net::TCPServerConnection::start() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x15a06172)
    #12 Poco::Net::TCPServerDispatcher::run() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:113:19 (clickhouse+0x15a0689e)
    #13 Poco::PooledThread::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x15b6f271)
    #14 Poco::(anonymous namespace)::RunnableHolder::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x15b6d80f)
    #15 Poco::ThreadImpl::runnableEntry(void*) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x15b6c017)

#20032

Copy link
Member

@alexey-milovidov alexey-milovidov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

@alexey-milovidov alexey-milovidov self-assigned this Feb 12, 2021
@alexey-milovidov alexey-milovidov merged commit 2fe2190 into ClickHouse:master Feb 12, 2021
@azat azat deleted the buffer-profile branch February 13, 2021 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-improvement Pull request with some product improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants