Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite PushingToViews #28582

Merged
merged 68 commits into from
Sep 27, 2021
Merged

Rewrite PushingToViews #28582

merged 68 commits into from
Sep 27, 2021

Conversation

KochetovNicolai
Copy link
Member

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Sep 3, 2021
@KochetovNicolai KochetovNicolai added force tests The label does nothing, NOOP, None, nil and removed force tests The label does nothing, NOOP, None, nil labels Sep 9, 2021
@KochetovNicolai KochetovNicolai added the force tests The label does nothing, NOOP, None, nil label Sep 9, 2021
@KochetovNicolai KochetovNicolai removed the force tests The label does nothing, NOOP, None, nil label Sep 10, 2021
@KochetovNicolai KochetovNicolai marked this pull request as ready for review September 23, 2021 10:44
@alexey-milovidov alexey-milovidov mentioned this pull request Sep 26, 2021
Closed
KochetovNicolai
KochetovNicolai commented Sep 27, 2021
View reviewed changes
Copy link
Member Author

@KochetovNicolai KochetovNicolai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@KochetovNicolai KochetovNicolai merged commit 236d71e into master Sep 27, 2021
@KochetovNicolai KochetovNicolai deleted the rewrite-pushing-to-views branch September 27, 2021 18:19
@Algunenano Algunenano mentioned this pull request Sep 28, 2021
Merged
@azat azat mentioned this pull request Sep 28, 2021
Merged
Algunenano
Algunenano reviewed Sep 29, 2021
View reviewed changes
src/DataStreams/PushingToViewsBlockOutputStream.cpp
processors.emplace_front(std::move(copying_data));
processors.emplace_back(std::move(finalizing_views));
result_chain = Chain(std::move(processors));
result_chain.setNumThreads(max_parallel_streams);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I'm reading this correctly, but this seems to be ignoring the settings.parallel_view_processing set above right? It seems that now views_data->max_threads is only used to log, but not to control parallelism. Is that correct and intended?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have probably missed it.
Actually, the idea is that views could be processed always in parallel (or not, which depends only on number of executing threads). But probably the proper number of threads is not set.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I'm reading this correctly, but this seems to be ignoring the settings.parallel_view_processing set above right?

It should be accounted here -

ClickHouse/src/Interpreters/InterpreterInsertQuery.cpp

Lines 410 to 412 in 3663d23

/// Don't use more threads for insert then for select to reduce memory consumption.
if (!settings.parallel_view_processing && pipeline.getNumThreads() > num_select_threads)
pipeline.setMaxThreads(num_select_threads);

But probably the proper number of threads is not set.

#29786

azat added a commit to azat/ClickHouse that referenced this pull request Sep 29, 2021
@azat
Fix LOGICAL_ERROR for INSERT with metadata changes in parallel
4618636 
In ClickHouse#28582 the problem was introduced due to it obtains new metadata
snapshot and so it may fail when it will check that headers does not
differs, like in [1]:

<details>

    {7b1256b1-09dc-4cc3-93d9-e9f6e26b0b99} <Fatal> : Logical error: 'Block structure mismatch in  function connect between ConvertingTransform and ReplicatedMergeTreeSink stream: different number of columns:
    a UInt8 UInt8(size = 0), b Int16 Int16(size = 0), c Float32 Float32(size = 0), d String String(size = 0), e Array(UInt8) Array(size = 0, UInt64(size = 0), UInt8(size = 0)), f Nullable(UUID) Nullable(size = 0, UUID(size = 0), UInt8(size = 0)), g Tuple(UInt8, UInt16) Tuple(size = 0, UInt8(size = 0), UInt16(size = 0)), h UInt64 UInt64(size = 0)
    a UInt8 UInt8(size = 0), b Int16 Int16(size = 0), c Float32 Float32(size = 0), d String String(size = 0), e Array(UInt8) Array(size = 0, UInt64(size = 0), UInt8(size = 0)), f Nullable(UUID) Nullable(size = 0, UUID(size = 0), UInt8(size = 0)), g Tuple(UInt8, UInt16) Tuple(size = 0, UInt8(size = 0), UInt16(size = 0))'.
    {7b1256b1-09dc-4cc3-93d9-e9f6e26b0b99} <Fatal> : Logical error: 'Block structure mismatch in  function connect between ConvertingTransform and ReplicatedMergeTreeSink stream: different number of columns:
    {} <Fatal> BaseDaemon: ########################################
    {} <Fatal> BaseDaemon: (version 21.11.1.8227, build id: E012027D3A9E94FD61655974C45C15F58215605B) (from thread 2180) (query_id: 7b1256b1-09dc-4cc3-93d9-e9f6e26b0b99) Received signal Aborted (6)
    {} <Fatal> BaseDaemon:
    {} <Fatal> BaseDaemon: Stack trace: 0x7f5661a3b18b 0x7f5661a1a859 0x1333bcb8 0x1333bdc2 0x1d9f04d1 0x1d9eec3c 0x1d9eeeca 0x1f7cc44c 0x1f79b992 0x1e52123f 0x1e522b8e 0x1eb7e27b 0x1eb7be44 0x1f751e26 0x1f75f0e5 0x23ba1179 0x23ba1988 0x23cef8b4 0x23cec39a 0x23ceb17c 0x7f5661c01609 0x7f5661b17293
    {} <Fatal> BaseDaemon: 4. raise @ 0x4618b in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    {} <Fatal> BaseDaemon: 5. abort @ 0x25859 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:53: DB::handle_error_code(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool, std::__1::vector<void*, std::__1::allocator<void*> > const&) @ 0x1333bcb8 in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:60: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x1333bdc2 in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Core/Block.cpp:32: void DB::onError<void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1d9f04d1 in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Core/Block.cpp:86: void DB::checkBlockStructure<void>(DB::Block const&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) @ 0x1d9eec3c in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Core/Block.cpp:607: DB::assertCompatibleHeader(DB::Block const&, DB::Block const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x1d9eeeca in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Processors/Port.cpp:19: DB::connect(DB::OutputPort&, DB::InputPort&) @ 0x1f7cc44c in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../src/Processors/Chain.cpp:81: DB::Chain::addSource(std::__1::shared_ptr<DB::IProcessor>) @ 0x1f79b992 in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterInsertQuery.cpp:222: DB::InterpreterInsertQuery::buildChainImpl(std::__1::shared_ptr<DB::IStorage> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Block const&, DB::ThreadStatus*, std::__1::atomic<unsigned long>*) @ 0x1e52123f in /usr/bin/clickhouse
    {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterInsertQuery.cpp:368: DB::InterpreterInsertQuery::execute() @ 0x1e522b8e in /usr/bin/clickhouse

</details>

  [1]: https://clickhouse-test-reports.s3.yandex.net/29461/d6cd057e2c0c8b04d8a712a7b2a83526507ef512/functional_stateless_tests_(debug).html#fail1

Fixes: ClickHouse#28582 (cc: @KochetovNicolai)
@Algunenano Algunenano mentioned this pull request Oct 1, 2021
Closed
This was referenced Oct 5, 2021
Closed
Merged
@filimonov filimonov mentioned this pull request Oct 21, 2021
Merged
@azat azat mentioned this pull request Nov 18, 2021
Merged
@azat azat mentioned this pull request Jul 2, 2022
Merged
@yej2021 yej2021 mentioned this pull request Sep 2, 2023
Open
@mgorshkov mgorshkov mentioned this pull request Sep 9, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@azat azat azat left review comments

@Algunenano Algunenano Algunenano left review comments

Assignees
No one assigned
Labels
pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@KochetovNicolai @azat @Algunenano @woshisbsg @robot-clickhouse