Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arraySymmetricDifference function #64414

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Conversation

pheepa
Copy link

@pheepa pheepa commented May 26, 2024

This PR re-does #62262.

Fixes #61673

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added function arraySymmetricDifference

Documentation entry for user-facing changes

  • Documentation is written

@pheepa

This comment was marked as resolved.

@nikitamikhaylov nikitamikhaylov added the can be tested Allows running workflows for external contributors label May 26, 2024
@rschu1ze rschu1ze self-assigned this May 26, 2024
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-feature Pull request with new product feature label May 26, 2024
@robot-clickhouse-ci-2
Copy link
Contributor

robot-clickhouse-ci-2 commented May 26, 2024

This is an automated comment for commit 87b24ff with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check nameDescriptionStatus
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR❌ failure
Performance ComparisonMeasure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests❌ failure
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc❌ failure
Successful checks
Check nameDescriptionStatus
AST fuzzerRuns randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help✅ success
ClickBenchRuns [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table✅ success
ClickHouse build checkBuilds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process✅ success
Cloud fork sync (only for ClickHouse Inc. employees)If it fails, ask a maintainer for help✅ success
Compatibility checkChecks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help✅ success
Docker keeper imageThe check to build and optionally push the mentioned image to docker hub✅ success
Docker server imageThe check to build and optionally push the mentioned image to docker hub✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Install packagesChecks that the built packages are installable in a clear environment✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
PR CheckChecks correctness of the PR's body✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stress testRuns stateless functional tests concurrently from several clients to detect concurrency-related errors✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success
Upgrade checkRuns stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts✅ success

@rschu1ze
Copy link
Member

@pheepa It is only a stylistic change but in 7288c9f (#64414) I made the formatting consistent with the rest of the codebase, with d86b84a it is now back to the previous formatting. Would you like to revert that last commit? Thanks!

@pheepa
Copy link
Author

pheepa commented May 29, 2024

@rschu1ze sorry, didn't notice
But style check for 7288c9f failed
d86b84a - just formating with clang-format

what do you suppose to do? the style check will not pass again if we revert this commit.

@rschu1ze
Copy link
Member

@pheepa The AST fuzzer fails. Simplifying the fuzzed query gives this query:

select arraySymmetricDifference([CAST('str_', 'Dynamic(max_types = 2)'), 71], [1, 2]) settings allow_experimental_dynamic_type = 1

(crash pasted below)

"Dynamic" is the new dynamic and experimental data type in ClickHouse.

As far as I see, arraySymmetricDifference calls getLeastSupertype to determine its return type. getLeastSuperType then returns Dynamic (I think, didn't debug it).

During execution,ColumnDynamic::compareAt is called which expects that two ColumnDynamic-s are compared. This is reasonable and other column types make similar assumptions.

I guess what we need to do is to wrap the non-dynamic column as dynamic during execution.
But perhaps it would be easier to simply reject dynamic columns as input types. This could be checked in getReturnTypeImpl

The crash is because compareAt of

[ip-172-31-8-55] 2024.05.29 12:31:37.900136 [ 918008 ] {5ec085be-c25c-471b-9869-df8bb8b8f33d} <Fatal> : Logical error: 'Bad cast from type DB::ColumnVector<char8_t> to DB::ColumnDynamic'.
[ip-172-31-8-55] 2024.05.29 12:31:37.900735 [ 919437 ] <Fatal> BaseDaemon: ########################################
[ip-172-31-8-55] 2024.05.29 12:31:37.900816 [ 919437 ] <Fatal> BaseDaemon: (version 24.6.1.1, build id: 386186AB4879B18B42C6F748A2A0EEA9BEFE2548, git hash: d86b84a059c699020c2a720f7ae7bd78fcadb8f1) (from thread 918008) (query_id: 5ec085be-c25c-471b-9869-df8bb8b8f33d) (query: select arraySymmetricDifference([CAST('str_', 'Dynamic(max_types = 2)'), 71], [1, 2]) settings allow_experimental_dynamic_type = 1) Received signal Aborted (6)
[ip-172-31-8-55] 2024.05.29 12:31:37.900866 [ 919437 ] <Fatal> BaseDaemon:
[ip-172-31-8-55] 2024.05.29 12:31:37.900911 [ 919437 ] <Fatal> BaseDaemon: Stack trace: 0x000000000c5a9caa 0x000000000c5a9ba0 0x000000000c980ad1 0x0000785d7c245320 0x0000785d7c29eb1d 0x0000785d7c24526e 0x0000785d7c2288ff 0x000000000c556a62 0x000000000c556bd5 0x000000000c557180 0x0000000004dd206a 0x0000000004dcf869 0x0000000004dd506d 0x00000000122e5921 0x0000000014bacdc0 0x0000000009fa9eb7 0x0000000009edb2e3 0x0000000009edb78d 0x0000000009ec4e70 0x0000000005a3c06d 0x00000
00011daa7fa 0x0000000011dab8c6 0x0000000011dac835 0x0000000004e29a52 0x000000000bf382da 0x000000000bf3b23c 0x000000000bf3943b 0x0000000004dced59 0x0000000005a3c0cd 0x0000000011daa7a6 0x0000000011daa323 0x0000000011daa683 0x0000000011dab8c6 0x0000000011dac835 0x00000000134ebb30 0x00000000134d1172 0x00000000134cf6aa 0x00000000134d9027 0x00000000134cafc6 0x00000000134c9194 0x00000000134c8e53 0x00000000134b9906 0x00000000142241ac 0x0000000014222dce 0x0000000014225f37
[ip-172-31-8-55] 2024.05.29 12:31:37.914129 [ 919437 ] <Fatal> BaseDaemon: 0. /data/ch1/src/Common/StackTrace.cpp:349: StackTrace::tryCapture() @ 0x000000000c5a9caa
[ip-172-31-8-55] 2024.05.29 12:31:37.926249 [ 919437 ] <Fatal> BaseDaemon: 1. /data/ch1/src/Common/StackTrace.cpp:323: StackTrace::StackTrace(ucontext_t const&) @ 0x000000000c5a9ba0
[ip-172-31-8-55] 2024.05.29 12:31:37.950106 [ 919437 ] <Fatal> BaseDaemon: 2. /data/ch1/src/Daemon/BaseDaemon.cpp:157: signalHandler(int, siginfo_t*, void*) @ 0x000000000c980ad1
[ip-172-31-8-55] 2024.05.29 12:31:37.950157 [ 919437 ] <Fatal> BaseDaemon: 3. ? @ 0x0000785d7c245320
[ip-172-31-8-55] 2024.05.29 12:31:37.950204 [ 919437 ] <Fatal> BaseDaemon: 4. ? @ 0x0000785d7c29eb1d
[ip-172-31-8-55] 2024.05.29 12:31:37.950250 [ 919437 ] <Fatal> BaseDaemon: 5. ? @ 0x0000785d7c24526e
[ip-172-31-8-55] 2024.05.29 12:31:37.950294 [ 919437 ] <Fatal> BaseDaemon: 6. ? @ 0x0000785d7c2288ff
[ip-172-31-8-55] 2024.05.29 12:31:37.965032 [ 919437 ] <Fatal> BaseDaemon: 7. /data/ch1/src/Common/Exception.cpp:0: DB::abortOnFailedAssertion(String const&) @ 0x000000000c556a62
[ip-172-31-8-55] 2024.05.29 12:31:37.979413 [ 919437 ] <Fatal> BaseDaemon: 8. /data/ch1/src/Common/Exception.cpp:65: DB::handle_error_code(String const&, int, bool, std::vector<void*, std::allocator<void*>> const&) @ 0x000000000c556bd5
[ip-172-31-8-55] 2024.05.29 12:31:37.992157 [ 919437 ] <Fatal> BaseDaemon: 9. /data/ch1/src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c557180
[ip-172-31-8-55] 2024.05.29 12:31:38.003268 [ 919437 ] <Fatal> BaseDaemon: 10. /data/ch1/src/Common/Exception.h:95: DB::Exception::Exception(String&&, int, bool) @ 0x0000000004dd206a
[ip-172-31-8-55] 2024.05.29 12:31:38.014167 [ 919437 ] <Fatal> BaseDaemon: 11. /data/ch1/src/Common/Exception.h:68: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000004dcf869
[ip-172-31-8-55] 2024.05.29 12:31:38.024730 [ 919437 ] <Fatal> BaseDaemon: 12. /data/ch1/src/Common/Exception.h:113: DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000004dd506d
[ip-172-31-8-55] 2024.05.29 12:31:38.161140 [ 919437 ] <Fatal> BaseDaemon: 13. /data/ch1/src/Common/assert_cast.h:47: DB::ColumnDynamic const& assert_cast<DB::ColumnDynamic const&, DB::IColumn const&>(DB::IColumn const&) @ 0x00000000122e5921
[ip-172-31-8-55] 2024.05.29 12:31:38.187370 [ 919437 ] <Fatal> BaseDaemon: 14. /data/ch1/src/Columns/ColumnDynamic.cpp:593: DB::ColumnDynamic::compareAt(unsigned long, unsigned long, DB::IColumn const&, int) const @ 0x0000000014bacdc0
[ip-172-31-8-55] 2024.05.29 12:31:38.246957 [ 919437 ] <Fatal> BaseDaemon: 15. /data/ch1/src/Functions/FunctionsComparison.h:556: DB::GenericComparisonImpl<DB::NotEqualsOp<int, int>>::vectorVector(DB::IColumn const&, DB::IColumn const&, DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&) @ 0x0000000009fa9eb7
[ip-172-31-8-55] 2024.05.29 12:31:38.291169 [ 919437 ] <Fatal> BaseDaemon: 16. /data/ch1/src/Functions/FunctionsComparison.h:1136: DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeGenericIdenticalTypes(DB::IColumn const*, DB::IColumn const*) const @ 0x0000000009edb2e3
[ip-172-31-8-55] 2024.05.29 12:31:38.336096 [ 919437 ] <Fatal> BaseDaemon: 17. /data/ch1/src/Functions/FunctionsComparison.h:1149: DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeGeneric(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const @ 0x0000000009edb78d
[ip-172-31-8-55] 2024.05.29 12:31:38.379755 [ 919437 ] <Fatal> BaseDaemon: 18. /data/ch1/src/Functions/FunctionsComparison.h:1387: DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000009ec4e70
[ip-172-31-8-55] 2024.05.29 12:31:38.444235 [ 919437 ] <Fatal> BaseDaemon: 19. /data/ch1/src/Functions/IFunctionAdaptors.h:21: DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000005a3c06d
[ip-172-31-8-55] 2024.05.29 12:31:38.455407 [ 919437 ] <Fatal> BaseDaemon: 20. /data/ch1/src/Functions/IFunction.cpp:248: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011daa7fa
[ip-172-31-8-55] 2024.05.29 12:31:38.466447 [ 919437 ] <Fatal> BaseDaemon: 21. /data/ch1/src/Functions/IFunction.cpp:303: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011dab8c6
[ip-172-31-8-55] 2024.05.29 12:31:38.477239 [ 919437 ] <Fatal> BaseDaemon: 22. /data/ch1/src/Functions/IFunction.cpp:378: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011dac835
[ip-172-31-8-55] 2024.05.29 12:31:38.493791 [ 919437 ] <Fatal> BaseDaemon: 23. /data/ch1/src/Functions/IFunction.h:138: DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000004e29a52
[ip-172-31-8-55] 2024.05.29 12:31:38.543321 [ 919437 ] <Fatal> BaseDaemon: 24. /data/ch1/src/Functions/array/arrayLogicalFunctionBase.h:255: DB::callFunctionNotEquals(DB::ColumnWithTypeAndName, DB::ColumnWithTypeAndName, std::shared_ptr<DB::Context const>) @ 0x000000000bf382da
[ip-172-31-8-55] 2024.05.29 12:31:38.580780 [ 919437 ] <Fatal> BaseDaemon: 25. /data/ch1/src/Functions/array/arrayLogicalFunctionBase.h:312: DB::FunctionArrayLogicalBase<false>::prepareArrays(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&) const @ 0x000000000bf3b23c
[ip-172-31-8-55] 2024.05.29 12:31:38.617846 [ 919437 ] <Fatal> BaseDaemon: 26. /data/ch1/src/Functions/array/arrayLogicalFunctionBase.h:642: DB::FunctionArrayLogicalBase<false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000bf3943b
[ip-172-31-8-55] 2024.05.29 12:31:38.628040 [ 919437 ] <Fatal> BaseDaemon: 27. /data/ch1/src/Functions/IFunction.h:432: DB::IFunction::executeImplDryRun(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000004dced59
[ip-172-31-8-55] 2024.05.29 12:31:38.692340 [ 919437 ] <Fatal> BaseDaemon: 28. /data/ch1/src/Functions/IFunctionAdaptors.h:26: DB::FunctionToExecutableFunctionAdaptor::executeDryRunImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000005a3c0cd

@pheepa
Copy link
Author

pheepa commented May 29, 2024

Thank you a lot!
At the time not so easy to find main error in a row of checks
I will see what I can do

What about code style in 7288c9f, what commit will we keep?

@rschu1ze
Copy link
Member

Yeah, the CI makes it not too easy to find an error, and usually one needs to checkout the change, build it, and simplify issue ...

@rschu1ze
Copy link
Member

Regarding formatting: Let's go the path of least resistance and go with your reformatting commit.

@rschu1ze
Copy link
Member

@pheepa AST Fuzzer failed unfortunately once more (here).

The simplified, offending query is: SELECT arraySymmetricDifference([2], [toDecimal64(0., 9)])

2024.05.30 08:59:55.068521 [ 1395318 ] {9272f477-d56b-4931-99cf-75674c6b4c9c} <Fatal> : Logical error: 'Wrong column in Decimal comparison'.
2024.05.30 08:59:55.068982 [ 1395310 ] {} <Trace> BaseDaemon: Received signal 6
2024.05.30 08:59:55.069254 [ 1396536 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
2024.05.30 08:59:55.069325 [ 1396536 ] {} <Fatal> BaseDaemon: (version 24.6.1.1, build id: 8811E043C28CCB708A2DE1D1D018C89ECF859104, git hash: 90f3cef919f8328c9f005f7cb87261e9117e5325) (from thread 1395318) Received signal 6
2024.05.30 08:59:55.069363 [ 1396536 ] {} <Fatal> BaseDaemon: Signal description: Aborted
2024.05.30 08:59:55.069390 [ 1396536 ] {} <Fatal> BaseDaemon:
2024.05.30 08:59:55.069435 [ 1396536 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c5b4a2a 0x000000000c5b4920 0x000000000c98b071 0x000072573ee45320 0x000072573ee9eb1d 0x000072573ee4526e 0x000072573ee288ff 0x000000000c5617e2 0x000000000c561955 0x000000000c561f00 0x0000000004dd506a 0x0000000004dd2869 0x0000000004dda51a 0x0000000009fbb9e5 0x0000000009fbb19d 0x0000000009fbaf33 0x0000000009fb5e7b 0x0000000009ec9a00 0x0000000009ec958f 0x0000000009ee6976 0x0000000009ecfb39 0x0000000005a3f06d 0x0000000011db1c3a 0x0000000011db2d06 0x0000000011
db3c75 0x0000000004e2ca52 0x000000000bf42eda 0x000000000bf45fbc 0x000000000bf4403b 0x0000000004dd1d59 0x0000000005a3f0cd 0x0000000011db1be6 0x0000000011db1763 0x0000000011db1ac3 0x0000000011db2d06 0x0000000011db3c75 0x00000000134ed930 0x00000000134d8991 0x00000000134d6d78 0x0000000013502aa7 0x00000000134d24a6 0x00000000134d0d54 0x00000000134d08b3 0x00000000134c1366 0x000000001422d1ec
2024.05.30 08:59:55.069468 [ 1396536 ] {} <Fatal> BaseDaemon: ########################################
2024.05.30 08:59:55.069582 [ 1396536 ] {} <Fatal> BaseDaemon: (version 24.6.1.1, build id: 8811E043C28CCB708A2DE1D1D018C89ECF859104, git hash: 90f3cef919f8328c9f005f7cb87261e9117e5325) (from thread 1395318) (query_id: 9272f477-d56b-4931-99cf-75674c6b4c9c) (query: SELECT arraySymmetricDifference([2], [toDecimal64(0., 9)])) Received signal Aborted (6)
2024.05.30 08:59:55.069657 [ 1396536 ] {} <Fatal> BaseDaemon:
2024.05.30 08:59:55.069724 [ 1396536 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c5b4a2a 0x000000000c5b4920 0x000000000c98b071 0x000072573ee45320 0x000072573ee9eb1d 0x000072573ee4526e 0x000072573ee288ff 0x000000000c5617e2 0x000000000c561955 0x000000000c561f00 0x0000000004dd506a 0x0000000004dd2869 0x0000000004dda51a 0x0000000009fbb9e5 0x0000000009fbb19d 0x0000000009fbaf33 0x0000000009fb5e7b 0x0000000009ec9a00 0x0000000009ec958f 0x0000000009ee6976 0x0000000009ecfb39 0x0000000005a3f06d 0x0000000011db1c3a 0x0000000011db2d06 0x0000000011
db3c75 0x0000000004e2ca52 0x000000000bf42eda 0x000000000bf45fbc 0x000000000bf4403b 0x0000000004dd1d59 0x0000000005a3f0cd 0x0000000011db1be6 0x0000000011db1763 0x0000000011db1ac3 0x0000000011db2d06 0x0000000011db3c75 0x00000000134ed930 0x00000000134d8991 0x00000000134d6d78 0x0000000013502aa7 0x00000000134d24a6 0x00000000134d0d54 0x00000000134d08b3 0x00000000134c1366 0x000000001422d1ec
2024.05.30 08:59:55.094745 [ 1396536 ] {} <Fatal> BaseDaemon: 0. /data/ch4/src/Common/StackTrace.cpp:349: StackTrace::tryCapture() @ 0x000000000c5b4a2a
2024.05.30 08:59:55.117245 [ 1396536 ] {} <Fatal> BaseDaemon: 1. /data/ch4/src/Common/StackTrace.cpp:323: StackTrace::StackTrace(ucontext_t const&) @ 0x000000000c5b4920
2024.05.30 08:59:55.158483 [ 1396536 ] {} <Fatal> BaseDaemon: 2. /data/ch4/src/Daemon/BaseDaemon.cpp:157: signalHandler(int, siginfo_t*, void*) @ 0x000000000c98b071
2024.05.30 08:59:55.158621 [ 1396536 ] {} <Fatal> BaseDaemon: 3. ? @ 0x000072573ee45320
2024.05.30 08:59:55.158712 [ 1396536 ] {} <Fatal> BaseDaemon: 4. ? @ 0x000072573ee9eb1d
2024.05.30 08:59:55.158798 [ 1396536 ] {} <Fatal> BaseDaemon: 5. ? @ 0x000072573ee4526e
2024.05.30 08:59:55.158872 [ 1396536 ] {} <Fatal> BaseDaemon: 6. ? @ 0x000072573ee288ff
2024.05.30 08:59:55.186178 [ 1396536 ] {} <Fatal> BaseDaemon: 7. /data/ch4/src/Common/Exception.cpp:0: DB::abortOnFailedAssertion(String const&) @ 0x000000000c5617e2
2024.05.30 08:59:55.213139 [ 1396536 ] {} <Fatal> BaseDaemon: 8. /data/ch4/src/Common/Exception.cpp:65: DB::handle_error_code(String const&, int, bool, std::vector<void*, std::allocator<void*>> const&) @ 0x000000000c561955
2024.05.30 08:59:55.237292 [ 1396536 ] {} <Fatal> BaseDaemon: 9. /data/ch4/src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c561f00
2024.05.30 08:59:55.256046 [ 1396536 ] {} <Fatal> BaseDaemon: 10. /data/ch4/src/Common/Exception.h:95: DB::Exception::Exception(String&&, int, bool) @ 0x0000000004dd506a
2024.05.30 08:59:55.274628 [ 1396536 ] {} <Fatal> BaseDaemon: 11. /data/ch4/src/Common/Exception.h:68: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000004dd2869
2024.05.30 08:59:55.293842 [ 1396536 ] {} <Fatal> BaseDaemon: 12. /data/ch4/src/Common/Exception.h:113: DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000004dda51a
2024.05.30 08:59:55.403695 [ 1396536 ] {} <Fatal> BaseDaemon: 13. /data/ch4/src/Core/DecimalComparison.h:213: COW<DB::IColumn>::immutable_ptr<DB::IColumn> DB::DecimalComparison<char8_t, DB::Decimal<long>, DB::NotEqualsOp, true, true>::apply<true, false>(COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, long) @ 0x0000000009fbb9e5
2024.05.30 08:59:55.512125 [ 1396536 ] {} <Fatal> BaseDaemon: 14. /data/ch4/src/Core/DecimalComparison.h:111: auto DB::DecimalComparison<char8_t, DB::Decimal<long>, DB::NotEqualsOp, true, true>::applyWithScale<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>(COW<DB::IColumn>::immutable_ptr<DB::IColumn>, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, DB::DecimalComparison<char8_t, DB::Decimal<long>, DB::NotEqualsOp, true, true>::Shift const&) @ 0x0000000009fbb19d
2024.05.30 08:59:55.620573 [ 1396536 ] {} <Fatal> BaseDaemon: 15. /data/ch4/src/Core/DecimalComparison.h:75: DB::DecimalComparison<char8_t, DB::Decimal<long>, DB::NotEqualsOp, true, true>::apply(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) @ 0x0000000009fbaf33
2024.05.30 08:59:55.728882 [ 1396536 ] {} <Fatal> BaseDaemon: 16. /data/ch4/src/Functions/FunctionsComparison.h:768: bool DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const::'lambda'(auto const&)::operator()<DB::TypePair<char8_t, DB::Decimal<long>>>(auto const&) const @ 0x0000000009fb5e7b
2024.05.30 08:59:55.857100 [ 1395886 ] {} <Trace> CgroupsMemoryUsageObserver: Read current memory usage 151.20 MiB from cgroups
2024.05.30 08:59:55.864484 [ 1395341 ] {} <Debug> DNSResolver: Updating DNS cache
2024.05.30 08:59:55.864576 [ 1395341 ] {} <Debug> DNSResolver: Updated DNS cache
2024.05.30 08:59:55.890166 [ 1396536 ] {} <Fatal> BaseDaemon: 17. /data/ch4/src/Core/callOnTypeIndex.h:53: bool DB::callOnBasicType<char8_t, true, false, true, true, DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const::'lambda'(auto const&)&>(DB::TypeIndex, DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const::'lambda'(auto const&)&) @ 0x0000000009
ec9a00
2024.05.30 08:59:56.000206 [ 1395956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 250.87 MiB, peak 250.88 MiB, free memory in arenas 0.00 B, will set to 343.50 MiB (RSS), difference: 92.63 MiB
2024.05.30 08:59:56.050443 [ 1396536 ] {} <Fatal> BaseDaemon: 18. /data/ch4/src/Core/callOnTypeIndex.h:96: bool DB::callOnBasicTypes<true, false, true, true, DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const::'lambda'(auto const&)&>(DB::TypeIndex, DB::TypeIndex, DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const::'lambda'(auto const&)&) @ 0x000
0000009ec958f
2024.05.30 08:59:56.131660 [ 1396536 ] {} <Fatal> BaseDaemon: 19. /data/ch4/src/Functions/FunctionsComparison.h:773: DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeDecimal(DB::ColumnWithTypeAndName const&, DB::ColumnWithTypeAndName const&) const @ 0x0000000009ee6976
2024.05.30 08:59:56.207112 [ 1396536 ] {} <Fatal> BaseDaemon: 20. /data/ch4/src/Functions/FunctionsComparison.h:1365: DB::FunctionComparison<DB::NotEqualsOp, DB::NameNotEquals>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000009ecfb39
^C2024.05.30 08:59:56.308115 [ 1395310 ] {} <Trace> BaseDaemon: Received signal 2
2024.05.30 08:59:56.308196 [ 1395310 ] {} <Information> Application: Received termination signal (Interrupt)
2024.05.30 08:59:56.308265 [ 1395302 ] {} <Debug> Application: Received termination signal.
2024.05.30 08:59:56.308433 [ 1395302 ] {} <Debug> Application: Waiting for current connections to close.
2024.05.30 08:59:56.326502 [ 1396536 ] {} <Fatal> BaseDaemon: 21. /data/ch4/src/Functions/IFunctionAdaptors.h:21: DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000005a3f06d
2024.05.30 08:59:56.346620 [ 1396536 ] {} <Fatal> BaseDaemon: 22. /data/ch4/src/Functions/IFunction.cpp:248: DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011db1c3a
2024.05.30 08:59:56.366039 [ 1396536 ] {} <Fatal> BaseDaemon: 23. /data/ch4/src/Functions/IFunction.cpp:303: DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011db2d06
2024.05.30 08:59:56.385597 [ 1396536 ] {} <Fatal> BaseDaemon: 24. /data/ch4/src/Functions/IFunction.cpp:378: DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000011db3c75
2024.05.30 08:59:56.415305 [ 1396536 ] {} <Fatal> BaseDaemon: 25. /data/ch4/src/Functions/IFunction.h:138: DB::IFunctionBase::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x0000000004e2ca52
2024.05.30 08:59:56.501583 [ 1396536 ] {} <Fatal> BaseDaemon: 26. /data/ch4/src/Functions/array/arrayLogicalFunctionBase.h:256: DB::callFunctionNotEquals(DB::ColumnWithTypeAndName, DB::ColumnWithTypeAndName, std::shared_ptr<DB::Context const>) @ 0x000000000bf42eda
2024.05.30 08:59:56.569940 [ 1396536 ] {} <Fatal> BaseDaemon: 27. /data/ch4/src/Functions/array/arrayLogicalFunctionBase.h:313: DB::FunctionArrayLogicalBase<false>::prepareArrays(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>>&) const @ 0x000000000bf45fbc
2024.05.30 08:59:56.637795 [ 1396536 ] {} <Fatal> BaseDaemon: 28. /data/ch4/src/Functions/array/arrayLogicalFunctionBase.h:651: DB::FunctionArrayLogicalBase<false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000000bf4403b

Interestingly, the function parameters for arrayIntersect fail gracefully with an exception:

SELECT arrayIntersect([2], [toDecimal64(0., 9)]);

-- Code: 386. DB::Exception: Received from localhost:9000. DB::Exception: There is no subtype for types UInt8, Decimal(18, 9) because some of them are numbers and some of them are not: In scope SELECT arrayIntersect([2], [toDecimal64(0., 9)]). (NO_COMMON_TYPE)

Perhaps we also need to disallow Decimal inputs too? (did not check in detail)

Another question: is the input data converted to the return type before comparing it?

On a related note: It would be great if you could add the crashing (simplified) queries (this one and the previous one) as test cases to 03033_arraySymmetricDifference.sql.

@pheepa
Copy link
Author

pheepa commented May 30, 2024

@rschu1ze, Thanks
One question: where did you find full log that you attached? I see only one line here and cant find other lines in main.log and run.log

I think Decimal should be supported. In this case we can cast UInt8 to Decimal(18, 9), so the result type will be Decimal(18, 9)

is the input data converted to the return type before comparing it?

Yes, here

I will add these queries to tests, yes

@rschu1ze
Copy link
Member

@pheepa Test errors are usually hidden in log files. In this case, go here, download the *.zst files, unpack them, and grep for Wrong column in Decimal comparison.

@rschu1ze
Copy link
Member

@pheepa It unfortunately looks like FastTest ran into a problem: https://s3.amazonaws.com/clickhouse-test-reports/64414/c4a0be7df4030048c58e31720f442c2f61b2141f/fast_test.html

@rschu1ze
Copy link
Member

rschu1ze commented Jun 21, 2024

@pheepa Functional tests are good now (thanks). The problem is that the runtime of existing function arrayIntersect blew up by magnitudes. Query select arrayIntersect(range((1 + number % 100)), range(1, (1 + number % 100) + 1)) from numbers(10000) format Null (as found in tests/performance/functions_with_hash_tables.xml, with just 10k rows) takes > 20 sec for me, previously it was almost instant.

It would be great if you could check that too! (use perf top to get an impression about the hotspot)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
can be tested Allows running workflows for external contributors pr-feature Pull request with new product feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement arraySymmetricDifference function
4 participants