New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merging Values with expressions #6781
Conversation
Failed tests and known issues:
|
Please provide description in some way about the feature in this PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also suggest to have tests covering this situations:
- Reading values with changed template format in different ways, like assuming default values, changing expressions (
lower
→upper
). - Reading values with same template format, but incastable types, like enums.
- Reading broken values, that will lead to failure with standard parser.
dbms/src/Processors/Formats/Impl/ConstantExpressionTemplate.cpp
Outdated
Show resolved
Hide resolved
Reviewed all changes and discussed all the questions in person. Awaiting for some of fixes:
|
Maybe that can also be fixed by the way: #7174 |
Can be merged after performance tests are OK. |
In case you pass array in VALUES section (ValuesBlockInputFormat), it will be copied after it had been created. This is not significant if array is small, however if you have huge enough array, then this can become significant (especially for array of bool, since for each element will be used Field anyway, and it's size is 56 byte). Here is a simple reproducer: $ curl -sS 'http://127.1:8123/?input_format_values_deduce_templates_of_expressions=0' -d@- <<<"insert into function null('_ Int') values (length([0,$(yes 1, | head -n2000000 | tr -d '\n')1]))" But note, that there is lots of work (evalute constant expressions from ClickHouse#6781) after parsing collection, and so total memory usage for query does not changed a lot (hence - no test). Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
For changelog. Remove if this is non-significant change.
Category (leave one):
Short description (up to few sentences):
Merging the second part of #4354 (optimization of parsing SQL expressions in
Values
)Detailed description:
If some field in
Values
contains an SQL expression, try to deduce template of the expression and parse the following rows using this template, supposing expressions in the following rows have the same structure, e.g.It allows to parse
Values
with expressions much faster (up to 100x faster than parsing expressions separately).