New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize array element function when input is array(map)/array(array(num)/array(array(string))/array(bigint)/array(decimal) #56403
Conversation
This is an automated comment for commit 231de4a with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page Successful checks
|
Some tests Optimize when input type is array(array(num))Q: Before:
After
Optimize when input type is array(array(string))Q: Before:
After:
Optimize when input type is map with its key/value type string/numQ: Before:
After:
|
5ec4cbd
to
ff60ac4
Compare
@alexey-milovidov can you help review it, thanks very much! |
@Avogar can you help review it, thanks very much ! |
It looks very promising; I didn't have time to review it yet. |
…string))/array(array(number))
c4ec765
to
4bbc0db
Compare
It doesn't matter. I'll wait for you.. |
Do we have any progress recently ? |
1 similar comment
Do we have any progress recently ? |
I will review this PR this week. Sorry for long wait. |
Actually, I think we can generalize new implementations a bit to support any nested arrays, not only 2 level arrays. We will just need to use a vector of nested and result offsets. Not sure if there will be an overhead for 2 level arrays because of iterating through vector with single element. WDYT? Also need to make new perf tests faster, they are too slow: https://s3.amazonaws.com/clickhouse-test-reports/56403/4bbc0db1540bae020fecc1f1d8ff3bcc4eb191c2/performance_comparison_[1_4]/report.html |
I'll commit again, and refersh the perf tests(current result seems not precise, similar situations happended in other prs before). You thoughts about support any nested arrsys is interesting, but I'm consused about how to implement it, do you have any example. |
I started to write an explanation, but it turned out to be difficult to explain without just writing this code :) |
Perf test still complains about |
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
@Avogar I don't known why there are always two errors about |
Error in |
I tried to implement my idea for any nested arrays, and it's not worth it, almost no improvement. |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Optimize array element function when input is array(map)/array(array(num)/array(array(string))/array(bigint)/array(decimal). Current implementation causes too many reallocs. The optimization speed up by ~6x especially when input type is array(map)