Fix sparse_hashed dict performance with sequential keys (wrong hash function)

[azat]

Changelog category (leave one):

• Bug Fix (user-visible misbehaviour in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix sparse_hashed dict performance with sequential keys (wrong hash function)

Detailed description / Documentation draft:
In #27152 the hash function for sparse_hash_map had been changed to
std::hash<> switch it back to DefaultHash<> (ClickHouse builtin), since
std::hash<> for numeric keys returns itself and this does not works
great with sparse_hash_map.

I've tried the example from #32480 and using some hash fixes the
performance of sparse_hashed layout.

Fixes: #32480
Fixes: #27152
Backport: 20.9+

[alexey-milovidov]

Special thanks for adding performance test! 👍

[alexey-milovidov]

@azat Performance test showed significant performance degradation: https://clickhouse-test-reports.s3.yandex.net/32536/6d31a389f12837f3b8168f17217dadab4f4f4974/performance_comparison/report.html#fail1

[alexey-milovidov]

PS. For our own HashTables, we should use crc32c.

[alexey-milovidov]

We should completely revert #27152 and remove useless files, because we don't use "Arcadia" build system anymore.

[azat]

@azat Performance test showed significant performance degradation: https://clickhouse-test-reports.s3.yandex.net/32536/6d31a389f12837f3b8168f17217dadab4f4f4974/performance_comparison/report.html#fail1

Yes, the problem is that for sequential keys DefaultHash does not works great, but it works significantly better if those sequential keys are randomly accessed/inserted, I'm going to update performance test for now (to fix the original issue) and get back to this later.

We should completely revert #27152 and remove useless files, because we don't use "Arcadia" build system anymore.

• Remove arcadia build support #32580 - BTW there is not a lot had been left

[azat]

So now perf tests shows improvement, but the query itself is too slow (there is lots of heuristics in report.py, that does not allow the query to run >~2 seconds each):

Old, s	New, s	Ratio of speedup (-) or slowdown (+)	Relative difference (new − old) / old	p < 0.01 threshold	Test	#	Query
12.281	5.111	-2.402x	-0.584	0.578	hashed_dictionary	1	SYSTEM RELOAD DICTIONARY SPARSE_HASHED_dictionary

Will rework the perf test.

[azat]

Actually is it not that easy to satisfy all the conditions (various timeouts for each query and test overall) for performance tests and write one for this issue, so let's do this separately.

Perf test had been removed for now.

[alexey-milovidov]

Could you please remove the SparseHashMap.h file?
It was added for "arcadia" and we don't use arcadia anymore.

Also need to figure out what is wrong with integration tests...

[azat]

https://s3.amazonaws.com/clickhouse-test-reports/32536/559b01a2e43d09571195fb1cc08d83133f559203/integration_tests__release__actions__[2/2].html

• test_merge_tree_blob_storage

E               compose.cli.verbose_proxy.proxy_callable: docker pull <- ('mcr.microsoft.com/azure-storage/azurite', tag='latest', stream=True, platform=None)
E               compose.cli.errors.log_api_error: Get "https://mcr.microsoft.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

https://s3.amazonaws.com/clickhouse-test-reports/32536/559b01a2e43d09571195fb1cc08d83133f559203/integration_tests__asan__actions__[2/3].html

• Test test_kafka_json_as_string_no_kdc failures #32644 - test_storage_kerberized_kafka

https://s3.amazonaws.com/clickhouse-test-reports/32536/559b01a2e43d09571195fb1cc08d83133f559203/integration_tests__asan__actions__[1/3].html

• tests/integration: fix wait_start()/start_clickhouse() #32739 - test_keeper_incorrect_config/test.py::test_duplicate_endpoint

https://s3.amazonaws.com/clickhouse-test-reports/32536/559b01a2e43d09571195fb1cc08d83133f559203/integration_tests__release__actions__[1/2].html

• tests/integration: fix waiting of mysql client container #32740 - test_mysql_protocol

[azat]

https://github.com/ClickHouse/ClickHouse/runs/4516380764?check_suite_focus=true

  Error: fatal: Unable to create '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/.git/modules/contrib/aws/index.lock': File exists.
  
  Another git process seems to be running in this repository, e.g.
  an editor opened by 'git commit'. Please make sure all processes
  are terminated then try again. If it still fails, a git process
  may have crashed in this repository earlier:
  remove the file manually to continue.
...
  Unable to checkout '00b03604543367d7e310cb0993973fdcb723ea79' in submodule path 'contrib/aws'
  Error: The process '/usr/bin/git' failed with exit code 1

Something left from the previous run in this workspace?