Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite arithmetic in aggregate functions optimisation #11899

Merged
merged 4 commits into from
Jun 24, 2020
Merged

Rewrite arithmetic in aggregate functions optimisation #11899

merged 4 commits into from
Jun 24, 2020

Conversation

4ertus2
Copy link
Contributor

@4ertus2 4ertus2 commented Jun 23, 2020

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Rewrite code for optimize_arithmetic_operations_in_aggregate_functions optimisation.

Detailed description / Documentation draft:
It rewrites functions like sum(a * 2) into faster sum(a) * 2.
It supports now sum([multiply|divide]) -> [multiply|divide](sum) and [min|max]([multiply|divide|plus|minus]) -> [multiply|divide|plus|minus]([min|max])

@blinkov blinkov added the pr-improvement Pull request with some product improvements label Jun 23, 2020
@4ertus2 4ertus2 merged commit 2391ac2 into ClickHouse:master Jun 24, 2020
alesapin added a commit that referenced this pull request Jul 2, 2020
@abyss7 @alesapin
@akuzm @bharatnc @azat @alexey-milovidov @KochetovNicolai @sundy-li @amudong @ucasfl @blinkov @theg4sh @hexiaoting @Akazz @tavplubix @4ertus2 @Vxider @Slach
Backport #11920 to 20.5: Fix using current database while checking ac…
e0cf236 
…cess rights. (#11963)

* Simple github hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* typo

* fixup

* add flag to continue on errors

* fixup

* fixup

* Compilable setColumns, setConstraints, setIndices

* Working setColumns, setConstraints, setIndices

* Move set*Key methods to StorageInMemoryMetadata

* Working set*Keys methods

* TTL methods in StorageInMemoryMetadata

* All set methods in metadata

* Metadata in read and write methods of IStorage

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* Copy some methods to metdata

* Get column dependencies in StorageInMemoryMetadata

* Move getSampleBlockNonMaterialized to StorageInMemoryMetadata

* getSampleBlockWithVirtuals in StorageInMemoryMetadata

* Compileable getSampleBlockWithColumns in StorageInMemoryMetadata

* Seems to be working getSampleBlockWithColumns in StorageInMemoryMetadata

* Buildable getSampleBlock in StorageInMemoryMetadata

* getConstraints() in StorageInMemoryMetadata (suspicious commit, but pretend to work)

* Fix race condition

* Secondary indices in StorageInMemoryMetadata

* Partition key in StorageInMemoryMetadata

* Compilable sorting key in metadata

* Fix race condition

* Sampling key in StorageInMemoryMetadata

* Primary key in storage metadata

* Missed change for primary key

* TTL in storage in memory metadata

* Settings changes in StorageInMemoryMetadata

* Select query in metadata

* Check methods in metadata

* Better getVirtuals method

* getColumns in StorageInMemoryMetadta (only compilable)

* Fix segmentation fault

* Fix build

* Fix crash

* Fix storage merge

* Truncate with metadata

* Fix SystemTables

* Really atomic metadata

* Fix style

* Fix storage buffer metadata

* Better locks in Storages

* Less locks

* Fix style

* Remove unused method

* Remove unused method better

* Remove int contention

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Fix locks

* Add missed check

* Move partition key initialization into registerMergeTree

* Remove unused constructor and rename method

* Better naming

* Better logging in storages

* Remove unused variable

* report the number of errors

* fixup

* fixup

* Send logs on fatal errors (development)

* Set send_logs_level to fatal by default

* Update tests

* Turn off traps

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* Fix memory accounting via HTTP interface

function perf_test()
{
    time yes '127.1:8123/?wait_end_of_query=1' | head -n10000 | xargs -P10000 curl -s -d 'select 1' | grep -x -c 1
}
function server()
{
    local limit=$1 && shift
    clickhouse-server "$@" -- --max_server_memory_usage=$limit
}

perf_test:

- before this patch with 1G  limit: succeed only ~800  queries
- after  this patch with 1G  limit: succeed      ~8000 queries

- before this patch with 10G limit: succeed only ~3000  queries
- after  this patch with 10G limit: succeed      ~10000 queries

Fixes: #11153

* Add a test for memory tracking via HTTP interface

* Update executeQuery.cpp

* Make LIVE VIEW test less wrong (it was depending on the timing of progress messages)

* Do not run 01238_http_memory_tracking under sanitizers

* Tune 01238_http_memory_tracking to make it more optimal

w/o the patch the test fails with, since it succeed only 512 queries.

* Merge with master

* Ad tests with long select and simulatenous alter

* Better comments

* Fix 01213_alter_rename_column_zookeeper

* report

* Fix bad test for LIVE VIEW

* Disable MySQL tests under tsan

* Better 00980_alter_settings test

* Support sumWithOverflow as SimpleAggregateFunction

* Clang-tidy fixes

* Fix header for nullable prewhere column.

* Update tests.

* Fix 01091_num_threads flaps (by adding unique identifier to the queries)

Actually there were two issues:
- missing \n, so it picked the wrong query anway
- no unique identifier, hence it may works incorrectly (and actually,
  event clickhouse-test script executes 'SELECT 1' query)

* Initialize GlobalThreadPool explicitly

* Do not render progress too much

* Stop tests on fatal errors

* jemalloc: fix HAVE_CPU_SPINWAIT for linux aarch64

Fixes: #11163

* Style

* constexpr intExp10

* Fix bugs in DateTime64 parsing

* Added a test

* Fix wrong code

* Fix strange code

* Update tests

* jemalloc: jemalloc_preamble.h is common

* jemalloc: fix JEMALLOC_NOTHROW/JEMALLOC_CXX_THROW for freebsd/darwin

* jemalloc: add freebsd pregenerated header

* jemalloc: add darwin pregenerated header

* jemalloc: enable by default for freebsd/darwin

* jemalloc: drop superfluous include directory

* jemalloc: explicitly disable pthread_setname_np on freebsd (requires 12.1)

* Add drop replica alter support

* fix merge state

Fix style && build

* Add tests and docs

Fix integration-tests

* only drop inactive replica

update doc

Increase timeout to release the zookeeper Ephemeral nodes

Fix code comment

use PartitionManager

make integrations test passed

* add SYSTEM DROP REPLICA

* fix: can not drop local table and replace ALTER with SYSTEM

fix test_drop_replica

fix drop replica '/path/to/zk/' ending in '/' and update doc

* add drop relica database and the whole replica

fix removeReplicaByZKPath

fix bug: add zkpath empty judge

fix: rewrite code

delete useless code.

fix:ast fromat

fix bug

add test_drop_replica

add drop_replica doc

add drop databse checkAccess

refactor dropReplica

update tests

add static method StorageReplicatedMergeTree::dropReplicaByZkPath

update doc and delete useless code

fix conflict

fix doc

fix doc

fix StorageReplicatedMergeTree::dropReplica

fix bug

delete useless code

* rewrite static StorageReplicatedMergeTree::dropReplica

* format comment

* SHA256 add FixedString support

* fixup

* Update index.md

* fix of incorrect example

was `curl -sS "<address>?param_id=2¶m_phrase=test" -d "SELECT * FROM table WHERE int_column = {id:UInt8} and string_column = {phrase:String}"`

* Add missed lock

* jemalloc: disable percpu_arena on non-linux (not supported anyway)

And since it does not supported it produces the following warning:
  <jemalloc>: No getcpu support: percpu_arena:percpu
  <jemalloc>: perCPU arena getcpu() not available. Setting narenas to 8.

However it is uses narenas=ncpu instead of default narenas=ncpu*4

* Optimize code

* jemalloc: disable it for darwin

Right now it aborts (Abort trap: 6) on osx, the issue does not looks
like [1], but should be investigated, so disable for now.

What is interesting that it works under debugger, so I guess it is
indeed the order of symbol replacements...

  [1]: jemalloc/jemalloc#420

* fix bug for query "select *, abc.*" should failed. (#11753)

* Remove redundat sync replica

* test commit

* Hook for push events

* add test

fix

fix

fix

* memory usage settings

* Update README.md

* Update README.md

* Merge pull request #11071 (#11874)

* Add's hasSubstr  array function

* Add's example in the english doc of hasSubStr

* Updates conditions to avoid prematurely calling isEqual

* Remove feathericons

Authored-by: Ryad ZENINE <ryad.zenine@contentsquare.com>

* allow implicit initialization

* Fix bad code

* Allow ClickHouse to run on Android

* Fix documentation

* Add docs about non-linux binaries (#11889)

* Add docs about non-linux binaries

* Better

* Update install.md

* Update install.md

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Review fixes

* Better instruction

* Fix reference

* Better

* Better links

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Fix anotehr one nullable prewhere column.

* minor improvements

* Update StorageSystemTables.cpp

* Added a test

* fix style check - removed unused LOGICAL_ERROR

* Fix if function with NULLs (#11807)

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* sync with goals

* Remove unused code

* Remove already fixed bug

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Fix using current database while checking access rights.

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* for the love of God just update this container already, i'm begging you

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* Update features.html

* fixpu

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Alexey Milovidov <milovidov@yandex-team.ru>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: sundy-li <543950155@qq.com>
Co-authored-by: amudong <xiaoxiong5288@gmail.com>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Alexandr Kondratev <theg4sh@gmail.com>
Co-authored-by: hexiaoting <hewenting_ict@163.com>
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Vxider <lb@vxider.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: robot-clickhouse <robot-clickhouse@yandex-team.ru>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @blinkov
@akuzm @alexey-milovidov @bharatnc @CurtizJ @azat @nikvas0 @zhang2014 @KochetovNicolai @BohuTANG @Slach @nikitamikhaylov @alesapin @4ertus2 @tavplubix @zlobober @ucasfl @oandrew @ithangzhou @Yuntao-Wu @wangfenjin
Backport #12013 to 20.5: Fix array size overflow in generateRandom (#…
9025fd0 
…12249)

* change used flag

* fix address formatting

* use the sentry logger hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* fixes

* experiment

* typo

* trigger ci

* fixup

* add flag to continue on errors

* trigger ci

* trigger ci

* trigger ci

* experiment

* trigger ci

* fixup

* fixup

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update http_server.py

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* improvements after review comments

* move the default endpoint to config

* remove extra line

* distinct combinator for single numeric arguments

* rework distinct combinator

* fix build

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Update 00816_long_concurrent_alter_column.sh

* distinct combinator for function of multiuple arguments

* report the number of errors

* fixup

* fixup

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* update

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Vasilev <vasnikserg@yandex.ru>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
Co-authored-by: Wang Fenjin <wangfenj@gmail.com>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @azat
@nikvas0 @alexey-milovidov @blinkov @zhang2014 @akuzm @CurtizJ @KochetovNicolai @bharatnc @BohuTANG @Slach @nikitamikhaylov @alesapin @4ertus2 @tavplubix @zlobober @ucasfl @oandrew @ithangzhou @Yuntao-Wu @wangfenjin @mgaidamak @dependabot-preview @BayoNet @ka1bi4 @abyss7
Backport #12019 to 20.5: Fix overflow in function "neighbor" (#12251)
c1eb71d 
* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Fixes

* Add comment

* Remove empty lines

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Add __init__.py

* Remove strange header

* Trying to fix ya.make

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* Temporary enable debug logs when releasing docs

* [docs] improve stable releases discovery

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Vasilev <vasnikserg@yandex.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: bharatnc <bharatnc@gmail.com>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
Co-authored-by: Wang Fenjin <wangfenj@gmail.com>
Co-authored-by: Mikhail Gaidamaka <magnoly.nsk@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: BayoNet <da-daos@yandex.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @blinkov
@akuzm @alexey-milovidov @bharatnc @CurtizJ @azat @nikvas0 @zhang2014 @KochetovNicolai @BohuTANG @Slach @nikitamikhaylov @alesapin @4ertus2 @tavplubix @zlobober @oandrew @ithangzhou @Yuntao-Wu
Backport #12005 to 20.5: Fix potential FPE. (#12245)
ee6e307 
* try to completely remove sentry from odbc-bridge

* Switch back to sentry upstream

* Try to disable linker options from sentry

* Do not cache frames inside StackTrace

* Lost part of refactoring

* fix

* less confusing

* fix

* maybe fix the unbundled gcc build

* experiment

* experiment

* back to upstream

* change used flag

* fix address formatting

* use the sentry logger hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* fixes

* experiment

* typo

* trigger ci

* fixup

* add flag to continue on errors

* trigger ci

* trigger ci

* trigger ci

* experiment

* trigger ci

* fixup

* fixup

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update http_server.py

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* improvements after review comments

* move the default endpoint to config

* remove extra line

* distinct combinator for single numeric arguments

* rework distinct combinator

* fix build

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Update 00816_long_concurrent_alter_column.sh

* distinct combinator for function of multiuple arguments

* report the number of errors

* fixup

* fixup

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Vasilev <vasnikserg@yandex.ru>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @blinkov
@akuzm @alexey-milovidov @bharatnc @CurtizJ @azat @nikvas0 @zhang2014 @KochetovNicolai @BohuTANG @Slach @nikitamikhaylov @alesapin @4ertus2 @tavplubix @zlobober @oandrew @ithangzhou @Yuntao-Wu
Backport #11997 to 20.5: Fix bad log message at server startup (#12244)
5e01180 
* old cmake compatibility

* more build fixes

* experiment

* apply comment

* apply comment

* fix compiling when disabled

* experiment

* fix compiling when disabled

* disable for splitted

* back to upstream

* change sentry-native commit

* experiment

* partial revert

* experiment with BUILD_SHARED_LIBS

* try to fix the glibc compatibility

* try to completely remove sentry from odbc-bridge

* Switch back to sentry upstream

* Try to disable linker options from sentry

* Do not cache frames inside StackTrace

* Lost part of refactoring

* fix

* less confusing

* fix

* maybe fix the unbundled gcc build

* experiment

* experiment

* back to upstream

* change used flag

* fix address formatting

* use the sentry logger hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* fixes

* experiment

* typo

* trigger ci

* fixup

* add flag to continue on errors

* trigger ci

* trigger ci

* trigger ci

* experiment

* trigger ci

* fixup

* fixup

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update http_server.py

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* improvements after review comments

* move the default endpoint to config

* remove extra line

* distinct combinator for single numeric arguments

* rework distinct combinator

* fix build

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Update 00816_long_concurrent_alter_column.sh

* distinct combinator for function of multiuple arguments

* report the number of errors

* fixup

* fixup

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Update replxx submodule

* disable tests in arcadia

* performance comparison

* performance comparison

* Fix bad log message at server startup

* Better diagnostics of "Replica {} appears to be already active" message

Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Vasilev <vasnikserg@yandex.ru>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @alexey-milovidov
@CurtizJ @BohuTANG @KochetovNicolai @blinkov @Slach @akuzm @nikitamikhaylov @alesapin @bharatnc @azat @4ertus2 @tavplubix @zlobober @ucasfl @oandrew @ithangzhou @Yuntao-Wu @wangfenjin @mgaidamak @dependabot-preview @BayoNet @ka1bi4 @abyss7 @chengy8934 @MicrochipQ @Akazz
Backport #12043 to 20.5: Make topK return Enum for Enum types (#12232)
0806c35 
* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Fixes

* Add comment

* Remove empty lines

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Add __init__.py

* Remove strange header

* Trying to fix ya.make

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* Try to enable long perf tests

* performance comparison

* longer

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* make a short test for DateTime

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix test.

* Update ITransformingStep.h

* Make topK return Enum for Enum types

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
Co-authored-by: Wang Fenjin <wangfenj@gmail.com>
Co-authored-by: Mikhail Gaidamaka <magnoly.nsk@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>
Co-authored-by: chengy8934 <67622393+chengy8934@users.noreply.github.com>
Co-authored-by: MicrochipQ <microchipq@gmail.com>
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
alexey-milovidov added a commit that referenced this pull request Jul 7, 2020
@robot-clickhouse @alexey-milovidov
@CurtizJ @BohuTANG @KochetovNicolai @blinkov @Slach @akuzm @nikitamikhaylov @alesapin @bharatnc @azat @4ertus2 @tavplubix @zlobober @ucasfl @oandrew @ithangzhou @Yuntao-Wu @wangfenjin @mgaidamak @dependabot-preview @BayoNet @ka1bi4 @abyss7 @chengy8934 @MicrochipQ @Akazz
Backport #12020 to 20.5: Fix ORDER BY tuple with COLLATE on const col…
64b062e 
…umn (#12272)

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Fixes

* Add comment

* Remove empty lines

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Add __init__.py

* Remove strange header

* Trying to fix ya.make

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* Try to enable long perf tests

* performance comparison

* longer

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* make a short test for DateTime

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix test.

* Update ITransformingStep.h

* Make topK return Enum for Enum types

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Cherry pick #12020 to 20.5: Fix ORDER BY tuple with COLLATE on const column (#12203)

* Fix ORDER BY tuple with COLLATE on const column

* Remove test that is not supported by "Arcadia" build system

Co-authored-by: Alexey Milovidov <milovidov@yandex-team.ru>

Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
Co-authored-by: Wang Fenjin <wangfenj@gmail.com>
Co-authored-by: Mikhail Gaidamaka <magnoly.nsk@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>
Co-authored-by: chengy8934 <67622393+chengy8934@users.noreply.github.com>
Co-authored-by: MicrochipQ <microchipq@gmail.com>
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
@ucasfl ucasfl mentioned this pull request Feb 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
no-docs-needed pr-improvement Pull request with some product improvements
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@4ertus2 @blinkov @qoega