Backport #12019 to 20.5: Fix overflow in function "neighbor" (#12251)

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <bloodjazman@gmail.com>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Fixes

* Add comment

* Remove empty lines

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“， and fix an error.

* Add __init__.py

* Remove strange header

* Trying to fix ya.make

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* Temporary enable debug logs when releasing docs

* [docs] improve stable releases discovery

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Vasilev <vasnikserg@yandex.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: bharatnc <bharatnc@gmail.com>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Eugene Klimov <bloodjazman@gmail.com>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
Co-authored-by: Maxim Akhmedov <max42@yandex-team.ru>
Co-authored-by: feng lv <fenglv15@mails.ucas.ac.cn>
Co-authored-by: Nikita Mikhailov <jakalletti@jakalletti-build.sas.yp-c.yandex.net>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Tom Bombadil <565258751@qq.com>
Co-authored-by: Yuntao Wu <wuyuntao2015@163.com>
Co-authored-by: Wang Fenjin <wangfenj@gmail.com>
Co-authored-by: Mikhail Gaidamaka <magnoly.nsk@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: BayoNet <da-daos@yandex.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>

src/Functions/neighbor.cpp

@@ -12,6 +12,7 @@ namespace ErrorCodes
 {
     extern const int ILLEGAL_TYPE_OF_ARGUMENT;
     extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+    extern const int ARGUMENT_OUT_OF_BOUND;
 }
 
 // Implements function, giving value for column within range of given
@@ -111,6 +112,10 @@ class FunctionNeighbor : public IFunction
 
             Int64 offset = offset_column->getInt(0);
 
+            /// Protection from possible overflow.
+            if (unlikely(offset > (1 << 30) || offset < -(1 << 30)))
+                throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Too large offset: {} in function {}", offset, getName());
+
             auto result_column = result_type->createColumn();
 
             auto insert_range_from = [&](bool is_const, const ColumnPtr & src, Int64 begin, Int64 size)
@@ -145,7 +150,9 @@ class FunctionNeighbor : public IFunction
             if (offset == 0)
             {
                 /// Degenerate case, just copy source column as is.
-                block.getByPosition(result).column = source_is_constant ? ColumnConst::create(source_column_casted, input_rows_count) : source_column_casted;
+                block.getByPosition(result).column = source_is_constant
+                    ? ColumnConst::create(source_column_casted, input_rows_count)
+                    : source_column_casted;
             }
             else if (offset > 0)
             {
@@ -166,7 +173,13 @@ class FunctionNeighbor : public IFunction
 
             for (size_t row = 0; row < input_rows_count; ++row)
             {
-                Int64 src_idx = row + offset_column->getInt(offset_is_constant ? 0 : row);
+                Int64 offset = offset_column->getInt(offset_is_constant ? 0 : row);
+
+                /// Protection from possible overflow.
+                if (unlikely(offset > (1 << 30) || offset < -(1 << 30)))
+                    throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Too large offset: {} in function {}", offset, getName());
+
+                Int64 src_idx = row + offset;
 
                 if (src_idx >= 0 && src_idx < Int64(input_rows_count))
                     result_column->insertFrom(*source_column_casted, source_is_constant ? 0 : src_idx);

tests/queries/0_stateless/01353_neighbor_overflow.sql

@@ -0,0 +1,2 @@
+SELECT neighbor(toString(number), -9223372036854775808) FROM numbers(100); -- { serverError 69 }
+WITH neighbor(toString(number), toInt64(rand64())) AS x SELECT * FROM system.numbers WHERE NOT ignore(x); -- { serverError 69 }