Merge branch 'master' into fix_inconsistent_parts_after_clone_replica

.github/label-pr.yml

@@ -0,0 +1,2 @@
+- regExp: ".*\\.md$"
+  labels: ["documentation", "pr-documentation"]

.github/main.workflow

@@ -0,0 +1,9 @@
+workflow "Main workflow" {
+  resolves = ["Label PR"]
+  on = "pull_request"
+}
+
+action "Label PR" {
+  uses = "decathlon/pull-request-labeler-action@v1.0.0"
+  secrets = ["GITHUB_TOKEN"]
+}

.gitmodules

@@ -93,7 +93,7 @@
 	url = https://github.com/ClickHouse-Extras/libunwind.git
 [submodule "contrib/simdjson"]
 	path = contrib/simdjson
-	url = https://github.com/ClickHouse-Extras/simdjson.git
+	url = https://github.com/lemire/simdjson.git
 [submodule "contrib/rapidjson"]
 	path = contrib/rapidjson
 	url = https://github.com/Tencent/rapidjson

CHANGELOG.md

@@ -1,3 +1,54 @@
+## ClickHouse release 19.13.2.19, 2019-08-14
+
+### New Feature
+* Sampling profiler on query level. [Example](https://gist.github.com/alexey-milovidov/92758583dd41c24c360fdb8d6a4da194). [#4247](https://github.com/yandex/ClickHouse/issues/4247) ([laplab](https://github.com/laplab)) [#6124](https://github.com/yandex/ClickHouse/pull/6124) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#6250](https://github.com/yandex/ClickHouse/pull/6250) [#6283](https://github.com/yandex/ClickHouse/pull/6283) [#6386](https://github.com/yandex/ClickHouse/pull/6386) 
+* Allow to specify a list of columns with `COLUMNS('regexp')` expression that works like a more sophisticated variant of `*` asterisk. [#5951](https://github.com/yandex/ClickHouse/pull/5951)  ([mfridental](https://github.com/mfridental)), ([alexey-milovidov](https://github.com/alexey-milovidov))
+* `CREATE TABLE AS table_function()` is now possible [#6057](https://github.com/yandex/ClickHouse/pull/6057) ([dimarub2000](https://github.com/dimarub2000))
+* Adam optimizer for stochastic gradient descent is used by default in `stochasticLinearRegression()` and `stochasticLogisticRegression()` aggregate functions, because it shows good quality without almost any tuning. [#6000](https://github.com/yandex/ClickHouse/pull/6000) ([Quid37](https://github.com/Quid37))
+* Added functions for working with the сustom week number [#5212](https://github.com/yandex/ClickHouse/pull/5212) ([Andy Yang](https://github.com/andyyzh))
+* `RENAME` queries now work with all storages. [#5953](https://github.com/yandex/ClickHouse/pull/5953) ([Ivan](https://github.com/abyss7))
+* Now client receive logs from server with any desired level by setting `send_logs_level` regardless to the log level specified in server settings. [#5964](https://github.com/yandex/ClickHouse/pull/5964) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov))
+
+### Experimental features
+* New query processing pipeline. Use `experimental_use_processors=1` option to enable it. Use for your own trouble. [#4914](https://github.com/yandex/ClickHouse/pull/4914) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+
+### Bug Fix
+* Kafka integration has been fixed in this version.
+* Fixed `DoubleDelta` encoding of `Int64` for large `DoubleDelta` values, improved `DoubleDelta` encoding for random data for `Int32`. [#5998](https://github.com/yandex/ClickHouse/pull/5998) ([Vasily Nemkov](https://github.com/Enmk))
+* Fixed overestimation of `max_rows_to_read` if the setting `merge_tree_uniform_read_distribution` is set to 0. [#6019](https://github.com/yandex/ClickHouse/pull/6019) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Improvement
+* The setting `input_format_defaults_for_omitted_fields` is enabled by default. It enables calculation of complex default expressions for omitted fields in `JSONEachRow` and `CSV*` formats. It should be the expected behaviour but may lead to negligible performance difference or subtle incompatibilities. [#6043](https://github.com/yandex/ClickHouse/pull/6043) ([Artem Zuikov](https://github.com/4ertus2)), [#5625](https://github.com/yandex/ClickHouse/pull/5625) ([akuzm](https://github.com/akuzm))
+* Throws an exception if `config.d` file doesn't have the corresponding root element as the config file [#6123](https://github.com/yandex/ClickHouse/pull/6123) ([dimarub2000](https://github.com/dimarub2000))
+
+### Performance Improvement
+* Optimize `count()`. Now it uses the smallest column (if possible). [#6028](https://github.com/yandex/ClickHouse/pull/6028) ([Amos Bird](https://github.com/amosbird))
+
+### Build/Testing/Packaging Improvement
+* Report memory usage in performance tests. [#5899](https://github.com/yandex/ClickHouse/pull/5899) ([akuzm](https://github.com/akuzm))
+* Fix build with external `libcxx` [#6010](https://github.com/yandex/ClickHouse/pull/6010) ([Ivan](https://github.com/abyss7))
+* Fix shared build with `rdkafka` library [#6101](https://github.com/yandex/ClickHouse/pull/6101) ([Ivan](https://github.com/abyss7))
+
+## ClickHouse release 19.11.7.40, 2019-08-14
+
+### Bug fix
+* Kafka integration has been fixed in this version.
+* Fix segfault when using `arrayReduce` for constant arguments. [#6326](https://github.com/yandex/ClickHouse/pull/6326) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed `toFloat()` monotonicity. [#6374](https://github.com/yandex/ClickHouse/pull/6374) ([dimarub2000](https://github.com/dimarub2000))
+* Fix segfault with enabled `optimize_skip_unused_shards` and missing sharding key. [#6384](https://github.com/yandex/ClickHouse/pull/6384) ([CurtizJ](https://github.com/CurtizJ))
+* Fixed logic of `arrayEnumerateUniqRanked` function. [#6423](https://github.com/yandex/ClickHouse/pull/6423) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Removed extra verbose logging from MySQL handler. [#6389](https://github.com/yandex/ClickHouse/pull/6389) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix wrong behavior and possible segfaults in `topK` and `topKWeighted` aggregated functions. [#6404](https://github.com/yandex/ClickHouse/pull/6404) ([CurtizJ](https://github.com/CurtizJ))
+* Do not expose virtual columns in `system.columns` table. This is required for backward compatibility. [#6406](https://github.com/yandex/ClickHouse/pull/6406) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix bug with memory allocation for string fields in complex key cache dictionary. [#6447](https://github.com/yandex/ClickHouse/pull/6447) ([alesapin](https://github.com/alesapin))
+* Fix bug with enabling adaptive granularity when creating new replica for `Replicated*MergeTree` table. [#6452](https://github.com/yandex/ClickHouse/pull/6452) ([alesapin](https://github.com/alesapin))
+* Fix infinite loop when reading Kafka messages. [#6354](https://github.com/yandex/ClickHouse/pull/6354) ([abyss7](https://github.com/abyss7))
+* Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser and possibility of stack overflow in `Merge` and `Distributed` tables [#6433](https://github.com/yandex/ClickHouse/pull/6433) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed Gorilla encoding error on small sequences. [#6444](https://github.com/yandex/ClickHouse/pull/6444) ([Enmk](https://github.com/Enmk))
+
+### Improvement
+* Allow user to override `poll_interval` and `idle_connection_timeout` settings on connection. [#6230](https://github.com/yandex/ClickHouse/pull/6230) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
 ## ClickHouse release 19.11.5.28, 2019-08-05
 
 ### Bug fix

CMakeLists.txt

@@ -264,7 +264,9 @@ if (USE_STATIC_LIBRARIES AND HAVE_NO_PIE)
     set (CMAKE_C_FLAGS                   "${CMAKE_C_FLAGS} ${FLAG_NO_PIE}")
 endif ()
 
-if (NOT SANITIZE AND NOT SPLIT_SHARED_LIBRARIES)
+# TODO: only make this extra-checks in CI builds, since a lot of contrib libs won't link -
+#       CI works around this problem by explicitly adding GLIBC_COMPATIBILITY flag.
+if (NOT SANITIZE AND YANDEX_OFFICIAL_BUILD)
     set (CMAKE_EXE_LINKER_FLAGS              "${CMAKE_EXE_LINKER_FLAGS} -Wl,--no-undefined")
     set (CMAKE_SHARED_LINKER_FLAGS           "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-undefined")
 endif ()
@@ -328,7 +330,7 @@ if (OS_LINUX AND NOT UNBUNDLED AND (GLIBC_COMPATIBILITY OR USE_INTERNAL_UNWIND_L
         if (USE_INTERNAL_LIBCXX_LIBRARY)
             set (LIBCXX_LIBS "${ClickHouse_BINARY_DIR}/contrib/libcxx-cmake/libcxx_static${${CMAKE_POSTFIX_VARIABLE}}.a ${ClickHouse_BINARY_DIR}/contrib/libcxxabi-cmake/libcxxabi_static${${CMAKE_POSTFIX_VARIABLE}}.a")
         else ()
-            set (LIBCXX_LIBS "-lc++ -lc++abi")
+            set (LIBCXX_LIBS "-lc++ -lc++abi -lc++fs")
         endif ()
 
         set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic ${LIBCXX_LIBS} ${EXCEPTION_HANDLING_LIBRARY} ${BUILTINS_LIB_PATH} -Wl,-Bdynamic")

README.md

@@ -13,7 +13,8 @@ ClickHouse is an open-source column-oriented database management system that all
 * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
 
 ## Upcoming Events
-* [ClickHouse Meetup in Mountain View](https://www.eventbrite.com/e/meetup-clickhouse-in-the-south-bay-registration-65935505873) on August 13.
 * [ClickHouse Meetup in Moscow](https://yandex.ru/promo/clickhouse/moscow-2019) on September 5.
+* [ClickHouse Meetup in Paris](https://www.eventbrite.com/e/clickhouse-paris-meetup-2019-registration-68493270215) on October 3.
+* [ClickHouse Meetup in Hong Kong](https://www.meetup.com/Hong-Kong-Machine-Learning-Meetup/events/263580542/) on October 17.
 * [ClickHouse Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
 * [ClickHouse Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.

cmake/find_cpuinfo.cmake

@@ -1,8 +1,8 @@
 option(USE_INTERNAL_CPUINFO_LIBRARY "Set to FALSE to use system cpuinfo library instead of bundled" ${NOT_UNBUNDLED})
 
 # Now we have no contrib/libcpuinfo, use from system.
-if (USE_INTERNAL_CPUINFO_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcpuinfo/include")
-   #message (WARNING "submodule contrib/libcpuid is missing. to fix try run: \n git submodule update --init --recursive")
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcpuinfo/include")
+   #message (WARNING "submodule contrib/libcpuinfo is missing. to fix try run: \n git submodule update --init --recursive")
    set (USE_INTERNAL_CPUINFO_LIBRARY 0)
    set (MISSING_INTERNAL_CPUINFO_LIBRARY 1)
 endif ()
@@ -12,7 +12,7 @@ if(NOT USE_INTERNAL_CPUINFO_LIBRARY)
     find_path(CPUINFO_INCLUDE_DIR NAMES cpuinfo.h PATHS ${CPUINFO_INCLUDE_PATHS})
 endif()
 
-if(CPUID_LIBRARY AND CPUID_INCLUDE_DIR)
+if(CPUINFO_LIBRARY AND CPUINFO_INCLUDE_DIR)
     set(USE_CPUINFO 1)
 elseif(NOT MISSING_INTERNAL_CPUINFO_LIBRARY)
     set(CPUINFO_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libcpuinfo/include)

cmake/find_simdjson.cmake

@@ -3,8 +3,11 @@ if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/simdjson/include/simdjson/jsonp
     return()
 endif ()
 
-if (NOT HAVE_AVX2)
-    message (WARNING "submodule contrib/simdjson requires AVX2 support")
+if (NOT HAVE_SSE42)
+    message (WARNING "submodule contrib/simdjson requires support of SSE4.2 instructions")
+    return()
+elseif (NOT HAVE_PCLMULQDQ)
+    message (WARNING "submodule contrib/simdjson requires support of PCLMULQDQ instructions")
     return()
 endif ()
 

cmake/test_cpu.cmake

@@ -81,6 +81,17 @@ check_cxx_source_compiles("
     }
 " HAVE_AVX2)
 
+set (TEST_FLAG "-mpclmul")
+set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0")
+check_cxx_source_compiles("
+    #include <wmmintrin.h>
+    int main() {
+        auto a = _mm_clmulepi64_si128(__m128i(), __m128i(), 0);
+        (void)a;
+        return 0;
+    }
+" HAVE_PCLMULQDQ)
+
 # gcc -dM -E -mpopcnt - < /dev/null | sort > gcc-dump-popcnt
 #define __POPCNT__ 1
 

contrib/mariadb-connector-c-cmake/CMakeLists.txt

@@ -31,6 +31,7 @@ ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_stmt_codec.c
 ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_string.c
 ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_time.c
 ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_tls.c
+${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/openssl_crypt.c
 #${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/gnutls.c
 #${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/ma_schannel.c
 #${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/schannel.c
@@ -42,6 +43,7 @@ ${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/mariadb_cleartext.c
 ${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/my_auth.c
 ${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/old_password.c
 ${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/sha256_pw.c
+${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/caching_sha2_pw.c
 #${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/sspi_client.c
 #${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/sspi_errmsg.c
 ${MARIADB_CLIENT_SOURCE_DIR}/plugins/connection/aurora.c

contrib/mariadb-connector-c-cmake/linux_x86_64/libmariadb/ma_client_plugin.c

@@ -76,17 +76,20 @@ struct st_client_plugin_int *plugin_list[MYSQL_CLIENT_MAX_PLUGINS + MARIADB_CLIE
 static pthread_mutex_t LOCK_load_client_plugin;
 #endif
 
- extern struct st_mysql_client_plugin mysql_native_password_client_plugin;
- extern struct st_mysql_client_plugin mysql_old_password_client_plugin;
- extern struct st_mysql_client_plugin pvio_socket_client_plugin;
+extern struct st_mysql_client_plugin mysql_native_password_client_plugin;
+extern struct st_mysql_client_plugin mysql_old_password_client_plugin;
+extern struct st_mysql_client_plugin pvio_socket_client_plugin;
+extern struct st_mysql_client_plugin sha256_password_client_plugin;
+extern struct st_mysql_client_plugin caching_sha2_password_client_plugin;
 
 
 struct st_mysql_client_plugin *mysql_client_builtins[]=
 {
-     (struct st_mysql_client_plugin *)&mysql_native_password_client_plugin,
-   (struct st_mysql_client_plugin *)&mysql_old_password_client_plugin,
-   (struct st_mysql_client_plugin *)&pvio_socket_client_plugin,
-
+  (struct st_mysql_client_plugin *)&mysql_native_password_client_plugin,
+  (struct st_mysql_client_plugin *)&mysql_old_password_client_plugin,
+  (struct st_mysql_client_plugin *)&pvio_socket_client_plugin,
+  (struct st_mysql_client_plugin *)&sha256_password_client_plugin,
+  (struct st_mysql_client_plugin *)&caching_sha2_password_client_plugin,
   0
 };
 

contrib/simdjson-cmake/CMakeLists.txt

@@ -1,6 +1,3 @@
-if (NOT HAVE_AVX2)
-    message (FATAL_ERROR "No AVX2 support")
-endif ()
 set(SIMDJSON_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/simdjson/include")
 set(SIMDJSON_SRC_DIR "${SIMDJSON_INCLUDE_DIR}/../src")
 set(SIMDJSON_SRC
@@ -16,4 +13,3 @@ set(SIMDJSON_SRC
 
 add_library(${SIMDJSON_LIBRARY} ${SIMDJSON_SRC})
 target_include_directories(${SIMDJSON_LIBRARY} SYSTEM PUBLIC "${SIMDJSON_INCLUDE_DIR}")
-target_compile_options(${SIMDJSON_LIBRARY} PRIVATE -mavx2 -mbmi -mbmi2 -mpclmul)

dbms/cmake/version.cmake

@@ -1,11 +1,11 @@
 # This strings autochanged from release_lib.sh:
 set(VERSION_REVISION 54425)
 set(VERSION_MAJOR 19)
-set(VERSION_MINOR 13)
+set(VERSION_MINOR 14)
 set(VERSION_PATCH 1)
 set(VERSION_GITHASH adfc36917222bdb03eba069f0cad0f4f5b8f1c94)
-set(VERSION_DESCRIBE v19.13.1.1-prestable)
-set(VERSION_STRING 19.13.1.1)
+set(VERSION_DESCRIBE v19.14.1.1-prestable)
+set(VERSION_STRING 19.14.1.1)
 # end of autochange
 
 set(VERSION_EXTRA "" CACHE STRING "")

dbms/programs/performance-test/PerformanceTest.cpp

@@ -3,6 +3,8 @@
 #include <Core/Types.h>
 #include <Common/CpuId.h>
 #include <common/getMemoryAmount.h>
+#include <DataStreams/copyData.h>
+#include <DataStreams/NullBlockOutputStream.h>
 #include <DataStreams/RemoteBlockInputStream.h>
 #include <IO/ConnectionTimeouts.h>
 #include <IO/ReadBufferFromFile.h>
@@ -249,6 +251,54 @@ std::vector<TestStats> PerformanceTest::execute()
 
         runQueries(queries_with_indexes, statistics_by_run);
     }
+
+    if (got_SIGINT)
+    {
+        return statistics_by_run;
+    }
+
+    // Pull memory usage data from query log. The log is normally filled in
+    // background, so we have to flush it synchronously here to see all the
+    // previous queries.
+    {
+        NullBlockOutputStream null_output(Block{});
+        RemoteBlockInputStream flush_log(connection, "system flush logs",
+            {} /* header */, context);
+        copyData(flush_log, null_output);
+    }
+
+    for (auto & statistics : statistics_by_run)
+    {
+        if (statistics.query_id.empty())
+        {
+            // We have statistics structs for skipped queries as well, so we
+            // have to filter them out.
+            continue;
+        }
+
+        // We run some test queries several times, specifying the same query id,
+        // so this query to the log may return several records. Choose the
+        // last one, because this is when the query performance has stabilized.
+        RemoteBlockInputStream log_reader(connection,
+            "select memory_usage, query_start_time from system.query_log "
+            "where type = 2 and query_id = '" + statistics.query_id + "' "
+            "order by query_start_time desc",
+            {} /* header */, context);
+
+        log_reader.readPrefix();
+        Block block = log_reader.read();
+        if (block.columns() == 0)
+        {
+            LOG_WARNING(log, "Query '" << statistics.query_id << "' is not found in query log.");
+            continue;
+        }
+
+        auto column = block.getByName("memory_usage").column;
+        statistics.memory_usage = column->get64(0);
+
+        log_reader.readSuffix();
+    }
+
     return statistics_by_run;
 }
 
@@ -298,47 +348,6 @@ void PerformanceTest::runQueries(
             break;
         }
     }
-
-    if (got_SIGINT)
-    {
-        return;
-    }
-
-    // Pull memory usage data from query log. The log is normally filled in
-    // background, so we have to flush it synchronously here to see all the
-    // previous queries.
-    {
-        RemoteBlockInputStream flush_log(connection, "system flush logs",
-            {} /* header */, context);
-        flush_log.readPrefix();
-        while (flush_log.read());
-        flush_log.readSuffix();
-    }
-
-    for (auto & statistics : statistics_by_run)
-    {
-        RemoteBlockInputStream log_reader(connection,
-            "select memory_usage from system.query_log where type = 2 and query_id = '"
-                                   + statistics.query_id + "'",
-            {} /* header */, context);
-
-        log_reader.readPrefix();
-        Block block = log_reader.read();
-        if (block.columns() == 0)
-        {
-            LOG_WARNING(log, "Query '" << statistics.query_id << "' is not found in query log.");
-            continue;
-        }
-
-        assert(block.columns() == 1);
-        assert(block.getDataTypes()[0]->getName() == "UInt64");
-        ColumnPtr column = block.getByPosition(0).column;
-        assert(column->size() == 1);
-        StringRef ref = column->getDataAt(0);
-        assert(ref.size == sizeof(UInt64));
-        statistics.memory_usage = *reinterpret_cast<const UInt64*>(ref.data);
-        log_reader.readSuffix();
-    }
 }