Merge remote-tracking branch 'rschu1ze/master' into more-mysql-compat

docker/test/integration/runner/compose/docker_compose_ldap.yml

@@ -0,0 +1,16 @@
+version: '2.3'
+services:
+    openldap:
+        image: bitnami/openldap:2.6.6
+        restart: always
+        environment:
+            LDAP_ROOT: dc=example,dc=org
+            LDAP_ADMIN_DN: cn=admin,dc=example,dc=org
+            LDAP_ADMIN_USERNAME: admin
+            LDAP_ADMIN_PASSWORD: clickhouse
+            LDAP_USER_DC: users
+            LDAP_USERS: janedoe,johndoe
+            LDAP_PASSWORDS: qwerty,qwertz
+            LDAP_PORT_NUMBER: ${LDAP_INTERNAL_PORT:-1389}
+        ports:
+            - ${LDAP_EXTERNAL_PORT:-1389}:${LDAP_INTERNAL_PORT:-1389}

docs/en/engines/table-engines/integrations/hive.md

@@ -60,7 +60,7 @@ Before using cache, add it to `config.xml`
 - limit_size: Required. The maximum size(in bytes) of local cache files.
 - bytes_read_before_flush: Control bytes before flush to local filesystem when downloading file from remote filesystem. The default value is 1MB.
 
-When ClickHouse is started up with local cache for remote filesystem enabled, users can still choose not to use cache with `settings use_local_cache_for_remote_fs = 0` in their query. `use_local_cache_for_remote_fs` is `false` in default.
+When ClickHouse is started up with local cache for remote filesystem enabled, users can still choose not to use cache with `settings use_local_cache_for_remote_storage = 0` in their query. `use_local_cache_for_remote_storage` is `1` by default.
 
 ### Query Hive Table with ORC Input Format
 

docs/en/operations/query-cache.md

@@ -69,7 +69,7 @@ may return cached results then.
 The query cache can be cleared using statement `SYSTEM DROP QUERY CACHE`. The content of the query cache is displayed in system table
 `system.query_cache`. The number of query cache hits and misses since database start are shown as events "QueryCacheHits" and
 "QueryCacheMisses" in system table [system.events](system-tables/events.md). Both counters are only updated for `SELECT` queries which run
-with setting `use_query_cache = true`, other queries do not affect "QueryCacheMisses". Field `query_log_usage` in system table
+with setting `use_query_cache = true`, other queries do not affect "QueryCacheMisses". Field `query_cache_usage` in system table
 [system.query_log](system-tables/query_log.md) shows for each executed query whether the query result was written into or read from the
 query cache. Asynchronous metrics "QueryCacheEntries" and "QueryCacheBytes" in system table
 [system.asynchronous_metrics](system-tables/asynchronous_metrics.md) show how many entries / bytes the query cache currently contains.

docs/en/operations/settings/merge-tree-settings.md

@@ -502,7 +502,7 @@ Possible values:
 Default value: 480.
 
 After merging several parts into a new part, ClickHouse marks the original parts as inactive and deletes them only after `old_parts_lifetime` seconds.
-Inactive parts are removed if they are not used by current queries, i.e. if the `refcount` of the part is zero.
+Inactive parts are removed if they are not used by current queries, i.e. if the `refcount` of the part is 1.
 
 `fsync` is not called for new parts, so for some time new parts exist only in the server's RAM (OS cache). If the server is rebooted spontaneously, new parts can be lost or damaged.
 To protect data inactive parts are not deleted immediately.

docs/en/operations/settings/settings.md

@@ -3279,6 +3279,17 @@ Possible values:
 
 Default value: `0`.
 
+## use_mysql_types_in_show_columns {#use_mysql_types_in_show_columns}
+
+Show the names of MySQL data types corresponding to ClickHouse data types in [SHOW COLUMNS](../../sql-reference/statements/show.md#show_columns) and SELECTs on [system.columns](../system-tables/columns.md).
+
+Possible values:
+
+- 0 - Show names of native ClickHouse data types.
+- 1 - Show names of MySQL data types corresponding to ClickHouse data types.
+
+Default value: `0`.
+
 ## execute_merges_on_single_replica_time_threshold {#execute-merges-on-single-replica-time-threshold}
 
 Enables special logic to perform merges on replicas.

docs/en/operations/system-tables/columns.md

@@ -14,7 +14,7 @@ The `system.columns` table contains the following columns (the column type is sh
 - `database` ([String](../../sql-reference/data-types/string.md)) — Database name.
 - `table` ([String](../../sql-reference/data-types/string.md)) — Table name.
 - `name` ([String](../../sql-reference/data-types/string.md)) — Column name.
-- `type` ([String](../../sql-reference/data-types/string.md)) — Column type.
+- `type` ([String](../../sql-reference/data-types/string.md)) — Column type. If setting `[use_mysql_types_in_show_columns](../../operations/settings/settings.md#use_mysql_types_in_show_columns) = 1` (default: 0), then the equivalent type name in MySQL is shown.
 - `position` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Ordinal position of a column in a table starting with 1.
 - `default_kind` ([String](../../sql-reference/data-types/string.md)) — Expression type (`DEFAULT`, `MATERIALIZED`, `ALIAS`) for the default value, or an empty string if it is not defined.
 - `default_expression` ([String](../../sql-reference/data-types/string.md)) — Expression for the default value, or an empty string if it is not defined.

docs/en/sql-reference/functions/splitting-merging-functions.md

@@ -34,10 +34,10 @@ Returns an array of selected substrings. Empty substrings may be selected when:
 Type: [Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md)).
 
 :::note
-The behavior of parameter `max_substrings` changed starting with ClickHouse v22.11. In versions older than that, `max_substrings` > 0 meant that `max_substring`-many splits were performed and that the remainder of the string was returned as the final element of the list.
+The behavior of parameter `max_substrings` changed starting with ClickHouse v22.11. In versions older than that, `max_substrings > 0` meant that `max_substring`-many splits were performed and that the remainder of the string was returned as the final element of the list.
 For example,
-- in v22.10: `SELECT splitByChar('=', 'a=b=c=d', 2); -- ['a','b','c=d']`
-- in v22.11: `SELECT splitByChar('=', 'a=b=c=d', 2); -- ['a','b']`
+- in v22.10: `SELECT splitByChar('=', 'a=b=c=d', 2);` returned `['a','b','c=d']`
+- in v22.11: `SELECT splitByChar('=', 'a=b=c=d', 2);` returned `['a','b']`
 
 A behavior similar to ClickHouse pre-v22.11 can be achieved by setting
 [splitby_max_substrings_includes_remaining_string](../../operations/settings/settings.md#splitby_max_substrings_includes_remaining_string)

docs/en/sql-reference/statements/show.md

@@ -189,7 +189,7 @@ Result:
 - [Create Tables](https://clickhouse.com/docs/en/getting-started/tutorial/#create-tables)
 - [SHOW CREATE TABLE](https://clickhouse.com/docs/en/sql-reference/statements/show/#show-create-table)
 
-## SHOW COLUMNS
+## SHOW COLUMNS {#show_columns}
 
 Displays a list of columns
 
@@ -206,15 +206,15 @@ The optional keyword `EXTENDED` currently has no effect, it only exists for MySQ
 The optional keyword `FULL` causes the output to include the collation, comment and privilege columns.
 
 The statement produces a result table with the following structure:
-- field - The name of the column (String)
-- type - The column data type (String)
-- null - `YES` if the column data type is Nullable, `NO` otherwise (String)
-- key - `PRI` if the column is part of the primary key, `SOR` if the column is part of the sorting key, empty otherwise (String)
-- default - Default expression of the column if it is of type `ALIAS`, `DEFAULT`, or `MATERIALIZED`, otherwise `NULL`. (Nullable(String))
-- extra - Additional information, currently unused (String)
-- collation - (only if `FULL` keyword was specified) Collation of the column, always `NULL` because ClickHouse has no per-column collations (Nullable(String))
-- comment - (only if `FULL` keyword was specified) Comment on the column (String)
-- privilege - (only if `FULL` keyword was specified) The privilege you have on this column, currently not available (String)
+- `field` - The name of the column (String)
+- `type` - The column data type. If setting `[use_mysql_types_in_show_columns](../../operations/settings/settings.md#use_mysql_types_in_show_columns) = 1` (default: 0), then the equivalent type name in MySQL is shown. (String)
+- `null` - `YES` if the column data type is Nullable, `NO` otherwise (String)
+- `key` - `PRI` if the column is part of the primary key, `SOR` if the column is part of the sorting key, empty otherwise (String)
+- `default` - Default expression of the column if it is of type `ALIAS`, `DEFAULT`, or `MATERIALIZED`, otherwise `NULL`. (Nullable(String))
+- `extra` - Additional information, currently unused (String)
+- `collation` - (only if `FULL` keyword was specified) Collation of the column, always `NULL` because ClickHouse has no per-column collations (Nullable(String))
+- `comment` - (only if `FULL` keyword was specified) Comment on the column (String)
+- `privilege` - (only if `FULL` keyword was specified) The privilege you have on this column, currently not available (String)
 
 **Examples**
 
@@ -286,21 +286,21 @@ equivalent. If no database is specified, the query assumes the current database
 The optional keyword `EXTENDED` currently has no effect, it only exists for MySQL compatibility.
 
 The statement produces a result table with the following structure:
-- table - The name of the table. (String)
-- non_unique - Always `1` as ClickHouse does not support uniqueness constraints. (UInt8)
-- key_name - The name of the index, `PRIMARY` if the index is a primary key index. (String)
-- seq_in_index - For a primary key index, the position of the column starting from `1`. For a data skipping index: always `1`. (UInt8)
-- column_name - For a primary key index, the name of the column. For a data skipping index: `''` (empty string), see field "expression". (String)
-- collation - The sorting of the column in the index: `A` if ascending, `D` if descending, `NULL` if unsorted. (Nullable(String))
-- cardinality - An estimation of the index cardinality (number of unique values in the index). Currently always 0. (UInt64)
-- sub_part - Always `NULL` because ClickHouse does not support index prefixes like MySQL. (Nullable(String))
-- packed - Always `NULL` because ClickHouse does not support packed indexes (like MySQL). (Nullable(String))
-- null - Currently unused
-- index_type - The index type, e.g. `PRIMARY`, `MINMAX`, `BLOOM_FILTER` etc. (String)
-- comment - Additional information about the index, currently always `''` (empty string). (String)
-- index_comment - `''` (empty string) because indexes in ClickHouse cannot have a `COMMENT` field (like in MySQL). (String)
-- visible - If the index is visible to the optimizer, always `YES`. (String)
-- expression - For a data skipping index, the index expression. For a primary key index: `''` (empty string). (String)
+- `table` - The name of the table. (String)
+- `non_unique` - Always `1` as ClickHouse does not support uniqueness constraints. (UInt8)
+- `key_name` - The name of the index, `PRIMARY` if the index is a primary key index. (String)
+- `seq_in_index` - For a primary key index, the position of the column starting from `1`. For a data skipping index: always `1`. (UInt8)
+- `column_name` - For a primary key index, the name of the column. For a data skipping index: `''` (empty string), see field "expression". (String)
+- `collation` - The sorting of the column in the index: `A` if ascending, `D` if descending, `NULL` if unsorted. (Nullable(String))
+- `cardinality` - An estimation of the index cardinality (number of unique values in the index). Currently always 0. (UInt64)
+- `sub_part` - Always `NULL` because ClickHouse does not support index prefixes like MySQL. (Nullable(String))
+- `packed` - Always `NULL` because ClickHouse does not support packed indexes (like MySQL). (Nullable(String))
+- `null` - Currently unused
+- `index_type` - The index type, e.g. `PRIMARY`, `MINMAX`, `BLOOM_FILTER` etc. (String)
+- `comment` - Additional information about the index, currently always `''` (empty string). (String)
+- `index_comment` - `''` (empty string) because indexes in ClickHouse cannot have a `COMMENT` field (like in MySQL). (String)
+- `visible` - If the index is visible to the optimizer, always `YES`. (String)
+- `expression` - For a data skipping index, the index expression. For a primary key index: `''` (empty string). (String)
 
 **Examples**
 

docs/zh/engines/table-engines/integrations/hive.md

@@ -61,7 +61,7 @@ PARTITION BY expr
 - limit_size: 必需的。本地缓存文件的最大大小(单位为字节)。
 - bytes_read_before_flush: 从远程文件系统下载文件时，刷新到本地文件系统前的控制字节数。缺省值为1MB。
 
-当ClickHouse为远程文件系统启用了本地缓存时，用户仍然可以选择不使用缓存，并在查询中设置`use_local_cache_for_remote_fs = 0 `, `use_local_cache_for_remote_fs` 默认为 `false`。
+当ClickHouse为远程文件系统启用了本地缓存时，用户仍然可以选择不使用缓存，并在查询中设置 `use_local_cache_for_remote_storage = 0`, `use_local_cache_for_remote_storage` 默认为 `1`。
 
 ### 查询 ORC 输入格式的Hive 表
 

packages/build

@@ -84,8 +84,8 @@ function deb2tgz {
     FILE=$1
     PKG_NAME=${FILE##*/}; PKG_NAME=${PKG_NAME%%_*}
     PKG_DIR="$PKG_NAME-$CLICKHOUSE_VERSION_STRING"
-    PKG_PATH="$OUTPUT_DIR/$PKG_NAME-$CLICKHOUSE_VERSION_STRING"
-    TARBALL="$OUTPUT_DIR/$PKG_NAME-$CLICKHOUSE_VERSION_STRING-$DEB_ARCH.tgz"
+    PKG_PATH="$OUTPUT_DIR/$PKG_DIR"
+    TARBALL="$OUTPUT_DIR/$PKG_DIR-$DEB_ARCH.tgz"
     rm -rf "$PKG_PATH"
     dpkg-deb -R "$FILE" "$PKG_PATH"
     mkdir -p "$PKG_PATH/install"

packages/clickhouse-client.yaml

@@ -1,6 +1,13 @@
 # package sources should be placed in ${PWD}/root
 # nfpm should run from the same directory with a config
 name: "clickhouse-client"
+description: |
+  Client binary for ClickHouse
+    ClickHouse is a column-oriented database management system.
+    that allows generating analytical data reports in real time.
+    This package provides clickhouse-client, clickhouse-local and clickhouse-benchmark.
+
+# Common packages config
 arch: "${DEB_ARCH}" # amd64, arm64
 platform: "linux"
 version: "${CLICKHOUSE_VERSION_STRING}"
@@ -9,19 +16,17 @@ homepage: "https://clickhouse.com"
 license: "Apache"
 section: "database"
 priority: "optional"
+maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
+deb:
+  fields:
+    Source: clickhouse
 
+# Package specific content
 replaces:
 - clickhouse-compressor
 conflicts:
 - clickhouse-compressor
 
-maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
-description: |
-  Client binary for ClickHouse
-    ClickHouse is a column-oriented database management system.
-    that allows generating analytical data reports in real time.
-    This package provides clickhouse-client, clickhouse-local and clickhouse-benchmark.
-
 overrides:
   deb:
     depends:
@@ -30,10 +35,6 @@ overrides:
     depends:
     - clickhouse-common-static = ${CLICKHOUSE_VERSION_STRING}
 
-deb:
-  fields:
-    Source: clickhouse
-
 contents:
 - src: root/etc/clickhouse-client/config.xml
   dst: /etc/clickhouse-client/config.xml

packages/clickhouse-common-static-dbg.yaml

@@ -1,6 +1,13 @@
 # package sources should be placed in ${PWD}/root
 # nfpm should run from the same directory with a config
 name: "clickhouse-common-static-dbg"
+description: |
+  debugging symbols for clickhouse-common-static
+    This package contains the debugging symbols for clickhouse-common.
+
+
+#
+# Common packages config
 arch: "${DEB_ARCH}" # amd64, arm64
 platform: "linux"
 version: "${CLICKHOUSE_VERSION_STRING}"
@@ -9,21 +16,17 @@ homepage: "https://clickhouse.com"
 license: "Apache"
 section: "database"
 priority: "optional"
+maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
+deb:
+  fields:
+    Source: clickhouse
 
+# Package specific content
 replaces:
 - clickhouse-common-dbg
 conflicts:
 - clickhouse-common-dbg
 
-maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
-description: |
-  debugging symbols for clickhouse-common-static
-    This package contains the debugging symbols for clickhouse-common.
-
-deb:
-  fields:
-    Source: clickhouse
-
 contents:
 - src: root/usr/lib/debug/usr/bin/clickhouse.debug
   dst: /usr/lib/debug/usr/bin/clickhouse.debug

packages/clickhouse-common-static.yaml

@@ -1,6 +1,13 @@
 # package sources should be placed in ${PWD}/root
 # nfpm should run from the same directory with a config
 name: "clickhouse-common-static"
+description: |
+  Common files for ClickHouse
+    ClickHouse is a column-oriented database management system
+    that allows generating analytical data reports in real time.
+    This package provides common files for both clickhouse server and client
+
+# Common packages config
 arch: "${DEB_ARCH}" # amd64, arm64
 platform: "linux"
 version: "${CLICKHOUSE_VERSION_STRING}"
@@ -9,7 +16,12 @@ homepage: "https://clickhouse.com"
 license: "Apache"
 section: "database"
 priority: "optional"
+maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
+deb:
+  fields:
+    Source: clickhouse
 
+# Package specific content
 replaces:
 - clickhouse-common
 - clickhouse-server-base
@@ -19,17 +31,6 @@ provides:
 suggests:
 - clickhouse-common-static-dbg
 
-maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
-description: |
-  Common files for ClickHouse
-    ClickHouse is a column-oriented database management system
-    that allows generating analytical data reports in real time.
-    This package provides common files for both clickhouse server and client
-
-deb:
-  fields:
-    Source: clickhouse
-
 contents:
 - src: root/usr/bin/clickhouse
   dst: /usr/bin/clickhouse

packages/clickhouse-keeper-dbg.yaml

@@ -1,6 +1,13 @@
 # package sources should be placed in ${PWD}/root
 # nfpm should run from the same directory with a config
 name: "clickhouse-keeper-dbg"
+description: |
+  debugging symbols for clickhouse-keeper
+    This package contains the debugging symbols for clickhouse-keeper.
+
+
+#
+# Common packages config
 arch: "${DEB_ARCH}" # amd64, arm64
 platform: "linux"
 version: "${CLICKHOUSE_VERSION_STRING}"
@@ -10,14 +17,11 @@ license: "Apache"
 section: "database"
 priority: "optional"
 maintainer: "ClickHouse Dev Team <packages+linux@clickhouse.com>"
-description: |
-  debugging symbols for clickhouse-keeper
-    This package contains the debugging symbols for clickhouse-keeper.
-
 deb:
   fields:
     Source: clickhouse
 
+# Package specific content
 contents:
 - src: root/usr/lib/debug/usr/bin/clickhouse-keeper.debug
   dst: /usr/lib/debug/usr/bin/clickhouse-keeper.debug

packages/clickhouse-keeper.postinstall

@@ -3,12 +3,12 @@ set -e
 # set -x
 
 PROGRAM=clickhouse-keeper
-KEEPER_USER=${KEEPER_USER:=clickhouse}
-KEEPER_GROUP=${KEEPER_GROUP:=clickhouse}
+KEEPER_USER=${KEEPER_USER:-clickhouse}
+KEEPER_GROUP=${KEEPER_GROUP:-clickhouse}
 # Please note that we don't support paths with whitespaces. This is rather ignorant.
-KEEPER_CONFDIR=${KEEPER_CONFDIR:=/etc/$PROGRAM}
-KEEPER_DATADIR=${KEEPER_DATADIR:=/var/lib/clickhouse}
-KEEPER_LOGDIR=${KEEPER_LOGDIR:=/var/log/$PROGRAM}
+KEEPER_CONFDIR=${KEEPER_CONFDIR:-/etc/$PROGRAM}
+KEEPER_DATADIR=${KEEPER_DATADIR:-/var/lib/clickhouse}
+KEEPER_LOGDIR=${KEEPER_LOGDIR:-/var/log/$PROGRAM}
 
 [ -f /usr/share/debconf/confmodule ] && . /usr/share/debconf/confmodule
 [ -f /etc/default/clickhouse-keeper ] && . /etc/default/clickhouse-keeper