Backport #12116 to 20.5: Fix handling dictionary-table dependency (#1…

…2260)

* Whitespace

* Fix ORDER BY tuple with COLLATE on const column

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* make a short test for DateTime

* Fixup

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>

* Use ConcurentBoundedQueue instead of vector

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Update contrib/jemalloc-cmake/CMakeLists.txt

* Remove libevent library

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Fix function if with FixedString arguments of different sizes

* Add multiIf

* Fix error

* Remove test that is not supported by "Arcadia" build system

* Fix constraints check for constant columns

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix error

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <asiana21@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <43650329+AsiaKorushkina@users.noreply.github.com>
Co-authored-by: asiana21 <asiana21@yandex-team.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* Changelog for 20.1, 20.4

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Evgenia Sudarikova <56156889+otrazhenia@users.noreply.github.com>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <alesapin@gmail.com>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <alesapin@gmail.com>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: alesapin <alesapin@gmail.com>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <asiana21@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <43650329+AsiaKorushkina@users.noreply.github.com>
Co-authored-by: asiana21 <asiana21@yandex-team.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Changelog for 20.5

* Update CHANGELOG.md

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Fix handling dependency of table with ENGINE=Dictionary on dictionary.

* Add test.

* Update 01355_CSV_input_format_allow_errors.sh

* Update CHANGELOG.md

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* New ISO8601 year modificators for formatDateTime

* Added a test

* Remove harmful code from "geoDistance" #12117

* Update formatDateTime.cpp

* Fix warnings from CodeQL

* Added a test

* Minor modification

* Update libdivide to the latest master

* Replace exit to abort in libdivide

* Fix potential overflow in integer division #12119

* Added a test

* Whitespace

* Added a test for #4211

* Attempt to fix "Arcadia" build

* Rename test

* Update arraySum.cpp

* Make code clearer: use enum instead of `bool internal`.

* Update StorageDictionary.h

* Update StorageDictionary.h

* ILIKE operator (#12125)

* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <me-clickhouse@myrrec.space>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>

* add parseDateTimeBestEffortUS function (#12028)

* add function parseDateTimeBestEffortUS
* add test
* add doc

Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>

* Cleanup changelog (half done). Now it is acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Additions requested by @filimonov #12104

Co-authored-by: Alexey Milovidov <milovidov@yandex-team.ru>
Co-authored-by: Andrew Onyshchuk <andryk.rv@gmail.com>
Co-authored-by: Mikhail Gaidamaka <magnoly.nsk@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <akuzm@yandex-team.ru>
Co-authored-by: Nikolai Kochetov <nik-kochetov@yandex-team.ru>
Co-authored-by: Ivan Blinkov <github@blinkov.ru>
Co-authored-by: alesapin <alesapin@gmail.com>
Co-authored-by: BayoNet <da-daos@yandex.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: Roman Bug <rrrrrroman@gmail.com>
Co-authored-by: romanzhukov <romanzhukov@yandex-team.ru>
Co-authored-by: kssenii <sumarokovakseniia@mail.ru>
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>
Co-authored-by: Vitaly Baranov <vitbar@yandex-team.ru>
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: chengy8934 <67622393+chengy8934@users.noreply.github.com>
Co-authored-by: MicrochipQ <microchipq@gmail.com>
Co-authored-by: Artem Zuikov <chertus@gmail.com>
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
Co-authored-by: Vladimir Chebotarev <vladimir.chebotarev@gmail.com>
Co-authored-by: Nicolae Vartolomei <me@nvartolomei.com>
Co-authored-by: filimonov <1549571+filimonov@users.noreply.github.com>
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: AsiaKorushkina <43650329+AsiaKorushkina@users.noreply.github.com>
Co-authored-by: asiana21 <asiana21@yandex-team.ru>
Co-authored-by: BohuTANG <overred.shuttler@gmail.com>
Co-authored-by: zhang2014 <coswde@gmail.com>
Co-authored-by: Evgenia Sudarikova <56156889+otrazhenia@users.noreply.github.com>
Co-authored-by: Anton Popov <pad11rus@gmail.com>
Co-authored-by: manmitya <30998567+manmitya@users.noreply.github.com>
Co-authored-by: Nikolai Kochetov <KochetovNicolai@users.noreply.github.com>
Co-authored-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-authored-by: Anton Ivashkin <iantonspb@yandex-team.ru>
Co-authored-by: Ilya Yatsishin <2159081+qoega@users.noreply.github.com>
Co-authored-by: tavplubix <tavplubix@gmail.com>
Co-authored-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
Co-authored-by: vivarum <vivarum.msk@yandex.ru>
Co-authored-by: myrrc <me@myrrec.space>
Co-authored-by: myrrc <me-clickhouse@myrrec.space>
Co-authored-by: flynn <fenglv15@mails.ucas.ac.cn>

src/Databases/DatabaseDictionary.cpp

@@ -28,7 +28,11 @@ namespace
             if (!load_result.config)
                 return nullptr;
             DictionaryStructure dictionary_structure = ExternalDictionariesLoader::getDictionaryStructure(*load_result.config);
-            return StorageDictionary::create(StorageID(database_name, load_result.name), load_result.name, dictionary_structure);
+            return StorageDictionary::create(
+                StorageID(database_name, load_result.name),
+                load_result.name,
+                dictionary_structure,
+                StorageDictionary::Location::DictionaryDatabase);
         }
         catch (Exception & e)
         {

src/Databases/DatabaseWithDictionaries.cpp

@@ -49,7 +49,8 @@ void DatabaseWithDictionaries::attachDictionary(const String & dictionary_name,
                 StorageDictionary::create(
                     StorageID(getDatabaseName(), dictionary_name),
                     full_name,
-                    ExternalDictionariesLoader::getDictionaryStructure(*attach_info.config)),
+                    ExternalDictionariesLoader::getDictionaryStructure(*attach_info.config),
+                    StorageDictionary::Location::SameDatabaseAndNameAsDictionary),
                 lock);
         }
         catch (...)

src/Storages/StorageDictionary.cpp

@@ -96,17 +96,29 @@ String StorageDictionary::generateNamesAndTypesDescription(const NamesAndTypesLi
 StorageDictionary::StorageDictionary(
     const StorageID & table_id_,
     const String & dictionary_name_,
-    const DictionaryStructure & dictionary_structure_)
+    const ColumnsDescription & columns_,
+    Location location_)
     : IStorage(table_id_)
     , dictionary_name(dictionary_name_)
+    , location(location_)
+{
+    setColumns(columns_);
+}
+
+
+StorageDictionary::StorageDictionary(
+    const StorageID & table_id_, const String & dictionary_name_, const DictionaryStructure & dictionary_structure_, Location location_)
+    : StorageDictionary(table_id_, dictionary_name_, ColumnsDescription{getNamesAndTypes(dictionary_structure_)}, location_)
 {
-    setColumns(ColumnsDescription{getNamesAndTypes(dictionary_structure_)});
 }
 
 
 void StorageDictionary::checkTableCanBeDropped() const
 {
-    throw Exception("Cannot detach dictionary " + backQuote(dictionary_name) + " as table, use DETACH DICTIONARY query.", ErrorCodes::CANNOT_DETACH_DICTIONARY_AS_TABLE);
+    if (location == Location::SameDatabaseAndNameAsDictionary)
+        throw Exception("Cannot detach dictionary " + backQuote(dictionary_name) + " as table, use DETACH DICTIONARY query", ErrorCodes::CANNOT_DETACH_DICTIONARY_AS_TABLE);
+    if (location == Location::DictionaryDatabase)
+        throw Exception("Cannot detach table " + getStorageID().getFullTableName() + " from a database with DICTIONARY engine", ErrorCodes::CANNOT_DETACH_DICTIONARY_AS_TABLE);
 }
 
 Pipes StorageDictionary::read(
@@ -138,11 +150,14 @@ void registerStorageDictionary(StorageFactory & factory)
         args.engine_args[0] = evaluateConstantExpressionOrIdentifierAsLiteral(args.engine_args[0], args.local_context);
         String dictionary_name = args.engine_args[0]->as<ASTLiteral &>().value.safeGet<String>();
 
-        const auto & dictionary = args.context.getExternalDictionariesLoader().getDictionary(dictionary_name);
-        const DictionaryStructure & dictionary_structure = dictionary->getStructure();
-        checkNamesAndTypesCompatibleWithDictionary(dictionary_name, args.columns, dictionary_structure);
+        if (!args.attach)
+        {
+            const auto & dictionary = args.context.getExternalDictionariesLoader().getDictionary(dictionary_name);
+            const DictionaryStructure & dictionary_structure = dictionary->getStructure();
+            checkNamesAndTypesCompatibleWithDictionary(dictionary_name, args.columns, dictionary_structure);
+        }
 
-        return StorageDictionary::create(args.table_id, dictionary_name, dictionary_structure);
+        return StorageDictionary::create(args.table_id, dictionary_name, args.columns, StorageDictionary::Location::Custom);
     });
 }
 

src/Storages/StorageDictionary.h

@@ -28,14 +28,40 @@ class StorageDictionary final : public ext::shared_ptr_helper<StorageDictionary>
 
     const String & dictionaryName() const { return dictionary_name; }
 
+    /// Specifies where the table is located relative to the dictionary.
+    enum class Location
+    {
+        /// Table was created automatically as an element of a database with the Dictionary engine.
+        DictionaryDatabase,
+
+        /// Table was created automatically along with a dictionary
+        /// and has the same database and name as the dictionary.
+        /// It provides table-like access to the dictionary.
+        /// User cannot drop that table.
+        SameDatabaseAndNameAsDictionary,
+
+        /// Table was created explicitly by a statement like
+        /// CREATE TABLE ... ENGINE=Dictionary
+        /// User chose the table's database and name and can drop that table.
+        Custom,
+    };
+
 private:
-    String dictionary_name;
+    const String dictionary_name;
+    const Location location;
 
 protected:
     StorageDictionary(
         const StorageID & table_id_,
         const String & dictionary_name_,
-        const DictionaryStructure & dictionary_structure);
+        const ColumnsDescription & columns_,
+        Location location_);
+
+    StorageDictionary(
+        const StorageID & table_id_,
+        const String & dictionary_name_,
+        const DictionaryStructure & dictionary_structure,
+        Location location_);
 };
 
 }

tests/integration/test_dictionaries_dependency/configs/disable_lazy_load.xml

@@ -0,0 +1,4 @@
+<yandex>
+    <dictionaries_lazy_load>false</dictionaries_lazy_load>
+</yandex>
+

tests/integration/test_dictionaries_dependency/test.py

@@ -0,0 +1,109 @@
+import pytest
+from helpers.cluster import ClickHouseCluster
+
+cluster = ClickHouseCluster(__file__)
+node1 = cluster.add_instance('node1', stay_alive=True)
+node2 = cluster.add_instance('node2', stay_alive=True, main_configs=['configs/disable_lazy_load.xml'])
+nodes = [node1, node2]
+
+
+@pytest.fixture(scope="module", autouse=True)
+def start_cluster():
+    try:
+        cluster.start()
+        for node in nodes:
+            node.query("CREATE DATABASE IF NOT EXISTS test")
+            node.query("CREATE DATABASE IF NOT EXISTS atest")
+            node.query("CREATE DATABASE IF NOT EXISTS ztest")
+            node.query("CREATE TABLE test.source(x UInt64, y UInt64) ENGINE=Log")
+            node.query("INSERT INTO test.source VALUES (5,6)")
+
+            node.query("CREATE DICTIONARY test.dict(x UInt64, y UInt64) PRIMARY KEY x "\
+                        "SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'source' DB 'test')) "\
+                        "LAYOUT(FLAT()) LIFETIME(0)")
+        yield cluster
+
+    finally:
+        cluster.shutdown()
+
+
+@pytest.fixture(autouse=True)
+def cleanup_after_test():
+    try:
+        yield
+    finally:
+        for node in nodes:
+            node.query("DROP DICTIONARY IF EXISTS test.adict")
+            node.query("DROP DICTIONARY IF EXISTS test.zdict")
+            node.query("DROP DICTIONARY IF EXISTS atest.dict")
+            node.query("DROP DICTIONARY IF EXISTS ztest.dict")
+            node.query("DROP TABLE IF EXISTS test.atbl")
+            node.query("DROP TABLE IF EXISTS test.ztbl")
+            node.query("DROP TABLE IF EXISTS atest.tbl")
+            node.query("DROP TABLE IF EXISTS ztest.tbl")
+            node.query("DROP DATABASE IF EXISTS dict_db")
+
+
+@pytest.mark.parametrize("node", nodes)
+def test_dependency_via_implicit_table(node):
+    d_names = ["test.adict", "test.zdict", "atest.dict", "ztest.dict"]
+    for d_name in d_names:
+        node.query("CREATE DICTIONARY {}(x UInt64, y UInt64) PRIMARY KEY x "\
+                   "SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'dict' DB 'test')) "\
+                   "LAYOUT(FLAT()) LIFETIME(0)".format(d_name))
+
+    def check():
+        for d_name in d_names:
+            assert node.query("SELECT dictGet({}, 'y', toUInt64(5))".format(d_name)) == "6\n"
+
+    check()
+
+    # Restart must not break anything.
+    node.restart_clickhouse()
+    check()
+
+
+@pytest.mark.parametrize("node", nodes)
+def test_dependency_via_explicit_table(node):
+    tbl_names = ["test.atbl", "test.ztbl", "atest.tbl", "ztest.tbl"]
+    d_names = ["test.other_{}".format(i) for i in range(0, len(tbl_names))]
+    for i in range(0, len(tbl_names)):
+        tbl_name = tbl_names[i]
+        tbl_database, tbl_shortname = tbl_name.split('.')
+        d_name = d_names[i]
+        node.query("CREATE TABLE {}(x UInt64, y UInt64) ENGINE=Dictionary('test.dict')".format(tbl_name))
+        node.query("CREATE DICTIONARY {}(x UInt64, y UInt64) PRIMARY KEY x "\
+                   "SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE '{}' DB '{}')) "\
+                   "LAYOUT(FLAT()) LIFETIME(0)".format(d_name, tbl_shortname, tbl_database))
+
+    def check():
+        for d_name in d_names:
+            assert node.query("SELECT dictGet({}, 'y', toUInt64(5))".format(d_name)) == "6\n"
+
+    check()
+
+    # Restart must not break anything.
+    node.restart_clickhouse()
+    check()
+
+
+@pytest.mark.skip(reason="TODO: should be fixed")
+@pytest.mark.parametrize("node", nodes)
+def test_dependency_via_dictionary_database(node):
+    node.query("CREATE DATABASE dict_db ENGINE=Dictionary")
+
+    d_names = ["test.adict", "test.zdict", "atest.dict", "ztest.dict"]
+    for d_name in d_names:
+        node.query("CREATE DICTIONARY {}(x UInt64, y UInt64) PRIMARY KEY x "\
+                   "SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'test.dict' DB 'dict_db')) "\
+                   "LAYOUT(FLAT()) LIFETIME(0)".format(d_name))
+
+    def check():
+        for d_name in d_names:
+            assert node.query("SELECT dictGet({}, 'y', toUInt64(5))".format(d_name)) == "6\n"
+
+    check()
+
+    # Restart must not break anything.
+    node.restart_clickhouse()
+    check()