Introduce per-table cache of ColumnsDescription for parts to reduce memory usage

[azat]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Introduce a per-table cache of ColumnsDescription for parts, reducing memory usage when tables contain many parts and many columns.

Details

Each part currently stores its own ColumnsDescription (information about all columns). This includes not only full column metadata (including comments), but also IHints, which are used for correcting user input and are not trivial to decouple.

When a table has many columns, this structure becomes large, and storing a full copy for every part leads to significant memory consumption.

In one production setup, 12 GiB out of 73 GiB of memory was occupied by ColumnsDescription objects for 20 K parts with 6.5 M part columns.

As another example, even on an empty server with merges stopped, metric_log alone could cause the server to consume roughly 512 MiB over a few hours while doing almost nothing.

[clickhouse-gh]

Workflow [PR], commit [e8f5478]

Summary: ❌

job_name	test_name	status	info	comment
Stateless tests (arm_asan, targeted)		failure
	03279_database_backup_database_disk_engine	FAIL	cidb
	03277_database_backup_database_file_engine	FAIL	cidb
BuzzHouse (amd_debug)		failure
	Buzzing result	failure	cidb

[rschu1ze]

I didn't read this PR in detail but based on the PR description, average readers won't understand how caching helps to reduce memory consumption. Are you saying that, previously the same ColumnDescription existed multiple times in memory, and now it exists once (in the cache) and we are somehow referencing it?

[azat]

Are you saying that, previously the same ColumnDescription existed multiple times in memory, and now it exists once (in the cache) and we are somehow referencing it?

Yes

I didn't read this PR in detail but based on the PR description, average readers won't understand how caching helps to reduce memory consumption.

I've updated the description to make it cleaner.

[azat]

I've rolled out to ci-logs, and the memory consumption reduced (though there is still more), and it is stable.

UPD: ColumnsDescriptionsCacheSize looks wrong

[azat]

UPD: ColumnsDescriptionsCacheSize looks wrong

It is actually correct, we should not have it too big, more or less the same size as number of tables, and it indeed looks similar