Added move_ttl_info to system.parts

[excitoon]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added move_ttl_info to system.parts in order to provide introspection of move TTL functionality.

[filimonov]

what with basic ttl info (for deletion)?

[filimonov]

Проблема 1 - как показать move_ttl. Варианты - массив туплов (не self-desctibing) или Nested (не всегда удобный, self-describing, используется в разных местах - например system.query_log.ProfileEvents)

Проблема 2 - как показать ttl для части. Мне кажется просто 2 колонки (ttl_min / ttl_max) в system.parts

Проблема 3 - как показать ttl для колонки внутри части. Мне кажется просто 2 колонки (ttl_min / ttl_max) в system.parts_columns

Nested - это по сути несколько массивов с одинаковым префиксом. Типа
move_ttl_info.name Array(String),
move_ttl_info.ttl_min Array(DateTime),
move_ttl_info.ttl_max Array(DateTime)
для Nested есть специальный синтакс - ARRAY JOIN move_ttl_info

Можно вынести это на обсуждение. Но порефакторить это потом будет никак. Как только люди начнут этим пользоваться - нужно будет чтоб структура была всегда такая иначе проблемы с обратной совместимостью.

[alex-zaitsev]

Agree, but it is probably sufficient to have two columns, not 3:

move_ttl_info.name Array(String),
move_ttl_info.ttl_minmax Array((DateTime, DateTime)),

[alexey-milovidov]

For table TTLs we already have:

TTL expr MOVE
TTL expr DELETE

and there are upcoming

TTL expr DELETE WHERE cond
TTL expr GROUP BY columns... SET col = agg(col), ...

Now suppose we will allow multiple table TTLs.

And we should not limit this new info for TTL MOVE, because it is rarely used feature that is still in experimental stage (see https://github.com/ClickHouse/ClickHouse/pulls/excitoon).

[alexey-milovidov]

1. ttl_info should not be only about moves.
2. it should not be in a single Tuple column, but in multiple separate columns instead.

[alexey-milovidov]

It's Ok to add info about MOVE TTL but only after we will already have info about ordinary TTL.
I propose to add ttl_info without distinguishing between MOVE and other TTLs.

[excitoon]

@alexey-milovidov Historically, there is no expression stored in part for DELETE TTL. If I merge move and delete TTLs into one array, what should I output for delete TTL as an expression?

[alexey-milovidov]

I don't understand why expression is stored for move ttl but not for delete ttl?
And why we need expression in ttl if we already have it in table definition?

[filimonov]

I don't understand why expression is stored for move ttl but not for delete ttl?

It was not used originally, and i think @excitoon kept it as is for backward compatibility. Otherwise older version will not be able to read ttl for parts created in a newer version (and not sure how it will fail, as the quite simple and stupid parser is used).

And why we need expression in ttl if we already have it in table definition?

If there are (let's say) 3 ttl move rules it's hard to distinguish them otherwise.

So for such ttl rules:

TTL FlightDate TO VOLUME 'fast',
   FlightDate + INTERVAL 3 YEAR TO VOLUME 'med',
   FlightDate + INTERVAL 5 YEAR TO VOLUME 'slow',
   FlightDate + INTERVAL 7 YEAR DELETE

it writes smth like

$ sudo cat /data/fast/data/default/ontime_chunk/2017_113_113_0/ttl.txt
ttl format version: 1
{
 "table": {
    "min":1710457200,
    "max":1710975600
 },
 "moves":[
     {"expression":"FlightDate","min":1489532400,"max":1490050800},
     {"expression":"plus(FlightDate, toIntervalYear(3))","min":1584226800,"max":1584745200},
     {"expression":"plus(FlightDate, toIntervalYear(5))","min":1647298800,"max":1647817200}
 ]
}

and there are upcoming
TTL expr DELETE WHERE cond
TTL expr GROUP BY columns... SET col = agg(col), ...

I'm don't understand the first case - what is the difference with existing TTL delete?
Can you share some 'specs' on that?

#10537

[filimonov]

Regarding the usability:

1. most of the user will use DELETE only, and storing it in the array / other complicated structures seem a bit overkill for that use case
2. if it will be stored in some complicated structures (array with expressions etc) it will be no clear way to get the list of parts which should be removed in X hour (which were not removed at a time) - i think that would be the most common use case (plus check if the ttl_info was properly updated)

Ie.

SELECT * FROM system.parts WHERE max_ttl < now();

is much clearer than

SELECT * FROM system.parts WHERE ttl_info.max_ttl[indexOf(ttl_info.expression,'plus(FlightDate, toIntervalYear(3))')] < now()

[alexey-milovidov]

Ok.

BTW, why delete ttl is Array? If we can only have one delete TTL for table, let's make it simple.

[alexey-milovidov]

I see that this is just a mistake.
Fixed in 8bde836.

[filimonov]

LGTM

[alexey-milovidov]

But there is no test for this feature :(

[vzakaznikov]

There is move_ttl_info.expression but there is no delete_ttl_info.expression?

[vzakaznikov]

Also, we wanted to have something like last_ttl and next_ttl. Are we going to implement this?

[xiaogaozi]

I have same question as @vzakaznikov asks.