Parts metadata cache and failing mutation

[tavplubix]

https://s3.amazonaws.com/clickhouse-test-reports/0/2079699f7542527d62cbaed828940d54b0ca00a2/stress_test__undefined__actions_.html

BC check in Stress Tests sometimes fail with

2022.07.13 10:26:18.097238 [ 1051250 ] {} <Error> MutateFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 47. DB::Exception: Missing columns: 'v3' while processing query: 'v3, k, v1, v2, p', required columns: 'v3' 'k' 'v1' 'v2' 'p', maybe you meant: ['v1','k','v1','v2','p']. (UNKNOWN_IDENTIFIER), Stack trace (when copying this message, always include the lines below):
2022.07.13 10:26:18.098966 [ 1051240 ] {} <Error> MutateFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 47. DB::Exception: Missing columns: 'v3' while processing query: 'v3, k, v1, v2, p', required columns: 'v3' 'k' 'v1' 'v2' 'p', maybe you meant: ['v1','k','v1','v2','p']. (UNKNOWN_IDENTIFIER), Stack trace (when copying this message, always include the lines below):
2022.07.13 10:26:18.101742 [ 1051251 ] {} <Error> MutateFromLogEntryTask: virtual bool DB::ReplicatedMergeMutateTaskBase::executeStep(): Code: 47. DB::Exception: Missing columns: 'v3' while processing query: 'v3, k, v1, v2, p', required columns: 'v3' 'k' 'v1' 'v2' 'p', maybe you meant: ['v1','k','v1','v2','p']. (UNKNOWN_IDENTIFIER), Stack trace (when copying this message, always include the lines below):

i.e. some mutation cannot be executed (slightly related to #39174). This mutation was started by the test 01233_check_table_with_metadata_cache. Logs:
by table id: https://pastila.nl/?024eb85d/65a0e50dba5aafb52d0ecec2cf2d5737
by query ids: https://pastila.nl/?002a7f2e/7505e0215864bbb1ea3da78c4a0151db

The second mutation (MODIFY TTL + implicit MATERIALIZE TTL) is trying to read from column that was dropped by the first mutation (DROP COLUMN v3). The error persists after server restart, so I suppose metadata was corrupted. Probably not metadata itself, but metadata cache.

cc: @taiyang-li, @CurtizJ

[taiyang-li]

@tavplubix Thanks for your issue. I'll try to fix it soon

[alexey-milovidov]

We decided to remove the metadata cache instead.