Cannot start clickhouse server

[iulianroman]

I have a clickhouse setup which is using zookeeper for data replication. One of the nodes experienced some storage issues and after restart clickhouse server does not start.
In the error log i have the following message:

Application: DB::ErrnoException: Allocator: Cannot realloc from 16.00 MiB to 0.00 B., errno: 2, strerror: No such file or directory: Cannot attach table graphite.data from metadata file /clickhouse/data/metadata/graphite/data.sql from query ATTACH TABLE data (Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/smart/graphite.data', '{replica}', Date, (Path, Time), 8192, 'graphite_rollup')

Any idea how can it be fixed ?

[cangyin]

Can you possibly post the full stack trace ?

[iulianroman]

Can you possibly post the full stack trace ?

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) @ 0x102e0d8c in /usr/bin/clickhouse
1. DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, int, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > const&) @ 0x8f31b77 in /usr/bin/clickhouse
2. DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, int) @ 0x8f3155e in /usr/bin/clickhouse
3. Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x8f6a26c in /usr/bin/clickhouse
4. DB::DataTypeString::deserializeBinary(DB::IColumn&, DB::ReadBuffer&) const @ 0xcdcf47c in /usr/bin/clickhouse
5. DB::IMergeTreeDataPart::loadIndex() @ 0xd70ef39 in /usr/bin/clickhouse
6. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0xd70f0d9 in /usr/bin/clickhouse
7. ? @ 0xd73f982 in /usr/bin/clickhouse
8. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x8f51c4b in /usr/bin/clickhouse
9. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda1'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda1'()&&...)::'lambda'()::operator()() const @ 0x8f528c4 in /usr/bin/clickhouse
10. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8f50b4b in /usr/bin/clickhouse
11. ? @ 0x8f4f00f in /usr/bin/clickhouse
12. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
13. __clone @ 0x121a3f in /lib/x86_64-linux-gnu/libc-2.27.so
    (version 20.3.4.10 (official build))
    2023.01.19 13:05:11.386159 [ 2292 ] {} Application: DB::ErrnoException: Allocator: Cannot realloc from 16.00 MiB to 0.00 B., errno: 2, strerror: No such file or directory: Cannot attach table graphite.data from metadata file /clickhouse/data/metadata/graphite/data.sql from query ATTACH TABLE data (Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/smart/graphite.data', '{replica}', Date, (Path, Time), 8192, 'graphite_rollup')

[cangyin]

From the stack trace, file "primary.idx" in some data parts may be missing or is lost.

[iulianroman]

From the stack trace, file "primary.idx" in some data parts may be missing or is lost.

Any idea how can I repair that ? I could even start with an empty db because i hope it will be replicated from the other systems in the cluster.