When querying, err-mimir-store-consistency check failed, but the blocks with consistency errors are normal and theoretically should not report an error.

[hexiang1219]

Report an error

expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HJ4YEWDPGWNTS7XYTSEERMMR 01HJ4YAH2BA4ZYY48D505JA3RT 01HJ4YAZWZP8ZNA28R7B55BCTZ"

Information

compactor logs：
ts=2023-12-21T02:50:55.535155248Z caller=bucket_compactor.go:281 level=info component=compactor user=test groupKey=0@17241709254077376921-split-3_of_3-1703109600000-1703116800000 minTime="2023-12-20 22:00:00 +0000 UTC" maxTime="2023-12-21 00:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01HJ4YAZWZP8ZNA28R7B55BCTZ (min time: 1703109600000, max time: 1703116800000) 01HJ4YAH2BA4ZYY48D505JA3RT (min time: 1703109600000, max time: 1703116800000) 01HJ4YEWDPGWNTS7XYTSEERMMR (min time: 1703109600000, max time: 1703116800000)]"

ts=2023-12-21T02:55:36.285326449Z caller=bucket_compactor.go:329 level=info component=compactor user=test groupKey=0@17241709254077376921-split-3_of_3-1703109600000-1703116800000 minTime="2023-12-20 22:00:00 +0000 UTC" maxTime="2023-12-21 00:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YAZWZP8ZNA28R7B55BCTZ /data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YAH2BA4ZYY48D505JA3RT /data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YEWDPGWNTS7XYTSEERMMR]" duration=4m40.750126713s duration_ms=280750

ts=2023-12-21T02:57:59.085431742Z caller=compact.go:510 level=info component=compactor msg="compact blocks" count=3 mint=1703109600000 maxt=1703116800000 ulid=01HJ54SHB8VS8ZKKYCB1QF9565 sources="[01HJ4YAZWZP8ZNA28R7B55BCTZ 01HJ4YAH2BA4ZYY48D505JA3RT 01HJ4YEWDPGWNTS7XYTSEERMMR]" duration=2m22.800036065s shard=1_of_3

ts=2023-12-21T02:57:59.085484446Z caller=compact.go:510 level=info component=compactor msg="compact blocks" count=3 mint=1703109600000 maxt=1703116800000 ulid=01HJ54SHB8CGR7F60MHB9EZY3S sources="[01HJ4YAZWZP8ZNA28R7B55BCTZ 01HJ4YAH2BA4ZYY48D505JA3RT 01HJ4YEWDPGWNTS7XYTSEERMMR]" duration=2m22.800092965s shard=2_of_3

ts=2023-12-21T02:57:59.085497516Z caller=compact.go:510 level=info component=compactor msg="compact blocks" count=3 mint=1703109600000 maxt=1703116800000 ulid=01HJ54SHB8Q6MD72HGCXH310K3 sources="[01HJ4YAZWZP8ZNA28R7B55BCTZ 01HJ4YAH2BA4ZYY48D505JA3RT 01HJ4YEWDPGWNTS7XYTSEERMMR]" duration=2m22.800106087s shard=3_of_3

ts=2023-12-21T02:57:59.36678737Z caller=bucket_compactor.go:359 level=info component=compactor user=test groupKey=0@17241709254077376921-split-3_of_3-1703109600000-1703116800000 minTime="2023-12-20 22:00:00 +0000 UTC" maxTime="2023-12-21 00:00:00 +0000 UTC" msg="compacted blocks" new="[01HJ54SHB8VS8ZKKYCB1QF9565 01HJ54SHB8CGR7F60MHB9EZY3S 01HJ54SHB8Q6MD72HGCXH310K3]" blocks="[/data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YAZWZP8ZNA28R7B55BCTZ /data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YAH2BA4ZYY48D505JA3RT /data/compactor/compact/0@17241709254077376921-split-3_of_3-1703109600000-1703116800000/01HJ4YEWDPGWNTS7XYTSEERMMR]" duration=2m23.081365247s duration_ms=143081

ts=2023-12-21T02:58:18.684034891Z caller=bucket_compactor.go:558 level=info component=compactor user=test groupKey=0@17241709254077376921-split-3_of_3-1703109600000-1703116800000 minTime="2023-12-20 22:00:00 +0000 UTC" maxTime="2023-12-21 00:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01HJ4YEWDPGWNTS7XYTSEERMMR

ts=2023-12-21T02:58:18.696670438Z caller=block.go:170 level=info component=compactor user=test groupKey=0@17241709254077376921-split-3_of_3-1703109600000-1703116800000 minTime="2023-12-20 22:00:00 +0000 UTC" maxTime="2023-12-21 00:00:00 +0000 UTC" msg="block has been marked for deletion" block=01HJ4YEWDPGWNTS7XYTSEERMMR

ts=2023-12-21T04:03:33.218359434Z caller=blocks_cleaner.go:454 level=info component=cleaner user=test msg="deleted block marked for deletion" block=01HJ4YEWDPGWNTS7XYTSEERMMR

The data of this block on the object storage has also been cleared
ll 01HJ4YAH2BA4ZYY48D505JA3RT/chunks total 0

Environment

Mimir v2.9.0

What is the problem that causes the above phenomenon?

[dimitarvdimitrov]

I'd also check that the value of blocks_storage.bucket_store.ignore_deletion_mark_delay is lower than compactor.deletion_delay.

The compaction messages may be misleading. Another reason for failing consistency checks is unavailable store-gateways. Check whether there weren't any store-gateways out-of-memory or unavailable store-gateways in all 3 zones.

The full log line for the consistency check should contain more details about the reason of the failure. Can you post this too? I'm also curious to know when the consistency check failure happened relative to the compaction logs you posted.

[hexiang1219]

blocks_storage.bucket_store.ignore_deletion_mark_delay and compactor.deletion_delay use the default values

The Querier component only finds the following logs：
msg="failed consistency check" err=null
No related errors were found in other components.

I figured out there might be something wrong with the object storage I'm using. After a normal block is deleted, this block should no longer exist, but I found that previously deleted blocks on my object storage left empty directories of blocks.Then I deleted the empty directory corresponding to the block that failed to check consistency and restored it.

[dimitarvdimitrov]

msg="failed consistency check" err=null

ah, that's right. The null error was fixed by #6934

Then I deleted the empty directory corresponding to the block that failed to check consistency and restored it.

Can you clarify if deleting the empty directories resolved the consistency failure problems for the particular blocks? There is this issue which sounds related #5441, but the discussion linked there doesn't mention consistency check failures.

[aknuds1]

I figured out there might be something wrong with the object storage I'm using. After a normal block is deleted, this block should no longer exist, but I found that previously deleted blocks on my object storage left empty directories of blocks.

What kind of object storage are you using? Mimir doesn't support object storages with the notion of directories. Mimir expects S3 type of object storage, which only has the notion of files (ruling out the possibility of empty directories). It's also documented:

Note: Like Amazon S3, the chosen object storage implementation must not create directories. Grafana Mimir doesn’t have any notion of object storage directories, and so will leave empty directories behind when removing blocks. For example, if you use Azure Blob Storage, you must disable hierarchical namespace.

[hexiang1219]

ah, that's right. The null error was fixed by #6934

OK I will update it later

Can you clarify if deleting the empty directories resolved the consistency failure problems for the particular blocks? There is this issue which sounds related #5441, but the discussion linked there doesn't mention consistency check failures.

If the empty directory is indeed deleted, the consistency check problem will not occur.

What kind of object storage are you using? Mimir doesn't support object storages with the notion of directories. Mimir expects S3 type of object storage, which only has the notion of files (ruling out the possibility of empty directories). It's also documented:

I am using ChubaoFS S3. Its bottom layer is essentially a file system and has a directory concept. The above problems can be avoided by temporarily writing a scheduled task in the directory to clean up the empty directory. Storage will be switched uniformly later.

Thank you so much