Got 'invalid chunk checksum' error from query

[mecseid]

Describe the bug
When a chunk gets corrupted, then Loki sends back an invalid chunk checksum, although the other chunks are readable.

I saw a similar issue here: #8564

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (v2.7.4)
2. Query some log where the chunk is corrupted (like damaged or 0-sized)

Expected behavior
I think two solutions are fine as well:

1. Delete the unreadable chunks with user confirmation, or over some admin API
2. Skip the unreadable chunks and send back a warning message

The first one is preferred over the other because the restore operation is always an option (if has some kind of backup), but the end user doesn't want to always see a warning message about invalid chunks.

Two side note questions about this:

• Does the compactor or table manager will find and mark these chunks if they reached the retention time?
• When chunk deletion runs, then it will fail if a chunk missing or corrupted?

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm