Something is seriously wrong in chunk #

[StevenBlair123]

{"@t":"2022-10-28T13:00:18.2562300+00:00","@mt":"Error in StorageChaser. Terminating...","@l":"Fatal","@i":3945788006,"@x":"System.Exception: Prefix/suffix length inconsistency: prefix length(196) != suffix length (0).\nActual pre-position: 45670073. Something is seriously wrong in chunk #199-199 (chunk-000199.000000).\n at

Eventstore version 22.10.6

Our server ran out of disk space, and once we reclaimed some space, the database refused to startup (i have included the full error below)

Is there anyway we can prevent this from happening (obviously not running out of disk space would be ideal).
It's really quite worrying we would lose a database like this.

As for the current database, can it be rescued. For example, can we make it ignore that particular .chk file?

Full error:

{"@t":"2022-10-28T13:00:18.2562300+00:00","@mt":"Error in StorageChaser. Terminating...","@l":"Fatal","@i":3945788006,"@x":"System.Exception: Prefix/suffix length inconsistency: prefix length(196) != suffix length (0).\nActual pre-position: 45670073. Something is seriously wrong in chunk #199-199 (chunk-000199.000000).\n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TFChunkReadSide.TryReadForwardInternal(ReaderWorkItem workItem, Int64 actualPosition, Int32& length, ILogRecord& record) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/TransactionLog/Chunks/TFChunk/TFChunkReadSide.cs:line 510\n at EventStore.Core.TransactionLog.Chunks.TFChunk.TFChunk.TFChunkReadSideUnscavenged.TryReadClosestForward(Int64 logicalPosition) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/TransactionLog/Chunks/TFChunk/TFChunkReadSide.cs:line 82\n at EventStore.Core.TransactionLog.Chunks.TFChunkReader.TryReadNextInternal(Int32 retries) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/TransactionLog/Chunks/TFChunkReader.cs:line 82\n at EventStore.Core.Services.Storage.ReaderIndex.IndexCommitter1.Init(Int64 buildToPosition) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/Services/Storage/ReaderIndex/IndexCommitter.cs:line 181\n at EventStore.Core.Services.Storage.IndexCommitterService1.Init(Int64 chaserCheckpoint) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/Services/Storage/IndexCommitterService.cs:line 91\n at EventStore.Core.Services.Storage.StorageChaser`1.ChaseTransactionLog() in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/Services/Storage/StorageChaser.cs:line 98","SourceContext":"EventStore.Core.Services.Storage.StorageChaser","ProcessId":19933,"ThreadId":19}
{"@t":"2022-10-28T13:00:18.2571893+00:00","@mt":"Exiting with exit code: {exitCode}.\nExit reason: {e}","@l":"Error","@i":3348423431,"exitCode":1,"e":"Error in StorageChaser. Terminating...\nError: Prefix/suffix length inconsistency: prefix length(196) != suffix length (0).\nActual pre-position: 45670073. Something is seriously wrong in chunk #199-199 (chunk-000199.000000).","SourceContext":"EventStore.Common.Utils.Application","ProcessId":19933,"ThreadId":19}

[StevenBlair123]

#3452

This fixed the problem for us.

[hayley-jean]

Thanks for confirming that the fix worked @StevenBlair123

We have an internal issue to prevent this from happening, but at the moment the only prevention is to not run out of disk space.
That said, this isn't a data-loss scenario since the corrupted appends were never acked back to the clients.

I'm going to leave this issue open in case anyone else runs into it - the workaround is to copy the chaser.chk over the truncate.chk like you would when restoring a backup. This will truncate the corrupted data.