Failed to correct prefix of column family [49]

[korthout]

Describe the bug

The following error was reported by Google Error Reporting:

 io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamilyCorrectionException: Failed to correct prefix of column family [49] DMN_DECISION_REQUIREMENTS_KEY_BY_DECISION_REQUIREMENT_ID_AND_VERSION, due to unexpected data in column family - key[0x00000000: 00 00 00 01 6E 00 00 00 01                      |....n....|
], value[0x00000000: 00 08 00 00 00 2A B5 E1                         |.....*..|
]

This is a bug on the CF 49 migrator.

One of the checks is wrong. It attempts to move the data that is supposed to be in this CF. It should be moved if it can be wrapped up as processInstanceKeyByProcessDefinitionKey, but it does so when that throws an exception.

This bug exists on all versions: main, 8.4.3, 8.3.7, 8.2.22, and 8.1.23.
But it will only surface on 8.2.22 and 8.1.23 because the column family is deprecated since 8.3, and all expected data is moved out by another migration task.

To Reproduce

• Start cluster with 8.2.21
• Deploy a DMN model
• Restart and upgrade to 8.2.22
• Partitions crash due to error

Expected behavior

Correct data in the column family should be left alone and not crash the migration task.

Log/Stacktrace

Full Stacktrace

 io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamilyCorrectionException: Failed to correct prefix of column family [49] DMN_DECISION_REQUIREMENTS_KEY_BY_DECISION_REQUIREMENT_ID_AND_VERSION, due to unexpected data in column family - key[0x00000000: 00 00 00 01 6E 00 00 00 01                      |....n....|
], value[0x00000000: 00 08 00 00 00 2A B5 E1                         |.....*..|
]
	at io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamily49Corrector.tryMoveDataToCorrectColumnFamily(ColumnFamily49Corrector.java:155) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamily49Corrector.lambda$correctColumnFamilyPrefix$0(ColumnFamily49Corrector.java:94) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$forEach$6(TransactionalColumnFamily.java:165) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.visit(TransactionalColumnFamily.java:397) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$forEachInPrefix$20(TransactionalColumnFamily.java:376) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.ColumnFamilyContext.withPrefixKey(ColumnFamilyContext.java:112) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.forEachInPrefix(TransactionalColumnFamily.java:360) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.forEachInPrefix(TransactionalColumnFamily.java:332) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$forEach$7(TransactionalColumnFamily.java:162) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.lambda$ensureInOpenTransaction$19(TransactionalColumnFamily.java:315) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInNewTransaction(DefaultTransactionContext.java:61) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.DefaultTransactionContext.runInTransaction(DefaultTransactionContext.java:33) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.ensureInOpenTransaction(TransactionalColumnFamily.java:314) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.forEach(TransactionalColumnFamily.java:160) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamily49Corrector.correctColumnFamilyPrefix(ColumnFamily49Corrector.java:70) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.to_8_2.DbColumnFamilyCorrectionMigrationState.correctColumnFamilyPrefix(DbColumnFamilyCorrectionMigrationState.java:32) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.DbMigrationState.correctColumnFamilyPrefix(DbMigrationState.java:327) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.to_8_2.ColumnFamilyPrefixCorrectionMigration.runMigration(ColumnFamilyPrefixCorrectionMigration.java:29) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.DbMigratorImpl.runMigration(DbMigratorImpl.java:166) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.DbMigratorImpl.handleMigrationTask(DbMigratorImpl.java:130) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.DbMigratorImpl.runMigrations(DbMigratorImpl.java:70) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.DbMigrationController.onRecovered(DbMigrationController.java:40) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.stream.impl.StreamProcessor.lambda$startProcessing$9(StreamProcessor.java:337) ~[zeebe-stream-platform-8.2.22.jar:8.2.22]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:?]
	at io.camunda.zeebe.stream.impl.StreamProcessor.startProcessing(StreamProcessor.java:337) ~[zeebe-stream-platform-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.stream.impl.StreamProcessor.lambda$onRetrievingWriter$8(StreamProcessor.java:320) ~[zeebe-stream-platform-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.stream.impl.StreamProcessor.chainSteps(StreamProcessor.java:288) ~[zeebe-stream-platform-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.stream.impl.StreamProcessor.lambda$chainSteps$5(StreamProcessor.java:297) ~[zeebe-stream-platform-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:28) ~[zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:96) ~[zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:47) [zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:109) [zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) [zeebe-scheduler-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:205) [zeebe-scheduler-8.2.22.jar:8.2.22]
Caused by: java.lang.IndexOutOfBoundsException: index=8 length=8 capacity=9
	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:2221) ~[agrona-1.17.2.jar:1.17.2]
	at org.agrona.concurrent.UnsafeBuffer.getLong(UnsafeBuffer.java:401) ~[agrona-1.17.2.jar:1.17.2]
	at io.camunda.zeebe.db.impl.DbLong.wrap(DbLong.java:27) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.db.impl.DbCompositeKey.wrap(DbCompositeKey.java:47) ~[zeebe-db-8.2.22.jar:8.2.22]
	at io.camunda.zeebe.engine.state.migration.to_8_2.corrections.ColumnFamily49Corrector.tryMoveDataToCorrectColumnFamily(ColumnFamily49Corrector.java:149) ~[zeebe-workflow-engine-8.2.22.jar:8.2.22]
	... 34 more 

Environment:

• Zeebe Version: 8.2.22 and 8.1.23 with a DMN model deployed
• Error