Catalog data file gets broken after compaction #590
Comments
novoj
added a commit
that referenced
this issue
May 31, 2024
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
novoj
added a commit
that referenced
this issue
Jun 1, 2024
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
novoj
added a commit
that referenced
this issue
Jun 1, 2024
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
novoj
added a commit
that referenced
this issue
Jun 1, 2024
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
novoj
added a commit
that referenced
this issue
Jun 1, 2024
…ken-after-compaction fix(#590): Catalog data file gets broken after compaction
novoj
added a commit
that referenced
this issue
Jun 1, 2024
…ken-after-compaction fix(#590): Catalog data file gets broken after compaction
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
The text was updated successfully, but these errors were encountered: