Version 0.10.6

Removed limit on Block file sizes

• Added support for storing data in Block files that are larger than 2147483647 bytes (e.g. ~2.147GB) and fixed bugs that existed because of the previous limitation:
  • If a mutable Block exceeded the previous limit in memory it was not synced to disk and the storage engine didn't provide an error or warning, so indexing continued as normal. As a result, there was the potential for permanent data loss.
  • When a mutable Block failed to sync in the manner described above, the data held in the Block remained completely in memory, resulting in a memory leak.
• To accommodate the possibility of larger Block files, the BlockIndex now records position pointers using 8 bytes instead of 4. As a result, all Block files must be reindexed, which is automatically done when Concourse Server starts are new installation or upgrade.

Eliminated risks of data inconsistency caused by premature shutdown

• Fixed the logic that prevents duplicate data indexing when Concourse Server prematurely shuts down or the background indexing job terminates because of an unexpected error. The logic was previously implemented to address CON-83, but it relied on data values instead of data versions and was therefore not robust enough to handle corner cases descried in GH-441 and GH-442.
  • A concourse data repair CLI has been added to detect and remediate data files that are corrupted because of the abovementioned bugs. The CLI can be run at anytime. If no corrupt data files are detected, the CLI has no effect.
  • Upon upgrading to this version, as a precuation, the CLIs routine is run for each environment.

Other

• Fixed a bug that caused the default log_level to be DEBUG instead of INFO.