-
Notifications
You must be signed in to change notification settings - Fork 2
Features
This is LinearReader's main method of reducing a world's size. The .linear file format (created by xymb-endcrystalme, you can find their GitHub repo here) uses Zstd to optimize the region files on the disk, since Zstd compresses better than Zlib, which is what Minecraft uses for their Anvil file format. (The region files are the files stored in the region, entities, and poi folders in a world's save folder and in the DIM-1 and DIM1 folders.) The .linear format reduces storage usage considerably, usually by around 35%-50%.
Since the .linear format uses Zstd, there is a range of compression levels that may be used. LinearReader (initially, see the Idle Recompression entry below) compresses to the level 4 out of 22. Level 4 was chosen as the default since it provides a good balance between storage savings and CPU usage during normal gameplay. Higher compression levels save more storage space, but also require more CPU time.
Vanilla Minecraft cannot understand the .linear format, it needs to be told how to read it. Additionally, any other mod or tool that expects .mca files will not be able to read your world's .linear files. (LinearReader is made to be compatible with other mods, but some may depend on the .mca format too much to be compatible.) The basic fundamentals of LinearReader are that Minecraft can read .linear region files, and can write and save to .linear files. LinearReader accomplishes this by intercepting chunk data and saving it in the .linear format instead of Minecraft's .mca format. Minecraft will no longer understand the .mca format if LinearReader is installed. This also means that much of how chunks are saved has been altered and made a bit more complex, possibly increasing the likelihood of chunk corruption in edge cases. Therefore, either taking regular backups of your world(s) or enabling LinearReader's backup system (see below for details) is encouraged.
If LinearReader detects any .mca region files it will automatically convert them to the .linear format and remove the .mca files (as mentioned in the Getting Started page). See below for more information on the conversion.
LinearReader's main goal is to get the world size as low as possible and perform well while doing it. To achieve this, it uses two stages of compression: initial compression and recompression.
Many compression and maintenance behaviors are adaptive rather than fixed. LinearReader continuously monitors runtime conditions such as flush pressure, write activity, cache churn, and overall server load in order to balance storage efficiency with gameplay performance.
Initial compression refers to the Zstd compression done on region files during normal, active gameplay. The initial compression is by default at level 4 of 22. Since this compression occurs during gameplay, the initial compression level is relatively low by default in order to reduce the workload of the CPU, since Minecraft is largely a CPU based game. Under sustained write or flush pressure, LinearReader may temporarily reduce the live compression level or adjust flush behavior in order to reduce CPU strain and keep up with active chunk writes. These files can later be recompressed back to level 22 once server conditions become more favorable.
This initial compression produces significantly smaller files than Minecraft's Zlib compression, so even this initial compression is saving a large chunk of storage space. However, the second part of LinearReader's compression system is where Zstd is pushed to its limit, all in the name of saving more storage space.
Recompression is the process of recompressing existing region files to the maximum compression level (22).
During recompression, all .linear files in the region, entities, and poi folders are scanned. For each file, the following criteria must be met in order for it to be recompressed, otherwise it will be skipped:
- The file is a valid file, in that it has the correct file signature and passes basic structural validation.
- The file is not "unstable". An "unstable" file is one that either is dirty, which means that it has unsaved changes, or it is currently flushing (being written) to disk.
- It is important to note that a region being in the cache does not mean that it cannot be recompressed. Only files that are dirty or being flushed are considered "unstable".
- The file is not already compressed to level 22.
- LinearReader checks again immediately before writing to ensure file is stable. If it is dirty or flushing, it is skipped.
Additionally, the following must also be met for recompression to begin/continue:
- The server must have enough free RAM. The minimum required free RAM to allow recompression defaults to 15%, but it is configurable in the config.
- The recompression must actually result in a smaller file. Therefore, after attempting recompression, LinearReader compares the new compressed size to the old one. If the new size is not smaller, it is discarded and the old file is left unchanged.
Recompression priority is adaptive rather than strictly sequential. Colder, more stable, and maintenance-heavy regions are generally prioritized before highly active regions.
It is also worth noting that a short delay is introduced between files during recompression to reduce disk pressure and avoid spikes in I/O usage.
To check the status of the current recompression run /linearreader afk-compress.
Recompression is designed to run more aggressively during quieter periods and more conservatively during sustained server load. LinearReader continuously evaluates runtime conditions and maintenance pressure in order to decide when recompression work should be performed. Since active gameplay may temporarily prioritize throughput over compression ratio, recompression is responsible for eventually bringing regions back to maximum compression density. There are two ways for recompression to be triggered:
Manual recompression is initiated by running /linearreader afk-compress start. As reflected in the command's name, it is meant to be run when the server is quiet and people are not actively playing. Manual recompression stops once the recompression has finished or once the command /linearreader afk-compress stop is run.
After 20 minutes of no disk activity, or whatever time value is set for idleThresholdMinutes in the config, LinearReader will automatically begin recompressing all .linear files. If disk I/O resumes while idle recompression is running, LinearReader will cleanly stop recompression as soon as it finishes the current file, preventing partial writes or corruption.
Note
The idle recompressor requires that practically zero chunk disk activity is occurring. This means that the idle recompressor will basically never run on a singleplayer world, and that on dedicated servers it will only run when no players are online. Manual recompression is the best way to ensure that recompression actually happens.
The cache stores region files in RAM for easy access. Accessing a region file from disk is significantly slower than accessing it from RAM, so if a region file is in the cache, it will have faster loading times. Since the cache keeps region files in RAM, it reduces disk I/O during gameplay and provides smoother performance under load.
The cache system trades increased RAM usage for reduced disk I/O and improved performance.
Being in the cache does not prevent a region from being recompressed. Only dirty or actively flushing regions are skipped by the recompressor.
The cache has a (configurable) maximum size, and when it fills up it must evict existing regions to free memory for new ones. Evicted regions will need to be reloaded from the disk when later accessed. Dirty regions (regions with unsaved changes) must be flushed (saved) to the disk before being evicted. Flushes occur in batches or over time, reducing disk usage but keeping more data temporarily in memory.
Cache maintenance behavior is adaptive rather than fixed. LinearReader monitors factors such as cache churn, pinned-region count, write pressure, and overall server activity in order to decide how aggressively regions should be flushed or evicted.
Frequently accessed regions may be pinned to the cache so that they are never evicted. This is done by running the command /linearreader pin which pins the region the player is currently in. A specific region can also be pinned. For example, running /linearreader pin 0 -1 would pin the region file r.0.-1.linear. Pinned regions are never evicted from the cache, so pinning too many regions can significantly increase RAM usage.
Pinned regions are stored in the world save folder under data/linearreader/pinned_regions.txt.
For more information on the commands which deal with the cache, see the Commands page of this wiki.
Since LinearReader intercepts chunk data and saves it as .linear, and just generally messes with chunk saving, an automatic backup system felt like a needed feature. The backup system is intended to help recover most of a chunk's data in case of corruption or failed writes. The backups are not intended to be used to rollback the server to an earlier time, nor are they meant to be a super up-to-date save of your world. For those reasons, each region only keeps a single backup file that is updated over time rather than keeping multiple historical backups. The backup files are stored in backups subfolders within the region, entities, and poi folders.
The first backup of a region file is created automatically when no .bak exists for that region file yet, the region has been flushed, and backups are enabled.
After the initial backup is created, backup refreshing becomes activity-based. Rather than updating backups after a fixed number of saves, LinearReader now looks at several configurable factors such as:
- How many chunks changed since the previous backup
- How much total chunk data changed
- How old the current backup is
- Whether the region has been quiet long enough to safely refresh
This allows backups to stay reasonably up to date while avoiding unnecessary disk writes during heavy gameplay. Backup refresh scheduling is also integrated into LinearReader's broader runtime policy system so that backup activity scales more conservatively during periods of sustained server load.
The backup files are always compressed to level 22 so that they are as small as can be. However, even such files can take up a good amount of space, especially if there is a lot of them, which is why the backup system is toggle-able. To completely shut off the backup system, change backupEnabled in the config from true to false.
In the case that a main region file gets corrupted and cannot be used, LinearReader will automatically move the corrupted file to a folder titled corrupted and the backup file will be used in its stead. Also, if the creation of a backup file ever fails, it will not cause issues and LinearReader will try again to create one later.
Backup generation and refreshing are both done asynchronously so that gameplay threads are not blocked by backup writes. However, on very large or highly active worlds, it may still take some time before every region receives an initial backup.
Running /linearreader sync-backups will sync all backup files to the current .linear region files. It was mainly intended to be used after chunk pruning (see below), but it can also be used whenever desired. Running this command will also create backup files for any region file that is missing one. Since this command edits files, confirmation will be needed after running the initial command. To confirm, run /linearreader sync-backups confirm. The confirmation window duration is configurable in the config file.
Backup syncing usually takes a while to complete and will likely impact server performance. Additionally, busy regions (such as dirty or actively flushing regions) are skipped until they become stable.
LinearReader continuously monitors server activity and storage pressure in order to balance performance, disk I/O, compression density, and maintenance work.
Rather than relying entirely on fixed thresholds, LinearReader evaluates runtime signals such as write activity, flush pressure, cache churn, backlog growth, dirty-region pressure, and long-term server load patterns. These signals are used to dynamically adjust behaviors such as:
- Live compression levels
- Flush budgets
- Background maintenance intensity
- Cache trimming behavior
- Backup refresh scheduling
- Automatic recompression activity
This system is designed to keep gameplay responsive during periods of heavy activity while still gradually paying down deferred maintenance work such as low-compression regions or outdated backups during quieter periods.
When a world is launched with any .mca files in its region, entities, or poi folder, LinearReader will automatically convert those files into the .linear format. Conversion runs separately for each dimension's region, entities, and poi folders. After a file's successful conversion, the .mca file and any associated external chunk files (.mcc files) are deleted. This conversion occurs before any chunks are read or written. The conversion process normally will only run when launching a preexisting world with LinearReader, but conversion will also occur during any world launch if .mca files are detected in the specified folders.
The conversion works by reading the .mca file's chunk data, writing the exact same chunk data to the .linear format. The .mca file is only deleted after the conversion completes successfully. Chunk data is read using Minecraft's built-in region system, ensuring compatibility with all standard region formats. Any failed conversions can be retried by relaunching the world/restarting the server. Additionally, if the .linear version of the .mca file already exists, the .mca file will just be deleted to avoid duplicate region files. Interrupting the conversion process will not corrupt chunk data.
The conversion process is parallelized so that it can get done quickly. However, conversion of large worlds may take some time. Progress is logged intermittently, so check the logs for the current status of the conversion.
LinearReader allows you to export all of your world's current .linear files back to Minecraft's normal .mca format. These new .mca files are fully compatible with vanilla Minecraft and any external tools. During the export process, all chunk data from every dimension is copied to new files which are written as standard .mca files using Minecraft's built-in region system. The original .linear files are not modified at all when exporting, meaning that the export process cannot corrupt your .linear files. The exported .mca files are saved to a folder next to the world folder named <worldName>_mca_export. The normal folder structure for region files is preserved in the export folder.
Exporting runs in the background to not block gameplay and server performance. Also, the export process can be stopped at any point. If the export is restarted, the previous progress is preserved and any already exported files will be skipped. If any file fails to export it can be retried later.
Run the command /linearreader export-mca start to begin exporting all .linear files. Running the command /linearreader export-mca stop will stop the exporting where it is at until it is started again.
Exporting can take a while, especially for large worlds. Progress is logged, so check the logs to monitor it.
Chunk pruning refers to the removal of effectively useless chunks that contain no meaningful data. Safety is prioritized over results when it comes to deleting chunk data, so in order for a chunk to be deleted through chunk pruning it must meet the following criteria:
- The chunk was never entered by a player (
InhabitedTime = 0). - The chunk contains no entities.
- The chunk contains no block entities.
- The chunk has no structures.
- The chunk has no scheduled ticks/data.
If a chunk matches all of those criteria then it is considered safe to delete.
Since chunk pruning is a serious matter, confirmation is required before any chunk data is modified. Running /linearreader prune-chunks will do a dry-run of all region files, and return information such as how many chunks would be deleted of the total amount of chunks, and an approximate value for storage saved. It will also return relevant examples of affected regions, including the player's current region if applicable. After the dry-run, you have a limited amount of time to confirm via a command, after which the real chunk pruning will begin. The duration of this confirmation window is configurable in the config file.
Chunk pruning will skip any busy regions that are being written to. Additionally, if any affected region files change between the dry-run and the confirmation, the chunk pruning is cancelled for safety.
Since chunks must meet very strict criteria in order to be safely deleted, the storage gains from chunk pruning tend to be relatively small. It all depends on the world, like how large it is and how many untouched chunks there are. This is not a major optimization tool, rather it is meant to help save just a bit more storage space.
Chunk pruning works best when no players are online. There is also always the possibility that Minecraft will regenerate the pruned chunks, meaning that there is no guarantee that storage will be saved through chunk pruning.
Chunk pruning does not modify backup files. In order to remove the pruned chunks from any backup files, run the sync-backups command (see above).
LinearReader's default settings work well with normal gameplay, but can cause some issues with chunk pregeneration. In order to ensure that chunk pregeneration works well with LinearReader, certain settings are temporarily adjusted when pregeneration is detected via the standard log messages emitted by Distant Horizons and Chunky:
- The cache size is reduced. This lowers the cache's RAM usage, leaving more RAM for pregeneration.
- Backups are disabled to avoid unnecessary disk writes. Since everything is being rewritten during pregeneration anyway, having backups being created at the same time would double disk I/O for no benefit.
- The amount of regions per save tick is raised if necessary to prevent backlog.
LinearReader monitors server log messages in real time. When it detects the messages that indicate pregeneration has started those settings will automatically be adjusted until it detects that pregeneration has stopped. No manual configuration is required.
LinearReader's config file is not modified when pregeneration starts, rather its values are temporarily overridden in memory. As soon as pregeneration ends (or the server stops), overrides are removed and LinearReader once again follows the values specified in the config.
Compression behavior may temporarily prioritize throughput over compression ratio during pregeneration in order to keep up with large bursts of chunk writes.