-
Notifications
You must be signed in to change notification settings - Fork 2
Features
This is LinearReader's main method of reducing a world's size. The .linear file format (created by xymb-endcrystalme, you can find their GitHub repo here) uses Zstd to optimize the region files on the disk, since Zstd compresses better than Zlib, which is what Minecraft uses for their Anvil file format. (The region files are the files stored in the region, entities, and poi folders in a world's save folder and in the DIM-1 and DIM1 folders.) The .linear format reduces storage usage considerably, usually by around 35%-50%.
Since the .linear format uses Zstd, there is a range of compression levels that may be used. LinearReader (initially, see the Idle Recompression entry below) compresses to the level 2 out of 22. Level 2 was chosen as the default since it still saves a good amount of storage space while also not being too hard on the CPU. The higher the compression level, the more work the CPU has to do.
Vanilla Minecraft cannot understand the .linear format, it needs to be told how to read it. Additionally, any other mod or tool that expects .mca files will not be able to read your world's .linear files. (LinearReader is made to be compatible with other mods, but some may depend on the .mca format too much to be compatible.) The basic fundamentals of LinearReader are that Minecraft can read .linear region files, and can write and save to .linear files. LinearReader accomplishes this by intercepting chunk data and saving it in the .linear format instead of Minecraft's .mca format. Minecraft will no longer understand the .mca format if LinearReader is installed. This also means that much of how chunks are saved has been altered and made a bit more complex, possibly increasing the likelihood of chunk corruption in edge cases. Therefore, either taking regular backups of your world(s) or enabling LinearReader's backup system (see below for details) is encouraged.
If LinearReader detects any .mca region files it will automatically convert them to the .linear format and remove the .mca files (as mentioned in the Getting Started page). See below for more information on the conversion.
LinearReader's main goal is to get the world size as low as possible and perform well while doing it. To achieve this, it uses two stages of compression: initial compression and recompression.
Initial compression refers to the Zstd compression done on region files during normal, active gameplay. The initial compression is by default at level 2 of 22. Since this compression is occurring during gameplay, the initial compression level is quite low by default in order to decrease the workload of the CPU, since Minecraft is largely a CPU based game. Having a lower initial compression level helps maintain higher server performance during normal gameplay.
This initial compression produces significantly smaller files than Minecraft's Zlib compression, so even this initial compression is saving a large chunk of storage space. However, the second part of LinearReader's compression system is where Zstd is pushed to its limit, all in the name of saving more storage space.
Recompression is the process of recompressing existing region files to the maximum compression level (22).
During recompression, all .linear files in the region, entities, and poi folders are scanned. For each file, the following criteria must be met in order for it to be recompressed, otherwise it will be skipped:
- The file is a valid file, in that it has the correct file signature and passes basic structural validation.
- The file is not "unstable". An "unstable" file is one that either is dirty, which means that it has unsaved changes, or it is currently flushing (being written) to disk.
- It is important to note that a region being in the cache does not mean that it cannot be recompressed. Only files that are dirty or being flushed are considered "unstable".
- The file is not already compressed to level 22.
- LinearReader checks again immediately before writing to ensure file is stable. If it is dirty or flushing, it is skipped.
Additionally, the following must also be met for recompression to begin/continue:
- The server must have enough free RAM. The minimum required free RAM to allow recompression defaults to 15%, but it is configurable in the config.
- The recompression must actually result in a smaller file. Therefore, after attempting recompression, LinearReader compares the new compressed size to the old one. If the new size is not smaller, it is discarded and the old file is left unchanged.
It is also worth noting that a short delay is introduced between files during recompression to reduce disk pressure and avoid spikes in I/O usage.
To check the status of the current recompression run /linearreader afk-compress.
Recompression is intended to occur when the server is quiet, since it puts stress on the CPU and can cause lag. There are two ways for recompression to be triggered:
Manual recompression is initiated by running /linearreader afk-compress start. As reflected in the command's name, it is meant to be run when the server is quiet and people are not actively playing. Manual recompression stops once the recompression has been completed or once the command /linearreader afk-compress stop is run.
After 20 minutes of no disk activity, or whatever time value is set for idleThresholdMinutes in the config, LinearReader will automatically begin recompressing all .linear files. If disk I/O resumes while idle recompression is running, LinearReader will cleanly stop recompression as soon as it finishes the current file, preventing partial writes or corruption.
The cache stores region files in RAM for easy access. Accessing a region file from disk is significantly slower than accessing it from RAM, so if a region file is in the cache, it will have faster loading times. Since the cache keeps region files in RAM, it reduces disk I/O during gameplay and provides smoother performance under load.
The cache system trades increased RAM usage for reduced disk I/O and improved performance.
Being in the cache does not prevent a region from being recompressed. Only dirty or actively flushing regions are skipped by the recompressor.
The cache has a (configurable) maximum size, and when it fills up it must evict existing regions to free memory for new ones. Evicted regions will need to be reloaded from the disk when later accessed. Dirty regions (regions with unsaved changes) must be flushed (saved) to the disk before being evicted. Flushes occur in batches or over time, reducing disk usage but keeping more data temporarily in memory.
For frequently accessed regions may be pinned to the cache so that they are never evicted. This is done by running the command /linearreader pin which pins the region the player is currently in. A specific region can also be pinned. For example, running /linearreader pin 0 -1 would pin the region file r.0.-1.linear. Pinned regions are never evicted from the cache, so pinning too many regions can significantly increase RAM usage.
Pinned regions are stored in the world save folder under data/linearreader/pinned_regions.txt.
For more information on the commands which deal with the cache, see the Commands page of this wiki.
Since LinearReader intercepts chunk data and saves it as .linear, and just generally messes with chunk saving, an automatic backup system felt like a needed feature. The backup system is intended to help recover most of a chunk's data in case of corruption or failed writes. The backups are not intended to be used to rollback the server to an earlier time, nor are they meant to be a super up-to-date save of your world. For those reasons, each region only keeps a single backup file that is updated over time rather than keeping multiple historical backups. The backup files are stored alongside the original .linear file, so next to the region file r.2.1.linear would be the backup file r.2.1.linear.bak.
The first backup of a region file is created automatically when no .bak exists for that region file yet, and the region has been flushed, and backups are enabled. Backup files are updated after N amount of saves (the value of N can be changed in the config), and backups must still be enabled in the config. Backups are not saved every flush so as to not hurt the game's performance. Backups are also done asynchronously so as to not block the gameplay thread.
The backup files are always compressed to level 22 so that they are as small as can be. However, even such files can take up a good amount of space, especially if there is a lot of them, which is why the backup system is toggle-able. To completely shut off the backup system, change backupEnabled in the config from true to false.
In the case that a main region file gets corrupted and cannot be used, LinearReader will automatically move the corrupted file to a folder titled corrupted and the backup file will be used in its stead. Also, if the creation of a backup file ever fails, it will not cause issues and LinearReader will try again to create one later.
On new worlds, initial backup generation can take a while. There have been many times during testing where it has taken multiple minutes and world relaunches to get all the backups for a small world to be created. The backup system will likely be getting an update soon which will hopefully make the entire system better, and also fix this issue.
Running /linearreader sync-backups will sync all backup files to the current .linear region files. It was mainly intended to be used after chunk pruning (see below), but it can also be used whenever desired. Running this command will also create backup files for any region file that is missing one.
Backup syncing usually takes a while to complete and will likely impact server performance.
When a world is launched with any .mca files in its region, entities, or poi folder, LinearReader will automatically convert those files into the .linear format. Conversion runs separately for dimension's region, entities, and poi folders. After a file's successful conversion, the .mca file and any associated external chunk files (.mcc files) are deleted. This conversion occurs before any chunks are read or written. The conversion process normally will only run when launching a preexisting world with LinearReader, but conversion will also occur during any world launch if .mca files are detected in the specified folders.
The conversion works by reading the .mca file's chunk data, writing the exact same chunk data to the .linear format. The .mca file is only deleted after the conversion completes successfully. Chunk data is read using Minecraft's built-in region system, ensuring compatibility with all standard region formats. Any failed conversions can be retried by relaunching the world/restarting the server. Additionally, if the .linear version of the .mca file already exists, the .mca file will just be deleted to avoid duplicate region files. Interrupting the conversion process will not corrupt chunk data.
The conversion process is parallelized so that it can get done quickly. However, conversion of large worlds may take some time. Progress is logged intermittently, so check the logs for the current status of the conversion.
LinearReader's default settings work well with normal gameplay, but can cause some issues with chunk pregeneration. In order to ensure that chunk pregeneration works well with LinearReader, certain settings are temporarily adjusted when pregeneration is detected via the standard log messages emitted by Distant Horizons and Chunky:
- The cache size is reduced. This lowers the cache's RAM usage, leaving more RAM for pregeneration.
- Backups are disabled to avoid unnecessary disk writes. Since everything is being rewritten during pregeneration anyway, having backups being created at the same time would double disk I/O for no benefit.
- The amount of regions per save tick is raised if necessary to prevent backlog.
LinearReader monitors server log messages in real time. When it detects the messages that indicate pregeneration has started those settings will automatically be adjusted until it detects that pregeneration has stopped. No manual configuration is required.
LinearReader's config file is not modified when pregeneration starts, rather its values are temporarily overridden in memory. As soon as pregeneration ends (or the server stops), overrides are removed and and LinearReader once again follows the values specified in the config.