-
Notifications
You must be signed in to change notification settings - Fork 2
Features
This page provides a detailed overview of all of LinearReader's features.
This is LinearReader's main method of reducing a world's size. The .linear file format (created by xymb-endcrystalme, you can find their GitHub repo here) uses Zstd to optimize the region files on the disk, since Zstd compresses better than Zlib, which is what Minecraft uses for their Anvil file format. (The region files are the files stored in the region, entities, and poi folders in a world's save folder.) The .linear format reduces storage usage considerably, usually by around 35%-50%.
Since the .linear format uses Zstd, there is a range of compression levels that may be used. LinearReader (initially, see the Idle Recompression entry below) compresses to the level 2 out of 22. Level 2 was chosen as the default since it still saves a good amount of storage space while also not being too hard on the CPU. The higher the compression level, the more work the CPU has to do.
Vanilla Minecraft cannot understand the .linear format, it needs to be told how to read it. Additionally, any other mod or tool that expects .mca files will not be able to read your world's .linear files. (LinearReader is made to be compatible with other mods, but some may depend on the .mca format too much to be compatible.) The basic fundamentals of LinearReader are that Minecraft can read .linear region files, and can write and save to .linear files. LinearReader accomplishes this by intercepting chunk data and saving it in the .linear format instead of Minecraft's .mca format. Minecraft will no longer understand the .mca format if LinearReader is installed. This also means that much of how chunks are saved has been altered and made a bit more complex, possibly increasing the likelihood of chunk corruption in edge cases. Therefore, either taking regular backups of your world(s) or enabling LinearReader's backup system (see below for details) is encouraged.
If LinearReader detects any .mca region files it will automatically convert them to the .linear format and remove the .mca files (as mentioned in the Getting Started page). This means that it is crucial to make a backup of any preexisting worlds before installing LinearReader. Since the auto-converter only runs when it detects .mca files, it usually only runs the first time a world is launched with LinearReader installed. However, if more .mca files ended up in the region, entities, or poi folder, then the auto-converter would run again. So unless more .mca files get added to those folders, the auto-converter will only run once. Also, depending on how large your world is when LinearReader is installed, the conversion may take some time. Check the logs for status updates from the auto-converter.
LinearReader's main goal is to get the world size as low as possible and perform well while doing it. To achieve this, it uses two stages of compression: initial compression and recompression.
Initial compression refers to the Zstd compression done on region files during normal, active gameplay. The initial compression is by default at level 2 of 22. Since this compression is occurring during gameplay, the initial compression level is quite low by default in order to decrease the workload of the CPU, since Minecraft is largely a CPU based game. Having a lower initial compression level helps maintain higher server performance during normal gameplay.
This initial compression produces significantly smaller files than Minecraft's Zlib compression, so even this initial compression is saving a large chunk of storage space. However, the second part of LinearReader's compression system is where Zstd is pushed to its limit, all in the name of saving more storage space.
Recompression is the process of recompressing existing region files to the maximum compression level (22).
During recompression, all .linear files in the region, entities, and poi folders are scanned. For each file, the following criteria must be met in order for it to be recompressed, otherwise it will be skipped:
- The file is a valid file, in that it has the correct file signature and passes basic structural validation.
- The file is not "unstable". An "unstable" file is one that either is dirty, which means that it has unsaved changes, or it is currently flushing (being written) to disk.
- It is important to note that a region being in the cache does not mean that it cannot be recompressed. Only files that are dirty or being flushed are considered "unstable".
- The file is not already compressed to level 22.
- LinearReader checks again immediately before writing to ensure file is stable. If it is dirty or flushing, it is skipped.
Additionally, the following must also be met for recompression to begin/continue:
- The server must have enough free RAM. The minimum required free RAM to allow recompression defaults to 15%, but it is configurable in the config.
- The recompression must actually result in a smaller file. Therefore, after attempting recompression, LinearReader compares the new compressed size to the old one. If the new size is not smaller, it is discarded and the old file is left unchanged.
It is also worth noting that a short delay is introduced between files during recompression to reduce disk pressure and avoid spikes in I/O usage.
To check the status of the current recompression run /linearreader afk-compress.
Recompression is intended to occur when the server is quiet, since it puts stress on the CPU and can cause lag. There are two ways for recompression to be triggered:
Manual recompression is initiated by running /linearreader afk-compress start. As reflected in the command's name, it is meant to be run when the server is quiet and people are not actively playing. Manual recompression stops once the recompression has been completed or once the command /linearreader afk-compress stop is run.
After 20 minutes of no disk activity, or whatever time value is set for idleThresholdMinutes in the config, LinearReader will automatically begin recompressing all .linear files. If disk I/O resumes while idle recompression is running, LinearReader will cleanly stop recompression as soon as it finishes the current file, preventing partial writes or corruption.
The cache stores region files in RAM for easy access. Accessing a region file from disk is significantly slower than accessing it from RAM, so if a region file is in the cache, it will have faster loading times. Since the cache keeps region files in RAM, it reduces disk I/O during gameplay and provides smoother performance under load.
The cache system trades increased RAM usage for reduced disk I/O and improved performance.
Being in the cache does not prevent a region from being recompressed. Only dirty or actively flushing regions are skipped by the recompressor.
The cache has a (configurable) maximum size, and when it fills up it must evict existing regions to free memory for new ones. Evicted regions will need to be reloaded from the disk when later accessed. Dirty regions (regions with unsaved changes) must be flushed (saved) to the disk before being evicted. Flushes occur in batches or over time, reducing disk usage but keeping more data temporarily in memory.
For frequently accessed regions may be pinned to the cache so that they are never evicted. This is done by running the command /linearreader pin which pins the region the player is currently in. A specific region can also be pinned. For example, running /linearreader pin 0 -1 would pin the region file r.0.-1.linear. Pinned regions are never evicted from the cache, so pinning too many regions can significantly increase RAM usage.
Pinned regions are stored in the world save folder under data/linearreader/pinned_regions.txt.
For more information on the commands which deal with the cache, see the Commands page of this wiki.