Details on the litematica format?

[wgaylord]

Wondering if there are any details or documentation on the format for usage in making tools for working with them outside of minecraft.

[maruohon]

There isn't really any proper documentation on the format. It's kind of similar to the Sponge schematic format I believe, and it's also similar to the vanilla Chunk data format in 1.13+.

For the full implementation, look at https://github.com/maruohon/litematica/blob/master/src/main/java/fi/dy/masa/litematica/schematic/LitematicaSchematic.java

Also this screenshot from NBTExplorer of one of the schematic files should give a nice overview of the general structure. One thing to note is that since Litematica supports an arbitrary number of "sub-regions" in each schematic, the actual data for each of those is contained inside the per-region compound tags inside the Regions tag.

[wgaylord]

Okay. Very interesting. I am not familiar with the sponge schematic format at all. (I have done work with the old schematic format). Was going to work on a python api for reading and writing this type for use with my one program.

…

On Aug 11, 2019, at 8:35 PM, Matti Ruohonen ***@***.***> wrote: There isn't really any proper documentation on the format. It's kind of similar to the Sponge schematic format I believe, and it's also similar to the vanilla Chunk data format in 1.13+. For the full implementation, look at https://github.com/maruohon/litematica/blob/master/src/main/java/fi/dy/masa/litematica/schematic/LitematicaSchematic.java Also this screenshot from NBTExplorer of one of the schematic files should give a nice overview of the general structure. One thing to note is that since Litematica supports an arbitrary number of "sub-regions" in each schematic, the actual data for each of those is contained inside the per-region compound tags inside the Regions tag. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[maruohon]

Okay then I guess another useful detail is how the blocks are actually stored. So that part is basically the same as how vanilla stores Chunk data in 1.13+.

There is a "BlockStateContainer", which uses a block state palette to encode the values. So basically the palette allocates numeric IDs starting from 0, to each unique block state that is stored within that BlockStateContainer. One important detail to note is that Litematica always allocates minecraft:air as id 0, even if the schematic doesn't have any air!

In Litematica's case the container starts out at 2 bits per entry, and the number of bits required per block position increases by one bit each time the palette basically uses up all the available IDs within the current bit width. So if the container, ie. in this case one schematic sub-region has for example 75 unique/different block states, then it needs 7 bits per block position. That block state data is bit-packed and stored as a long array called BlockStates. The BlockStatePalette holds the block name and properties for each encoded ID, so the list index within the palette is the ID stored in the long array.

[wgaylord]

So it stores the blocks similar to the chunk packet format of storing them using a long array bit packed. Was expecting something like the old schematic format using just nested array type stuff.

…

On Aug 11, 2019, at 8:58 PM, Matti Ruohonen ***@***.***> wrote: Okay then I guess another useful detail is how the blocks are actually stored. So that part is basically the same as how vanilla stores Chunk data in 1.13+. There is a "BlockStateContainer", which uses a block state palette to encode the values. So basically the palette allocates numeric IDs starting from 0, to each unique block state that is stored within that BlockStateContainer. In Litematica's case the container starts out at 2 bits per entry, and the number of bits required per block position increases by one bit each time the palette basically uses up all the available IDs within the current bit width. So if the container, ie. in this case one schematic sub-region has for example 75 unique/different block states, then it needs 7 bits per block position. That block state data is bit-packed and stored as a long array called BlockStates. The BlockStatePalette holds the block name and properties for each encoded ID, so the list index within the palette is the ID stored in the long array. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[SunSerega]

Btw, why is it array of 8-byte integers and not an array of bytes? Wouldn't it be easier to work with raw byte array? Or is this, like, java specific optimization?

[wgaylord]

I noticed something strange and I want to ask. In the size of a region one of the dimensions are negative. What does that mean?

[maruohon]

It depends on in which corner the primary position (the red corner) is when doing the Area Selection.

Litematica will preserve all that relative positioning of the corners from the Area Selection sub-region boxes. That primary corner is where the position from the schematic/placement origin will point to, and if the sizes are negative, then the second corner goes towards the negative coordinate on a given axis. In other words, unless the red corner is the "minimum" on each axis, then some of the dimensions will be negative.

This design decision however meant that there is some extra complexity in some of the coordinate and position handling methods in the mod... Because the block state containers use normal simple 0 .. (size - 1) ranges, this means that the container has to be "overlayed" correctly on top of the sub-region placement, ie. the block state container's 0, 0, 0 corner is not at the sub-region box's primary corner unless that corner is at the minimum corner on each axis.

One thing to note is that there is no size of -1, as it would be the same as 1, ie. the corners are at the same position on that axis. So the negatives "start" from -2.

[wgaylord]

Not sure I am going to deal with that. Didn't think I would have to handle the Long array indexing specially based on the size problems. Also I currently have no idea how I am going to deal with the multiple sub-regions.

Also just wondering why did you decide to go with a custom format vs using Sponge's?

Didn't mean to click close.

[maruohon]

Depending on what you are doing, if you can just assume that the minimum corner is the primary corner, then you don't need to deal with that negative size and offset stuff. And for the long array and the block state container, for that you basically just convert the size into a normal positive size. The weirdness just comes into play when you need to position the sub-region in relation to the schematic origin. And without rotations and mirroring (which don't exist at the schematic level, only at the in-game placement level), the positioning weirdness comes down to a simple offset on where the block state container is in relation to the sub-region origin.

And as to why I didn't use the Sponge format, is mostly because I needed the multiple sub-regions. Otherwise I think the format should be quite similar.

[wgaylord]

Ah okay. Sponge uses an array of varints to hold the blocks palette index. So no bit packing to deal with. Also seeing as for now I am just going to support a single region because I have no idea how to handle the concept of sub regions.

…

On Aug 12, 2019, at 7:05 PM, Matti Ruohonen ***@***.***> wrote: Depending on what you are doing, if you can just assume that the minimum corner is the primary corner, then you don't need to deal with that negative size and offset stuff. And for the long array and the block state container, for that you basically just convert the size into a normal positive size. The weirdness just comes into play when you need to position the sub-region in relation to the schematic origin. And without rotations and mirroring (which don't exist at the schematic level, only at the in-game placement level), the positioning weirdness comes down to a simple offset on where the block state container is in relation to the sub-region origin. And as to why I didn't use the Sponge format, is mostly because I needed the multiple sub-regions. Otherwise I think the format should be quite similar. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[maruohon]

If you are reading the schematics for some external use, then supporting one or more sub-regions is kinda the same. You just read the blocks from each sub-region to a larger volume of blocks in your program or whatever. Saving into the litematic format on the other hand would require you to somehow be able to create those smaller selections in your program, either automatically or manually. I guess for the usual pixel art generator type things and most other stuff, it would make sense to just use one simple large region around everything.

That's also more or less how the litematic -> schematic export works in the mod. It calculates the enclosing box size around all the sub-regions, and then it reads that enclosing box sized volume of stuff from the temporary world and saves that as the old-style schematic.