Add support for importing schematic files

[acs]

And it seems that the Minetest plugin has already some support for it:

https://forum.minetest.net/viewtopic.php?p=192890#p192890

[acs]

Analysis of Minetest supporting of Schematics:

https://forum.minetest.net/viewtopic.php?f=53&t=24651

And a great article:

https://www.instructables.com/id/Star-Wars-and-Star-Trek-in-Minecraft-with-Raspberr/

[acs]

Pretty related to: #18

[acs]

Start from here: https://github.com/arpruss/raspberryjammod/blob/master/mcpipy/import.py

Schematics seems to use https://minecraft.gamepedia.com/NBT_format for which there is the library https://github.com/twoolie/NBT/blob/master/nbt/nbt.py to read it.

https://wiki.vg/NBT

In rjm it is using a modified mcpi with special methods "def setBlockWithNBT(self, *args)". I would try to avoid that.

But all in all, the path seems to be clear.

[acs]

So next steps:

• Download and try to read some schematics to understand the contents and recheck that they are in NBT format
• Try to use standard libraries so all will work in the future
• In there are specific logic like the one in setBlockWithNBT include it in our library

[acs]

Ok, let's use as reference for getting schematics: https://www.planetminecraft.com/projects/?share=schematic It seems to be the place with more contents (worlds, schematics ...). But this one is better for schematics: https://www.minecraft-schematics.com. Let's focus in a simple one:

https://www.minecraft-schematics.com/schematic/14445/

And let's use this old one to check for nbt format evolution and because its simplicity:
https://www.minecraft-schematics.com/schematic/68/

[acs]

The first step is just to download it and to try to load it using the NBT python library in order to take a look to the contents. The NBT library:

https://github.com/twoolie/NBT
https://pypi.org/project/NBT/

This is the one used in rjm.

There is another one:

https://pypi.org/project/nbtlib
https://github.com/vberlier/nbtlib/

and more:

https://github.com/theJ8910/jnbt

[acs]

Ok, first step done:

nbtfile = nbt.NBTFile(self.file_path, 'rb')

The NBT contents are basically tags (The format is designed to store data in a tree structure made up of various tags). And yes, we have TAG_compound with tags inside it (leaves):

And the blocks data seems to be in:

[acs]

Ok, next step is to:

• Identify blocks: individual or groups? type of block? position?
• Draw the blocks inside MC/MT

[acs]

After a first analysis of the data, this format is not the same than the one supported in https://github.com/arpruss/raspberryjammod/blob/master/mcpipy/import.py. So time to do our own analysis.

Taking a look to the tags, the dimension of the schematic is

so we have: 17229 = 3366 blocks. And this is the size of:

So it is clear where the blocks are defined. And for each block in the above list, we have a value from 0 to 13. And we have a palette field with schematic['Palette'].tags with values from 0 to 13. So it seems it is defining the kind of block in minecraft. So we just need to understand the order iin which blocks_data is filled.

Let's try to create a block from the palette definition. Also, I am going to check with all schematics is the format is the same.

[acs]

Ok, the format inside the schematic files is different. So we need to detect the version inside the file. And for the old version, there is no info about the version. After reviewing different schematics, it seems that the format is the same from the start (2012) to 2018 at least. In 2020 it seems that it has changed. So let's focus first in the legacy format. It seems that the format is defined by WorldEdit. The change is described here and here and it is related with the changes in the ids of the blocks in Minecraft. The same problem that has stopped the development of the raspi plugin for Spigot. This is the new format: https://github.com/SpongePowered/Schematic-Specification

The change in blocks data that arrived with MC 1.13: https://minecraft.gamepedia.com/Java_Edition_1.13/Flattening: «1.13 modified the IDs ​​of many blocks, items, biomes, particles, paintings, entities, statistics and sound events, removed numeric IDs, added and removed some block states, and changed NBT tags and display names.[1] This change is known as "The Flattening"» 1.13 was released in July 18, 2018.

https://bugs.mojang.com/secure/attachment/151784/the_flattening.txt

[acs]

So let's focus first in the old format, the one supported in https://github.com/arpruss/raspberryjammod/blob/master/mcpipy/import.py, the one that works in MC pre 1.13. The name is "Pre-flattening" and the block ids and data re here: https://minecraft.gamepedia.com/Java_Edition_data_values/Pre-flattening

[acs]

And the format description is at: https://minecraft.gamepedia.com/Schematic_file_format.

• Blocks: Block IDs defining the terrain. 8 bits per block. Sorted by height (bottom to top) then length then width—the index of the block at X,Y,Z is (Y×length + Z)×width + X.
• Data: Block data additionally defining parts of the terrain. Only the lower 4 bits of each byte are used. (Unlike in the chunk format, the block data in the schematic format occupies a full byte per block.)

[acs]

But, how is the Raspi Python working in 1.15.2? Because of:

[04:46:45] [Server thread/WARN]: Initializing Legacy Material Support. Unless you have legacy plugins and/or data this is a bug!
[04:46:49] [Server thread/WARN]: Legacy plugin RaspberryJuice v1.11 does not specify an api-version.
[04:46:49] [Server thread/INFO]: [RaspberryJuice] Loading RaspberryJuice v1.11

This is why it works no matter that we are using blocks ids pre-flattening.

[acs]

The implementation it is easy: just read the blocks (Blocks), get the additional data (Data) (only 4 bits) and show it when setBlock at the right position. We can improve the rendering if we detect Y-stripes (or group of Y-stripes) that can be drawn together using setBlocks because they share the material.

The key to implement how to locate the data in Data array for a (x,y,z) is the formula:
i = x + size_x * z + (size_x * size_z) * y
In the doc it says: (Y×length + Z)×width + X which is the same.

[acs]

O, my first schematic drawn:

Compare with

[acs]

The above schematic was created in 2012 and we are recreating it 8 years later. So all the creations for 8 years can be reused with current status of the technology. I am not using yet the Data info.

The code to create it:

for y in range(0, size_y):
    for z in range(0, size_z):
        for x in range(0, size_x):
            i = x + size_x * z + (size_x * size_z) * y
            b = blocks[i]
            if b != 0:
                mc.setBlock(init_x + x, init_y + y, init_z + z, b)

pretty simple once you understand the format. Next step is to add Data information.

[acs]

Another one:

Compare with https://www.minecraft-schematics.com/schematic/11813/:

This one needs more care with the missing data info not supported yet.

[acs]

Ok, time to add support for the data field to build schematics. According to the doc about the format:

https://minecraft.gamepedia.com/Schematic_file_format
«Data: Block data additionally defining parts of the terrain. Only the lower 4 bits of each byte are used.» So we need to extract this 4 bits and convert them (to what type? int?) to the data of the block. If the 4 higher bits are 0, it should work without any additional change. But let's try.

[acs]

Much better now using the data info:

[acs]

Ok, let's do a test with a complex schematic before closing this issue:

https://www.minecraft-schematics.com/schematic/9676/#

It needed around 15-20s to do it but it is awesome. So with the current code, we have a decent support for loading schematics!

[acs]

Closing this issue!