New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Program does not open development beta Minecraft 18x10b #125

Open
erich666 opened this Issue Mar 9, 2018 · 35 comments

Comments

Projects
None yet
8 participants
@erich666

erich666 commented Mar 9, 2018

The latest Minecraft beta, 18x10b, makes world files that cannot be opened by Minutor, which crashes. I suspect 18x10a, https://minecraft.gamepedia.com/18w10a, has the same problem, and my guess is that restructuring for the coral plants is causing problems. Given that Minecraft "used up" all the basic block IDs 0-255 (except for 253 and 254), https://minecraft.gamepedia.com/Java_Edition_data_values#Block_IDs, I suspect they're now using some new extension method that breaks readers - that'll be fun for everyone.

I hope to look into this problem in a bit (Mineways is built off an old version of minutor), but thought I'd give a heads-up and see if anyone's figured it out already.

@mrkite

This comment has been minimized.

Owner

mrkite commented Mar 9, 2018

I haven't been keeping up with Minecraft, so this is definitely news to me. I'll have to find some time to install Minecraft and see where things go haywire.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Mar 9, 2018

As far as I know, the next major release from Minecraft may bring a completely new storage format, but development is not finished now. I remember reading something like: worlds are converted on first load, old structures will not be converted properly, no way back, ...

@erich666

This comment has been minimized.

erich666 commented Mar 13, 2018

I finally had some time to look at this today (after recovering from updating to VS 2017). Yes, they're having fun with the file format. For starters, Biomes went from 256 bytes to 256 integers, and it looks like the fourth (high) byte of the integer is the biome value.

New format is on the left, as seen in NBT Explorer:

image

Sections are now pretty interesting, with each 256 slice having a palette associated with it. BlockStates are now 256 long integers. These integers are actually just a bunch of half-bytes if the palette of blocks is <= 16 palette entries (i.e. there are only 16 block types or less in the 16x16x16 Y-slice). I haven't decoded it beyond this - looks like a big puzzle. Palette entries now also hold the "data" orientation etc. information. Basically, a lot of what you know has changed. Since palettes are expandable to any number of entries, this lets them break through the 256 block limit - no blocks now have ID values per se, just names and Properties.

I decided to see what would happen if I added enough blocks to reach 29 palette entries, so that 4 bits would not be enough for each entry. The BlockStates expands to 320 long ints, i.e., 5 bits per entry. So palette size determines BlockStates's size and decode. Looks pretty straightforward, though tedious to decode by bytes, a "bit grabber" (e.g., "I want 5 bits") for the BlockState is probably better, but might be slow... Here's an example:

image

Anyway, I see that a lot of blocks are missing from the beta at this point (e.g., birch blocks are just birch bark, no interior), so I suspect the final release of 1.13 is far far away. I might look again in a long while, once it's all settled down - no reason to write a lot of code to support something they may revise.

@nnooney

This comment has been minimized.

Contributor

nnooney commented Jul 18, 2018

I've updated Minutor to parse the new format described above. Thanks @erich666 for providing the format that allowed me to understand the new format. The pull request I referenced is very rough around the edges, but I am able to open worlds that I generated in the 1.13 snapshots.

More work needs to be done to update the Definition Manager with the new block format, and there's some room for code cleanup. However, it should be enough to load worlds in Minecraft 1.13+ (I certainly broke backwards compatibility).

@erich666

This comment has been minimized.

erich666 commented Jul 19, 2018

And just in case, Minecraft 1.13 is now officially out, as of yesterday https://minecraft.gamepedia.com/1.13

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Jul 27, 2018

I found some time to look into the changed file format and to test the 1.13 branch from @nnooney .

Mojang's changes look straight forward, but not all new tags are clear at the moment.
I have not reviewed the new code jet. Maps can be loaded and look more or less ok. It seems that some transparency is not working, e.g. flowers look different. Where did you find the values for watermodifier for the different ocean types?

But definitely some functionality is broken:

  • hostile mob spawn detection is not working at all (as expected)
  • all Structure overlays are missing (except some villages) (expected as now stored differently)
  • Entity overlays have to be cleaned up, but work mostly
  • 1.13 branch Minutor is crashing with old maps
  • old Minutor crashes with 1.13 maps
  • automatic definition file update will break old Minutor releases

I would appreciate if Minutor could handle old and new map file format in one single version. For most users this might not be necessary, as they just migrate to 1.13 and never back. But I normally use different (old) versions for a longer time.
In the next days I try to think about ways to get it compatible (together with @nnooney 's changes). Here my current brainstorm ideas for commenting:

  • stored tags in the files have version information, so we can distinguish them
  • basis for data structures in new Minutor versions should be based on the current format (so probably similar to the changes from @nnooney )
  • a translator/upgrade module is needed to convert old map data (numerical IDs) to the new format (string names), but only that part we need for rendering
  • this translator can be fixed code (no .json file), as numerical IDs will never change again
  • chunk reading together with block identification should be encapsulated more from map rendering. At the moment the rendering has knowledge about storage format. If we abstract that, the ChunkLoader could load arbitrary version map data and MapView can render based on an abstract interface.
  • to stay backwards compatible, we may have to rename all/some definition (.json) files. As otherwise older Minutor versions automatically upgrade to incompatible definitions.
@erich666

This comment has been minimized.

erich666 commented Jul 27, 2018

Since Tuesday I've been actively working on 1.13 translation for Mineways, so feel free to ask any questions (not that I necessarily have answers, but I know more than I did a few days ago). My code's a hack, but works to translate 1.13 content into 1.12.2 block and data values. For new blocks I'll be crowbarring these in by using high-order bits in the data values for new (made up by me) block IDs, since I use a full byte to store data values, not just 4 bits (lucky I was lazy that way). Anyway, my code is based on ancient minutor code, but it might be of help. I hope to check it in by next Wednesday, and am happy to share it with anyone before then.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Jul 27, 2018

Hi @erich666,

the approach you describe sounds to me to translate new 1.13 string names to numerical (old) ID values.
Obviously this needs less storage space. But I fear that this will (shorter or later) hit value range boundaries when new blocks are introduced. And maintaining a very long ID list without any official reference might result in much work and problems.

To stay (very long) future prove I would go the other way around:
Translating old ID+data into string names. This will consume more memory and maybe reduce performance, but we will be able to follow any number of new blocks. And I expect many many new blocks in the next releases. The main advantage would be that this translation will be done only once. In the future there would be no need for further translation.
Instead of the complete string we might use a hash of that string (which is more or less the same than an ID). Then we can use a hash-map to store all possible blocks. Access key is the hashed string and no longer the numerical ID. If we use the same string hashing function in Minutor and Mineways, we would get the same hash-ID.

@madmaxoft

This comment has been minimized.

Contributor

madmaxoft commented Jul 27, 2018

How about using something in between? The in-memory representation could more or less mimic the 1.13 format - each chunk's section would have an array of ints and a Palette object that would translate those numbers into instances of block type classes. Each block type class instance would thus represent all blocks that have the same string ID and the same properties (and an assigned color / transparency / ... for rendering). We'll probably be implementing something similar in Cuberite as well.

@mrkite

This comment has been minimized.

Owner

mrkite commented Jul 27, 2018

I agree.. plus translating old maps into new maps will prevent us from having to update the translator for every future update... it's a one-and-done thing.

@madmaxoft

This comment has been minimized.

Contributor

madmaxoft commented Jul 27, 2018

So this is the pseudocode for the renderer reading a color / transparency value:

const auto & section = chunk->section[y / 16];
auto blockType = section->palette->resolve(section->blocks[blockIndex(x, z)]);
// Render using blockType.color and blockType.transparency etc.
@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Jul 28, 2018

Problem: backwards compatibility with automatic definition file updates

Description: Minutor regularly checks for changed definition files here on GitHub. If we change the definition files, older Minutor versions will get them after some days. In case we introduce incompatible changes (as in the current m1.13 branch) this will break Minutor at the client PCs of the users. This will happen without them knowing why.

Solution:

  • We will NOT change the syntax in the file vanilla_ids.json
  • Instead changes from @nnooney will get into a new file, e.g. vanilla_blocks.json
  • The old file vanilla_ids.json can be used by the Converter. Up to now it contains IDs and data field to identify all old blocks. If we add a new tag ("flatname": "minecraft:air") we can define all translation in that file.
@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Jul 28, 2018

@mrkite , could you explain the initial idea behind the Definition Packs?

During my current tests to change the code in a backwards compatible way, I have trouble to maintain handling of several enabled/disabled Packs. In the current m1.13 branch this feature is already broken, as the packs are not searched for blocks.

For my understanding, in the past there could be ambiguities for block IDs coming from plugins and feature packs. Therefore we needed complicated lists, and priority search to find the blocks.
As Mojang is now using long names and namespaces (minecraft:air), I understood, that no collisions are possible anymore.

@mrkite

This comment has been minimized.

Owner

mrkite commented Jul 28, 2018

@EtlamGit yeah the initial idea was that plugins might be incompatible with each other and the user would know which plugins he was using on which maps and could turn them on and off as needed.

Plus, mod makers could host their own definition files so we wouldn't have to do all the work ourselves.

It never really panned out. If I had to do it over, I'd use protobuf to load definitions and stick it all in a single file.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Jul 29, 2018

@erich666, as you asked to ask questions...:

Do we render blocks differently if they have specific Properties?
E.g. I found it for restone_ore -> lit:false/true or water and lava with different level (distance). Probably a lit furnace or repeater will also be stored in that way. Do you think we need to use this kind of Properties stored for some blocks in the palette for rendering in a different color?
Example: https://minecraft.gamepedia.com/Mushroom_Block#Block_state

This is not possible at the moment, as only the block name is used to find the BlockIdentifier entry. We would also need to find a JSON description to define Properties resulting in different colors.

NBT Heighmaps
Did you find any description/decoding for the Heighmaps that are stored for each Chunk?
We could speedup rendering with knowing the highest non-air block per XZ position.

@erich666

This comment has been minimized.

erich666 commented Jul 30, 2018

Do we render blocks differently if they have specific Properties?

Yes, you'll perhaps want to do this, if you wish. While objects like carpet now have individual names, vs. data values for each color, there are some properties you may wish to use to modify objects' appearance: mushroom blocks could have a different top color (e.g., red vs. the "internal" color), maybe use fluid level to (slightly) change water color, use moisture for farmland, use age for wheat color. But, other than red and brown mushroom blocks there's not a large difference that I've noticed (yet - I'm still slogging through the blocks).

Did you find any description/decoding for the Heightmaps that are stored for each Chunk?

I don't use these in Mineways so I haven't looked.

@erich666

This comment has been minimized.

erich666 commented Aug 2, 2018

FWIW, I'm done (for now) with the 1.13 version of Mineways. My code's a hacky mess, for the most part you should avert your eyes. But, the nbt.cpp comments and snippets might be of use, as might the new biomes list.

If anyone figures out the rainfall for these biomes, and temperature values for the ocean biomes, please do let me know. I poked around a fair bit, I couldn't find them (yet).

@erich666

This comment has been minimized.

erich666 commented Aug 6, 2018

Instead of the complete string we might use a hash of that string (which is more or less the same than an ID). Then we can use a hash-map to store all possible blocks. Access key is the hashed string and no longer the numerical ID.

Yes, I did this for Mineways as I felt it didn't even need any testing to know it would be faster than testing against the 500+ item strings. My hash system is pretty trivial, see this code, and the 500+ names are listed just before that. I didn't check over the hashes generated to see if they were well-distributed - I should do that at some point, just to see if there's not some massive clump in a few lists. I did check the special case of "air" before doing any hash computation and search, figuring that a high percentage of blocks are likely to be air and deserve a quick out. Again, didn't test that assumption, yet.

If we use the same string hashing function in Minutor and Mineways, we would get the same hash-ID.

No need. Mineways has diverged (or, I should say, been left in the dust - I wish Mineways had multiprocess loading) by Minutor at this point, so use any hash function you like.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Aug 6, 2018

I browsed your code already, and yes it is different...

By the way, vanilla Minecraft 1.13 has 8581 different Blocks. And they simply number them consecutive. Which means the internal number will change with every version.
At the moment I try to find out which of these variants we can render the same way and for which we have to invent a way to distinguish them.

@erich666

This comment has been minimized.

erich666 commented Aug 6, 2018

By the way, vanilla Minecraft 1.13 has 8581 different Blocks.

The 594 block names I have listed here appear to be the entire set of unique names in 1.13. There are of course tons of variants - 1 turtle egg vs. 2 vs. 3 vs. 4, which way a stair piece is oriented, on and on - but I suspect that for mapping purposes these names are just about all you need. Your call, of course.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Aug 7, 2018

e.g. Sandstone Stairs have a different bottom texture. To be correct we would even have to detect if they are facing downwards. And it is more obvious for the mushroom stem or the direction of the wood Logs that they look different from top.
But you are right most of the variants are irrelevant for rendering. And I think we can start with only 593 Blocks itself to get a very good map.
In your list the block "bed" is not present in 1.13 anymore. Therefore the official list would be 593 Blocks long. But I will also include backwards compatible descriptions to load old worlds.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Aug 24, 2018

@thewierdnut did you notice that with 1.13 structures are not stored in *.dat files anymore, but inside the region files together with the Chunk data?
Do you have any plans to update the Structure loading code? Otherwise I would have to start with that task...

@cloudlion-ca

This comment has been minimized.

cloudlion-ca commented Sep 25, 2018

I've tried to use Minutor to load worlds in Microsoft Windows 10 and OSX using the available latest download files. I have tried to use that release with Minecraft version 1.13.1 .... I assume that's the latest version of Minecraft.

On Microsoft Windows 10 on three different computers the application loads and then just shuts down when I attempt to load a world. On OSX Minutor just loads up the checker-board screen and I can't see anything. I move the pointer around and it indicates unknown biomes.

I can't imagine that I'm the only person experiencing this adventure.

Please note: I've gone through the process of cleaning out the Window's registry of any reference to seancode. Downloading the available package and re-installing it with Admin set for Windows. That has no effect what so ever.

I really miss Minutor when it's working.

Thanks for all your efforts to the development crew.

Cheers!

@mrkite

This comment has been minimized.

Owner

mrkite commented Sep 25, 2018

The pre-built binaries don't work with 1.13. You instead have to build Minutor from source (in particular, the m13 branch). Eventually I'll release new pre-built binaries, I just haven't had time.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Sep 28, 2018

I just updated the Wiki with images rendered with latest color and Block definitions.

@AdriaanRenting

This comment has been minimized.

AdriaanRenting commented Oct 7, 2018

I cloned m1.13 and built it on my MacOs 1.13.6 It runs for a bit but once I start scrolling the map it crashes quite quickly. Built with clang (Apple LLVM version 9.1.0 (clang-902.0.39.2) and Qt5.5

In debug mode it gets to
memcpy(this->biomes, biomes->toIntArray(), sizeof(int)*biomes->length());
line 62 in chunk.cpp before going into some low level assembly in platformMemmove$VARIANT$Nehalem

I think with an "Unhandled toIntArray" error or message.

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

I'm not terribly familiar with Qt Creator or its debugger, I just happen to have it on my machine for one piece of software I occasionally need to support.

@AdriaanRenting

This comment has been minimized.

AdriaanRenting commented Oct 7, 2018

I seem to have narrowed down my problem to 4 region files in the Overworld (and probably one in the Nether).

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world. I could have some other kind of corruption going on, but it would then affect at least 5 region files without seemingly causing problems in the game.
Any suggestions on how I could go about debugging if this is a minutor m1.13 issue or something else?

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Oct 7, 2018

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

Minecraft 1.13 converts Chunks during loading them the first time after "upgrade". So there is no complete world upgrade process. Chunks get updated when you travel "near" them. Your world might contain Chunks with several different versions, depending when you last visited them. Therefore we now have different parsers for different versions in Minutor.

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world.

When I try to load a world where Minecraft is writing during that very moment, I also get crashes. This is normally due to corrupt data in some region files. By traveling through these Chunks/Regions and than quit Minecraft the regular way before loading the world in Minutor it should work again.

@AdriaanRenting

This comment has been minimized.

AdriaanRenting commented Oct 8, 2018

Is it possible that it is encountering chunks that have not been converted to the new format yet? Does minecraft do a conversion on the fly, or convert everything when you run the new version for the first time?

Minecraft 1.13 converts Chunks during loading them the first time after "upgrade". So there is no complete world upgrade process. Chunks get updated when you travel "near" them. Your world might contain Chunks with several different versions, depending when you last visited them. Therefore we now have different parsers for different versions in Minutor.

The region files I have trouble with, likely all contain at least some chunks that have been touched by 1.13 then. Everything pre1.13 seems to load. But quite a few post1.13 regions, including where the main buildings are on our server, seem to be fine as well.

Is there any way to easily narrow my issues down to a certain chunk or block? To me it seems that it is trying to read a chunk, but getting a wrong size for the memcpy or something like that. But I don't know enough about the chunk format to understand if my interpretation is correct.

When I remove those 4 region files from the Overworld, minutor works as expected on the rest of the world.

When I try to load a world where Minecraft is writing during that very moment, I also get crashes. This is normally due to corrupt data in some region files. By travelling through these Chunks/Regions and than quit Minecraft the regular way before loading the world in Minutor it should work again.

I only work on files that are copied from our server after I have stopped the minecraft server, so I assume that is not the issue. I understand that minutor cannot be reliable on files that are open and being written to by another program.

@jamesmule

This comment has been minimized.

jamesmule commented Oct 8, 2018

I'm having the same problem with a few of my worlds. It seems the biome array is supposed to be an int array in the new versions and a byte array in the old ones, but in the chunks causing the errors, the DataVersion tag that's checked for the case differentiation doesn't match the array type, i.e. it's a newer version, but the biome data is still saved in a byte array.

EDIT: Meaning the DataVersion tag is useless for checking what array type the biomes are saved in.

@AdriaanRenting

This comment has been minimized.

AdriaanRenting commented Oct 8, 2018

I'm having the same problem with a few of my worlds. It seems the biome array is supposed to be an int array in the new versions and a byte array in the old ones, but in the chunks causing the errors, the DataVersion tag that's checked for the case differentiation doesn't match the array type, i.e. it's a newer version, but the biome data is still saved in a byte array.

EDIT: Meaning the DataVersion tag is useless for checking what array type the biomes are saved in.

That matches with what I'm seeing.

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Oct 8, 2018

Could you provide the non-working region file?

@jamesmule

This comment has been minimized.

jamesmule commented Oct 8, 2018

I don't have any more chunks with the inconsistency, so I manually deleted the int array and replaced it with a byte array at 0,0 in this one, but that shouldn't matter.

r.0.0.mca.zip

@EtlamGit

This comment has been minimized.

Collaborator

EtlamGit commented Oct 9, 2018

A self modified region file is not the same than a real one...
I would like to figure out what other data is around that int array in case of these failing Chunks.

@AdriaanRenting any region files from you?

And we should move discussion about this issue to #143.

@AdriaanRenting

This comment has been minimized.

AdriaanRenting commented Oct 10, 2018

A self modified region file is not the same than a real one...
I would like to figure out what other data is around that int array in case of these failing Chunks.

@AdriaanRenting any region files from you?

And we should move discussion about this issue to #143.

I added what I have to #143

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment