Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for 1.13 chunk format? #1454

Closed
msmollin opened this Issue May 15, 2018 · 91 comments

Comments

Projects
None yet
@msmollin
Copy link

msmollin commented May 15, 2018

Hello! I moved up to the 1.13 snapshots now that they support loading older worlds, and noticed Overviewer does not support the 1.13 chunk format. Has work begun on this already to support it? I'm decent at python and C code (although mostly data manipulation and not rendering code), but somewhat unfamiliar with Overviewer's codebase so any initial triage might be helpful from the maintainers.

Offending LOC:

# Check for versions of minecraft after the 17w47a changes

Research:
https://minecraft.net/en-us/article/minecraft-snapshot-17w47a
https://minecraft.gamepedia.com/17w47a
https://minecraft.gamepedia.com/Talk:Chunk_format#1.13_chunk_format_changes

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented May 16, 2018

Has work begun on this already to support it?

Nope, and it's generally agreed upon that it would be a ton of work to implement, but if someone wants to get started on it that'd be fantastic.

somewhat unfamiliar with Overviewer's codebase so any initial triage might be helpful from the maintainers.

Overviewer currently assumes hardcoded IDs and data bits in both the Python and the C code; this means it would require a fundamental refactor and rewrite of overviewer to change this. Furthermore, overviewer internally generates "fake" data bits for some blocks referred to as "ancil data", which is data changing how a block looks based on ancillary blocks.

Overviewer also doesn't make use of the block models Minecraft has, since Overviewer predates them by a long time. This means all sprites you see are essentially hardcoded PIL transforms of the textures. While not strictly needed to support 1.13, it's probably going to go into the direction of more complex block models down the road.

@bobfrankly

This comment has been minimized.

Copy link
Contributor

bobfrankly commented Jun 5, 2018

Ran across this article while looking into related stuff, looks like this Jupisoft guy has done some legwork on reading the new chunk formats. Sadly I'm out of my depth with this, so no idea if it helps or not.

@ChrisPittsley

This comment has been minimized.

Copy link

ChrisPittsley commented Jul 18, 2018

Nope, and it's generally agreed upon that it would be a ton of work to implement, but if someone wants to get started on it that'd be fantastic.

Are there plans to eventually implement 1.13 compatibility?

@msmollin

This comment has been minimized.

Copy link
Author

msmollin commented Jul 18, 2018

I started looking into it but, as described, it's a non-trivial implementation of a poorly documented format (Mojang's fault IMO) . So I was waiting to see if a better document of the format became available.

@fhavio

This comment has been minimized.

Copy link

fhavio commented Jul 23, 2018

Hi there, i was trying to understand how they save the information for monuments or swamp_huts now in 1.13 (since there are no temple.dat created), so i was working around with NBT and they saved the data in the chunk file.
My attemp was to duplicate a witch location but i'm kinda struggle to understand how they are organizing the chunk (see the image below)

image

The chunk numbers * 16 (chunk size) dont match with the coords.

Did anyone check this already? Also this is a single player world dont know if there are diferences for servers.

And i post here because seems like something usefull for overviewer/amidst

Edited: https://www.mojang.com/2012/02/new-minecraft-map-format-anvil/

@Javascript221

This comment has been minimized.

Copy link

Javascript221 commented Jul 26, 2018

The comment left by bobfrankly links to someone else's project that seems to handle the decoding and parsing of the new chunks. It might be possible to leverage their work to make updating Overviewer easier.

I'm a developer but new to python and Minecraft mods but love this app and want to lend a hand (let me find my base after getting hopelessly lost). Problem is I have no idea where to start (though I at least got the build working). I'm going to give it a shot but if anyone else is looking at this please don't hesitate to let me know. I could use a senior dev on this.

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented Jul 27, 2018

@Javascript221

I could use a senior dev on this.

People who used to work on Overviewer still hang out on our IRC channel, they can probably answer specific questions about the code if you have them.

EDIT: due to @freenode's ongoing incompetence, I'd like to say that me and achin can also be reached through discord: https://discord.gg/Vx2aSx6

@monkeyworknet

This comment has been minimized.

Copy link

monkeyworknet commented Jul 30, 2018

@CounterPillow is support for 1.13 on the roadmap / in the plans (with the understanding it will take work) or is it unreasonable to expect it to be actively worked on by the core team?

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented Jul 30, 2018

we're not working on it

@memiux

This comment has been minimized.

Copy link

memiux commented Jul 30, 2018

It seems that there's still some hope with Mapcrafter, though I prefer the render quality of Overviewer.

@mide

This comment has been minimized.

Copy link

mide commented Aug 1, 2018

I've placed a bounty of $100 on this issue on Bountysource.

@NigelWhatling

This comment has been minimized.

Copy link

NigelWhatling commented Aug 1, 2018

I'll add to that bounty. Bountysource

@gmcnew

This comment has been minimized.

Copy link
Contributor

gmcnew commented Aug 2, 2018

One of the changes in 1.13 is new paths for a lot of different textures, so I'm going through those and fixing them. I haven't touched the chunk parsing code, but it's a start: https://github.com/gmcnew/Minecraft-Overviewer/tree/minecraft113

@tazorax

This comment has been minimized.

Copy link
Contributor

tazorax commented Aug 3, 2018

@gmcnew looks good. Perhaps externalize texture paths in files/class constants to be able to switch between <=1.12 and >=1.13 resources pack (detect some specific png file for each version, or use pack.mcmeta) and to not breaking older maps ?

@jspanos71

This comment has been minimized.

Copy link
Contributor

jspanos71 commented Aug 3, 2018

Having the paths externalized is probably a needed requirement with upgraded worlds that have not been 'optimized' which means there could be some chunks with old formats and some chunks with the new format.

@Cubixmeister

This comment has been minimized.

Copy link

Cubixmeister commented Aug 3, 2018

Just added 500$ to the bounty as Craftserve - biggest Polish MC hosting. We're using Overviewer for more than 5 years and it is truly a great piece of software 😃

@gmcnew

This comment has been minimized.

Copy link
Contributor

gmcnew commented Aug 3, 2018

@tazorax, @jspanos71: Good point! I have intentionally ignored backward compatibility for now. I'm focused on getting this working for a fresh 1.13 world, and I figure we can sort out backward compatibility after that point. =)

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Aug 4, 2018

I'm not certain, but I think the paths are going to be the same regardless of the chunk format. It's not like wood blocks in 1.12 format look different than those in 1.13, just the layout of the resource packs have changed.

We haven't, historically, supported anything other than the most recent resource pack layout or map format, simply because it is a mountain of work. In practice this works ok, since Overviewer rarely has new features beyond supporting new blocks. People can continue to use the old Overviewer versions with no problems.

@mide

This comment has been minimized.

Copy link

mide commented Aug 4, 2018

That's a fair point @agrif. We could add something to the README saying "For renders of worlds prior to 1.13, use release XYZ." (And I propose we release prior to merging these changes).

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Aug 4, 2018

In the past we have created a branch before merging major world format changes like this. A note in the README would great too!

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Aug 4, 2018

Just going to comment on this for all those subscribed to the thread. I don't have the spare cycles to work on this now, but I do have some familiarity with the problem and the code, so here's a rough outline of how I would support 1.13.

The C code is tied to the idea that every XYZ position in space has a block ID and a data value, and it uses these two values to look up what block sprite to use at that position. The texture generation code is similarly tied to this idea. Changing this code to use a block palette, or not to use block IDs at all, would be a massive undertaking. Instead, I would translate from the new block format back to the old ID-based format when the world is loaded, and keep the texture generation and C code roughly the same. As a bonus, this method would easily allow supporting partially-converted worlds.

I only worry that this would be a pain to maintain in the future (not that Overviewer isn't already a bit painful to maintain). Any new blocks added would need us to essentially make up a fake block ID to use. Not too bad, but it adds another layer of indirection to the texture code which is already extremely hard to understand.

The alternative would be a massive rewrite of the texture generation and C rendering code. That's not a bad idea, but as a fair warning, it would be a ton of work.

Anyway, these are just my ideas. As I mentioned I probably won't have a chance to do much more on this than talk about it. Still, I'd be happy to help anyone who wants to tackle the problem as best I can, either on this issue tracker or on IRC.

@JonathanBourassa

This comment has been minimized.

Copy link

JonathanBourassa commented Aug 5, 2018

Here I found this post about someone who share source code capable of reading the 1.13 chunk format:
https://www.minecraftforum.net/forums/minecraft-java-edition/recent-updates-and-snapshots/2894808-minecraft-1-13-new-chunk-format-fully-decoded-read

@gmcnew

This comment has been minimized.

Copy link
Contributor

gmcnew commented Aug 7, 2018

This is implemented! https://github.com/gmcnew/Minecraft-Overviewer/tree/minecraft113

I took @agrif's excellent advice: the new world format is translated to the old one (to minimize the amount of code that needs to change).

Caveats:

  1. Lots of rare block types are ignored and treated as air. I don't expect this to be very noticeable, but in any case, everyone is welcome to extend the blockmap (see overviewer_core/world.py) with the block types I missed.
  2. Stairs, ladders, vines, and other blocks that look different depending on their orientation may not appear correctly. Glass panes are rendered as glass, and all glass is rendered as white. Blue ice is rendered as packed ice.
  3. There are some strange lighting artifacts at the edge of the map. I don't know if this was a preexisting bug.
  4. I didn't change the way underwater blocks are rendered -- you will still only see the surface of the water.

I don't expect items 1 and 2 will be difficult to fix, I simply ran out of time. =)

I hope the community finds this valuable!

@gmcnew gmcnew referenced this issue Aug 7, 2018

Merged

Minecraft113 #1471

@philgrim2

This comment has been minimized.

Copy link

philgrim2 commented Aug 7, 2018

@gmcnew I just cloned and built from your repo.
It still tells me it only works with 1.12

**Edit: Disregard - didn't check out the branch before I built. My bad. Testing now.

@gmcnew

This comment has been minimized.

Copy link
Contributor

gmcnew commented Aug 7, 2018

@philgrim2, did you check out and build the minecraft113 branch? (The master branch is unchanged.) You should get an error if your world is older than the 1.13 format. See here: https://github.com/gmcnew/Minecraft-Overviewer/blob/minecraft113/overviewer_core/world.py#L126

Edit: No problem. =) Please let us know if you discover any issues I failed to point out above.

@philgrim2

This comment has been minimized.

Copy link

philgrim2 commented Aug 7, 2018

@gmcnew Yeah, sorry (see my edit - I figured it out,)
Now I'm getting the following error:
``
2018-08-07 00:02:31 E Could not find the textures while searching for 'assets/minecraft/textures/block/grass_block_top.png'. Try specifying the 'texturepath' option in your config file.
Set it to the path to a Minecraft Resource pack.
Alternately, install the Minecraft client (which includes textures)
Also see http://docs.overviewer.org/en/latest/running/#installing-the-textures
(Remember, this version of Overviewer requires a 1.12-compatible resource pack)
(Also note that I won't automatically use snapshots; you'll have to use the texturepath option to use a snapshot jar)

Note that I started with the same config that worked with the stock overviewer and 1.12.
I've tried changing the texturepath from pointing at 1.12.2.jar in the client installation to a directory containing the unpacked files - that didn't work either. And grass_block_top.png isn't there, by the way - grass_top.png is. So I tried soft-linking it to grass_top_block.png just to see - got the same error.

*More info -
That path doesn't have a directory called "block", it's called "blocks". So...linked that too, and then the error message changed to a different file name.

@gmcnew

This comment has been minimized.

Copy link
Contributor

gmcnew commented Aug 7, 2018

@philgrim2: What do you get when you run python overviewer.py --check-terrain? If that isn't illuminating, you might need to set texturepath in your config file to the path of a 1.13 client .jar.

@tswsl1989

This comment has been minimized.

Copy link
Contributor

tswsl1989 commented Sep 18, 2018

@twoprops If you compiled overviewer yourself, compile it again to make sure it's linking to the correct versions of libraries on your system.
If not, I'd suggest opening this in a new issue

@twoprops

This comment has been minimized.

Copy link

twoprops commented Sep 18, 2018

Thanks! Sorry to post in this thread; I had simply assumed it was related to the 1.13 changes. Turns out it's a known problem with Ubuntu 18:

#1455

Solution is to downgrade Pillow:

sudo pip install 'Pillow<4.3.0'

--Ron

@Capri205

This comment has been minimized.

Copy link

Capri205 commented Sep 21, 2018

That solution worked for installed overviewer... how do we build the 113 version to use that? Not quite sure what needs to be set (config or environment) which would get it to use the old pillow library and not the installed default python-pil (5.2.1) on ubuntu 18.04.

@shelterx

This comment has been minimized.

Copy link

shelterx commented Sep 21, 2018

I'm going to bump this question, is the new overviewer only slow for me? I have a fairly huge world, it was never a problem before but the processing now takes forever. The world is on an SSD, nothing really changed apart from 1.3.1 and possibly a slightly newer python2 version (Python 2.7.15).

EDIT:
Running with --verbose stops the output here:
world.py:290 2889 2018-09-21 18:45:58 DEBUG Done scanning regions
files.py:43 2889 2018-09-21 18:47:30 DEBUG Detected that chmods work in '/var/www/web-game/map/survivalday'
files.py:59 2889 2018-09-21 18:47:30 DEBUG Detected that overwriting renames work in '/var/www/web-game/map/survivalday'
files.py:43 2911 2018-09-21 18:47:33 DEBUG Detected that chmods work in '/var/www/web-game/map/survivalday'
files.py:59 2911 2018-09-21 18:47:33 DEBUG Detected that overwriting renames work in '/var/www/web-game/map/survivalday'

Here's where it should show where it is in the rendering process but it doesn't, it does however render images.
There's no info from "observer.pyc:"

@twoprops

This comment has been minimized.

Copy link

twoprops commented Sep 21, 2018

I'm not a Python guy, but I think the pip install 'Pillow<4.3.0' command just replaced the library with the old version. I didn't have to rebuild overviewer (I'm using the 113 fork). I'm not using Python for anything else on that box; I assume there are possible consequences if you're running other Python software that expects a newer version of Pillow.

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Sep 21, 2018

@shelterx yes, the new branch is very slow. There is some low hanging fruit we can optimize later, but right now we are only worried about getting it correct.

@shelterx

This comment has been minimized.

Copy link

shelterx commented Sep 28, 2018

No one is working on this project much? I have some people that wants an up to date map with all bells and whistles,

@monkeyworknet

This comment has been minimized.

Copy link

monkeyworknet commented Sep 28, 2018

@Capri205

This comment has been minimized.

Copy link

Capri205 commented Sep 28, 2018

Last update appears to be a while back. Are you using this for 1.13? the point of this thread is overviewer for 1.13, not 1.10.

Written by m0r13 on Aug. 29, 2016 .
A while ago I already released Mapcrafter 2.3 with support for Minecraft 1.10.

https://mapcrafter.org/index equal quality - very similiar look - and runs much faster.

From the authors github:

Due to my university studies and other projects, unfortunately I'm not able to develop new features for Mapcrafter anymore.

@monkeyworknet

This comment has been minimized.

Copy link

monkeyworknet commented Sep 28, 2018

@Charlweed

This comment has been minimized.

Copy link

Charlweed commented Sep 30, 2018

Does anyone know if this mc1.13 branch cannot process POI's?
My regular render of a 1.13 world worked fine with this branch, but when i tried --genPOI:

2018-09-29 16:47:25 E An error has occurred. This may be a bug. Please let us know!
See http://docs.overviewer.org/en/latest/index.html#help

This is the error that occurred:
Traceback (most recent call last):
  File "/home/PANDIMONIUM/hseldon/.minecraft/bin/Overviewer/overviewer.py", line 628, in <module>
    ret = main()
  File "/home/PANDIMONIUM/hseldon/.minecraft/bin/Overviewer/overviewer.py", line 131, in main
    g.genPOI.main()
  File "/home/PANDIMONIUM/hseldon/.minecraft/bin/Overviewer/overviewer_core/aux_files/genPOI.py", line 524, in main
    handleEntities(rset, config, options.config, list(rset_filters), markers)
  File "/home/PANDIMONIUM/hseldon/.minecraft/bin/Overviewer/overviewer_core/aux_files/genPOI.py", line 200, in handleEntities
    results = pool.map(parseBucketChunks, ((buck, rset, filters) for buck in buckets))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
ChunkDoesntExist: Chunk 0,-15 doesn't exist
@Capri205

This comment has been minimized.

Copy link

Capri205 commented Oct 8, 2018

still can't get the 113 version to work. Does anyone have instructions / step-by-step for a build of this for 1.13? I get the usual core dump running it.

I've uninstalled the packaged version for 1.12. I've re-applied the pillow downgrade which was required for the packaged overviewer for it to work on 1.12 and Ubuntu 18.04:

root:/overviewer/build# pip install 'Pillow<4.3.0'
Requirement already satisfied: Pillow<4.3.0 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: olefile in /usr/lib/python2.7/dist-packages (from Pillow<4.3.0)

Python is ver 2.7 with all the libraries installed:
madmin:/overviewer/build$ python --version
Python 2.7.15rc1

The build goes through fine, no errors. I have the textures all linked into the build folder also.

madmin:/overviewer/build$ python ./setup.py build
...
Build Complete

madmin:/overviewer/build$ python ./overviewer.py --config=sample_config.py
2018-10-08 13:56:36 Welcome to Minecraft Overviewer!
2018-10-08 13:56:36 Generating textures...
Segmentation fault (core dumped)

Oct 8 13:56:36 kernel: [242871.626662] python[5245]: segfault at 5651099a2e50 ip 00007f6dbc83ee7c sp 00007ffe0b85c720 error 4 in c_overviewer.so[7f6dbc839000+1e000]

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented Oct 9, 2018

@Capri205 try a ./setup.py clean and then a ./setup.py build again, it could be that during your troubleshooting you built it against a different pillow version before you downgraded it or something.

@Capri205

This comment has been minimized.

Copy link

Capri205 commented Oct 10, 2018

yes, thanks. Done that many times. I will try a re-install of python.

@saltbreez

This comment has been minimized.

Copy link

saltbreez commented Oct 17, 2018

I have the same issue Capri205 has

Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

@Fusl

This comment has been minimized.

Copy link

Fusl commented Oct 21, 2018

@gmcnew Using your branch works pretty well here. However, I get lots of Found $n blocks of unknown type $type {$data} debug messages. I'm not sure if that's because of 1.13 or if this is a common debug message of Overviewer:

  14 minecraft:sign {u'waterlogged': u'false', u'rotation': u'5'}
  17 minecraft:chipped_anvil {u'facing': u'south'}
  18 minecraft:chipped_anvil {u'facing': u'west'}
  19 minecraft:sign {u'waterlogged': u'false', u'rotation': u'7'}
  20 minecraft:sign {u'waterlogged': u'false', u'rotation': u'6'}
  34 minecraft:sign {u'waterlogged': u'false', u'rotation': u'1'}
  34 minecraft:sign {u'waterlogged': u'false', u'rotation': u'12'}
  34 minecraft:sign {u'waterlogged': u'false', u'rotation': u'3'}
  60 minecraft:sign {u'waterlogged': u'false', u'rotation': u'8'}
  97 minecraft:sign {u'waterlogged': u'false', u'rotation': u'4'}
 125 minecraft:sign {u'waterlogged': u'false', u'rotation': u'0'}
1640 minecraft:wall_torch {u'facing': u'west'}
1853 minecraft:wall_torch {u'facing': u'south'}
1993 minecraft:wall_torch {u'facing': u'north'}
2095 minecraft:wall_torch {u'facing': u'east'}
@damn-simple-scripts

This comment has been minimized.

Copy link

damn-simple-scripts commented Nov 12, 2018

Hi,
I experienced problems with segmentation faults, I was using the downgraded version of Pillow.

This time upgrading fixed it:

pip install Pillow --upgrade
./setup.py clean
./setup.py build
@stwalkerster

This comment has been minimized.

Copy link

stwalkerster commented Nov 28, 2018

I'm also seeing the same error generating POIs that @Charlweed is seeing.

If I comment out all of my markers, then genPOI runs without error. If not, I get a message saying a specific chunk does not exist, and when I visit that chunk to force it into existence, the error message just points to a different chunk.

Markers and POIs are one of the most used features of the maps I generate, and it's rather annoying that they cannot be used with our 1.13 map.

Should I split this out into a different issue?

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented Nov 28, 2018

@stwalkerster this was already fixed, see 38bc400

You're probably using a build from before that fix.

@stwalkerster

This comment has been minimized.

Copy link

stwalkerster commented Nov 28, 2018

Urgh, yes, probably. I'll give that a go after my current set of renders are complete. Apologies that I didn't check I had the latest code before commenting here.

@CounterPillow

This comment has been minimized.

Copy link
Member

CounterPillow commented Nov 28, 2018

Here's some new Windows builds of the minecraft113 branch containing these fixes:

@Phreag

This comment has been minimized.

Copy link

Phreag commented Feb 20, 2019

@CounterPillow could you upload fresh build Win32 and Wind64 binaries?
Weould be really nice :)

@agrif agrif closed this in #1471 Feb 20, 2019

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Feb 21, 2019

@Phreag new 1.13 builds are now (and in the future) available on our downloads page. It might take a while for the cache to clear, so in the meantime:

@corentinbettiol

This comment has been minimized.

Copy link

corentinbettiol commented Feb 21, 2019

I did a quick search (using profiling) to understand what makes it so slow. Here's the answer. 97.1% of the time only to repeat this function on each chunk it has to parse :/

So the function is "_packed_longarray_to_shorts", added in this commit, but I don't understand the project or python enough to know what it does.

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Feb 21, 2019

Yes, that lines up with what I thought -- but thank you so much for actually profiling it and getting hard data.

The new chunk format decided to pack block ids together in a way that is very difficult for interpreted languages to read quickly, to save a tiny bit of space. Difficult, but not impossible. I have some ideas on how to make that function faster, and will implement them as time permits.

@agrif

This comment has been minimized.

Copy link
Member

agrif commented Feb 22, 2019

@corentinbettiol I rewrote that function to be quite a bit faster. Mind giving it a whirl?

#1519

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.