Wrong palettes+tiles in metatiles when trying to reuse general primary tiles in a custom secondary tileset

[SPMoo66]

After decompiling the general primary tileset for use with a custom secondary tileset it seems that any generated metatiles that should be reusing existing general tiles/palettes may be incorrect. All metatiles using custom palettes/tiles seem to be created correctly.
I am using dual-layer tiles, if that helps at all.

• For the overlapping snow/grass metatiles the snow/grass tiles are correct, but any sections that should be reusing palette 2 (or the custom palette generated for the snow/grass tile) and tiles 0x002/0x003 are wrong. Palette 3 is used and tile 0x002 is always flipped horizontally and vertically while tile 0x003 is always flipped vertically.

• The vanilla tree+snow/grass overlapping metatiles are rather strange. For the top edge of the vanilla trees, tile 0x0CD should be used with palette 2, but porytiles is generating a tile in the secondary tileset and uses palette 3, while no appropriate palette is generated for that tile.

• The inside tiles of the vanilla trees in the secondary tileset all seem to be set to vanilla tiles, but the wrong tiles and wrong palettes, almost like it recognized vanilla tiles but chose wrong. Tiles 0x009, 0x019, and 0x029 with palette 2 all should have worked there.

Command used to decompile general primary tileset:
porytiles decompile-primary -o $HOME/general $HOME/pokeemerald-expansion/data/tilesets/primary/general $HOME/pokeemerald-expansion/include/constants/metatile_behaviors.h

Command being used to compile the secondary tileset:
porytiles compile-secondary -dual-layer -Wall -o $HOME/pokeemerald-expansion/data/tilesets/secondary/snow $HOME/porytiles-snow $HOME/general $HOME/pokeemerald-expansion/include/constants/metatile_behaviors.h

Here are the bottom/middle/top images used in compilation

Environment (please complete the following information):

• amd64
• Porytiles release version: nightly-623dc85a9cfdb1c493e92cd298a9bfb2dabc7cf6

[grunt-lucas]

This is perfect, thanks. Should be everything I need to attempt a reproduction. I am pretty busy for the next week or so, but after that I will definitely make time to look in to this. I want to do a 1.0.0 release this year (it's been in beta forever), and fixing bugs is my top priority for that release.

Just looking at what you are doing, I have a suspicion I know what the problem is. It may not be a bug per se, but it's definitely a really nasty "gotcha" that users should not have to deal with. So either way, there's potential for improvement here.

[grunt-lucas]

@SPMoo66 Ok looking more carefully at your post. When you decompile the primary general tileset to make changes, you need to recompile it, and import that recompiled primary general as a new tileset in Porymap. So you'll effectively have two primary general sets. The vanilla one, and a recompiled-with-porytiles one. Then you have to make sure that any map in Porymap that is compiled via Porytiles, and which uses one of your custom secondary sets (like the snow one), are using the recompiled-with-porytiles general set, not the original vanilla one. Otherwise you're going to get in to all kinds of trouble.

Porytiles completely reorders tiles.png and the .pal files. So when you compile a custom secondary set, it will be "linked" to the recompiled-with-porytiles primary set, not the vanilla one. If you try to use the vanilla one in Porymap, the tile and palette references won't line up and that's what causes the artifacting.

I know this isn't very well explained at the moment, sorry for the confusion. I am still trying to fill out more of the wiki pages, and eventually have some kind of video tutorial series.

Please go ahead and give the above steps a try. If that fixes your problem, I'll close this issue. But I've opened a new feature issue to track some possible improvements, since I don't think what Porytiles is doing here is a good user experience.

[SPMoo66]

Recompiling the tileset as you recommend fixed the artifacting, but I did have an issue with dual vs triple layer and had to manually edit all the decompiled images.
It appears that the default decompile command is for triple-layer tilesets and it doesn't seem like the -dual-layer argument works for decompile as it does for compiling. If you just take the decompiled tileset and recompile as dual-layer you get a tileset with a bunch of the transparency color covering up the tiles. I was able to fix this temporarily by blowing away all the transparency colors on the middle and top images, but I imagine this wouldn't be what you want users to be doing.
If I did just miss something simple in recompiling the tileset, please let me know.

I'm assuming that recompiling a decompiled tileset works well for triple-layer tilesets, but I didn't test it myself.

[grunt-lucas]

@SPMoo66 The decompile command automatically detects the compiled layer type and adjusts accordingly. When it decompiles a dual-layer set, in places where there was no tile information at all (e.g. the middle tile of a SPLIT metatile), it generates a tile with all pixels set to alpha=0.

This problem you described, was it when you decompiled the vanilla primary set? And recompiled it? The reason you probably had this issue is because, annoyingly, different palettes in the original vanilla primary general set use different transparency colors. The Porytiles decompiler is technically OK with this, it assumes you know what you're doing when you go to recompile things. But Porytiles compilation only supports a single transparency color, which means you either have to manually fix up the tileset (like you did), or supply the decompiler with -normalize-transparency, which will change all transparency in the original tileset to the 255 0 255 magenta color.

Go ahead and try decompiling again, this time with -normalize-transparency. See if it fixes everything, and then we can finally mark this closed :)

[grunt-lucas]

Oh also, the compiler allows you to adjust which color is treated as transparency via the -transparency-color option. This works fine for some vanilla sets, which sometimes use 0,0,0 or 24,41,82 for transparency. However, many tend to mix and match transparency colors across the palettes. So it's usually better to just decompile with -normalize-transparency[=<RGB>] to change all transparency to a color of your choice

[SPMoo66]

Adding -normalize-transparency worked perfectly. Thank you so much for all of the help!

[grunt-lucas]

Awesome! I'll close this issue. But I opened #47 to track improvements to some of the rough edges you discovered. Thanks!