[Feature Request]: Texture Replacements: Add option for wildcard characters

[Kupo91]

Description

Apologies if this feature (request) already exists, I googled but couldn't find anything about that. So if texture dumping is enabled pcsx2 tends to fill empty space on textures with junk. And due to this we can't replace these textures because the filenames are always different. But not the whole name! Usually it's only the first hash which is different. Example:

36fcd419707a35cf-3b0f5ac99a2574db-00006653.png

6884d36bd5704ba9-3b0f5ac99a2574db-00006653.png

619389eab4a92ed9-3b0f5ac99a2574db-00006653.png

In Dolphin they allow using wildcard characters for texture replacements. The hash which is different will be replaced by a character like "$" so the emulator knows the texture replacement file $-3b0f5ac99a2574db-00006653.png counts for all textures which end with 3b0f5ac99a2574db-00006653.png, no matter what the first hash looks like.

Reason

Wildcard characters can save tons of MBs in texture packs. Often it's also the only way to replace certain textures.

Examples

for reference: https://bugs.dolphin-emu.org/issues/12505

[MrCK1]

The contents of the texture have little to do with the hash, if there is unique hash then it is seen as a unique texture regardless. It sounds like you're just going to have to find the correct texture to replace accordingly.

Nothing do be done about the "junk" inside the texture either, it is the nature of the console. If it concerns you any more, further debugging in PCSX2 can tell you what area of the texture is being utilized.

[F0bes]

If you are attempting to replace an indexed texture, there is a chance that an unused portion of the CLUT will be modified. In this case it may be useful to ignore the CLUT hash.

[Kupo91]

The contents of the texture have little to do with the hash, if there is unique hash then it is seen as a unique texture regardless. It sounds like you're just going to have to find the correct texture to replace accordingly.

Nothing do be done about the "junk" inside the texture either, it is the nature of the console. If it concerns you any more, further debugging in PCSX2 can tell you what area of the texture is being utilized.

This behaviour might be the nature of the console, I don't say anything against that. But that doesn't change the fact that if there are let's say 100 textures which all show the same on your screen and are only different in regions which won't be used for the replacement anyway, it would be useful to load just one texture for all cases to save space. And that's where the wildcard character comes in. It's only for texture loading not for texture dumping.
In the worst case it "only" saves a lot of space. In the best case it allows loading textures which could not be loaded before, because they change everytime. In some cases I have thousands of textures which show the same and are just slightly different in regions, which won't be used for texture loading. In this case it's not possible to replace them without a wildcard character, you would end up with 100 gb texture packs if you try to replace all of them and probably reach nothing

[Kupo91]

If you are attempting to replace an indexed texture, there is a chance that an unused portion of the CLUT will be modified. In this case it may be useful to ignore the CLUT hash.

What exactly do you mean by "ignore"? It sounds a bit like that I can just keep this hash away in my replacement file? Or did you agree, that a wildcard character which has this job might be useful? Sorry for being slow on the uptake

[F0bes]

I was agreeing with the wildcard character use.

[stenzek]

The hash which is different will be replaced by a character like "$" so the emulator knows the texture replacement file $-3b0f5ac99a2574db-00006653.png counts for all textures which end with 3b0f5ac99a2574db-00006653.png, no matter what the first hash looks like.

This won't work. The first part of the hash is the texture data/indices, the second is the CLUT/palette. So, you would be replacing all textures which use one palette. Those textures could be different sizes, have completely different appearances, etc.

Wildcards with hashes alone makes no sense, because there's no logical relation between the characters to the actual data.

If you are attempting to replace an indexed texture, there is a chance that an unused portion of the CLUT will be modified. In this case it may be useful to ignore the CLUT hash.

What fobes said here is doable (by letting it use a replacement without CLUT as a CLUT texture). But that's all. Although I don't really like the idea, because again, false positives.

[Kupo91]

I can just tell about my experience. I just dumped 33k textures in a game and when I tried to sort them, searching for the CLUT hash helped me a lot to find "duplicates" and group them. And yes, I noticed that there might be false positives like those avatars:

1a53c420e6579b21-1fc1ff16c2607850-00005ddb.png

1a5665bdde2600a2-1fc1ff16c2607850-00005ddb.png

1e3732186321da95-1fc1ff16c2607850-00005ddb.png

But as long as the wildcard character is just optional it doesn't hurt anybody, does it? The texture pack creator has to take care where he uses it. I just see many textures atm which can't be replaced and that's sad so it might be worth a try?

[stenzek]

But as long as the wildcard character is just optional it doesn't hurt anybody, does it?

It does. Instead of an O(1) lookup potentially thousands of times per frame, now you're looking at O(N), where N is the number of replacement textures. And wildcard matching is more expensive than a string comparison.

[Kupo91]

Another option would be to introduce a texture.ini as ppsspp does.
It allows to assign multiple hashes to a certain texture file.

https://github.com/hrydgard/ppsspp/wiki/Texture-replacement-ini-syntax

It doesn't use wildcards, the hashes must be named completely.
That doesn't solve the issue that certain textures might not be replaced due to constantly changing file names (I played and dumped textures of a game for two weeks, still it dumped a new texture for the title screen this morning), but atleast it can increase the chance that a texture will be replaced while the texture pack size stays moderate.

In my case it would save 700 textures alone for a world map, monster avatars and some other UI. If I had to replace all of these duplicates I need 2 gb extra space. For 4 textures.
I hope we will find a solution for that, because nobody wants to spend 2 gb for duplicates. And in other games it won't be much different.

[stenzek]

inis are a terrible idea imo, it just adds a ton of work to the whole process.

If you're getting many variants of the texture data itself (the first part of the hash), then there's a good chance it's just whatever junk happened to be in local memory before, and there's a huge number of possibilities it could end up with. Not all games will work well with texture replacement, as like other enhancements, it's a giant hack.

[jedinjapan]

To build on the ini conversation, what if PCSX2 just scanned for the ini file first, and defaulted to matching filenames if none was found? As an option, it would keep it simple for making a basic pack, but allow HD Texture replacements to massively decrease their memory footprint and download size, while also being much more logically organized as a project into folders.

It really can make some games much more viable for texture replacement versus file matching, and lower the overall memory cost on weaker systems. Racing games or sports titles will often only need 1 texture for items like a billboard or a racer, but the system memory will produce 6-10 unique hashes per item, resulting in an incredible amount of file duplication bloating any attempts at a texture replacement pack.

I definitely don't want to be looking a gift horse in the mouth here, as it is incredible we have this implemented at all. I just want to make a solid use case for inis as an option, if only for making packs more logically organized during the production phase.

Edit: An additional thought: this would allow for one texture pack to be made semi-easily compatible with other regions via appending the additional hashes to the ini without producing an maintaining a separate pack.

[stenzek]

For gosh sake, no, we are not having "inis" for texture packs. It's way too much extra work for basically no gain.

Making wildcards match is on my todo list, but it's already a mountain long and there's more important things to do. If anyone wants to attempt it themselves, I'm happy to provide guidance/code review.

while also being much more logically organized as a project into folders

Pretty sure it searches folders already. This is not contingent on "ini"s.

lower the overall memory cost on weaker systems

No, it'll make no difference. Only the textures which are needed at any given point are loaded, and they're done so asynchronously.

but the system memory will produce 6-10 unique hashes per item

If it's duplicates with the same base hash but different CLUT hash, wildcards will solve this.

Edit: if you really want inis, nothing's stopping you from throwing together some scripts to generate a new "set" of replacements based on your "ini". Doesn't require emu support.

[jedinjapan]

Pretty sure it searches folders already. This is not contingent on "ini"s.

Was not aware the feature searched recursively. Apologies.

If it's duplicates with the same base hash but different CLUT hash, wildcards will solve this.

Understood. As you say, if wildcards are implemented then between that and the folder scanning it covers the majority of those use cases.

[Kupo91]

If it's duplicates with the same base hash but different CLUT hash, wildcards will solve this.

Understood. As you say, if wildcards are implemented then between that and the folder scanning it covers the majority of those use cases.

I can't speak for other games, but in Dragon Quest VIII it's always the base hash which is different, while the CLUT hash is the same.
Here are just some examples of duplicates pcsx2 is generating in DQ8:

Wildcards only for base hashes would bring no benefit.

[stenzek]

While it would be possible to have wildcards for CLUTs, it has a massive risk of false positives/collisions, and I think it's a terrible idea. If two textures share the same 16-colour/256-colour palette, they'll get replaced the same.

[Kupo91]

I didn't test other games, but in Dragon Quest VIII 95% of all textures which share the same CLUT hashes are just duplicates. As workaround I made a script to create symlinks for those, so I don't bloat the texture pack too much and don't have to replace all duplicates, if I edit one of them. The script is more or less also a list of duplicates. They are mainly UI textures:

https://pastebin.com/VyQBp8w0

The script feels like a drop in the ocean though. After 150 hours of playtime it contains like 200 duplicates of the worldmap and still the original worldmap pops up in the game here and there. It's a never ending story and a bit depressing.

Face textures are the only false positives that I encountered. I don't know what it looks like in other games. Maybe others can share their experience too.
But if someone makes a texture pack and uses false positives because he doesn't check the textures which he uses wildcards for, it's his own fault anyway. I don't see why others shouldn't get the chance to fix this issue because of that.

[gabriele2000]

Sorry for the off-topic but: what are the supported formats for the texture replacements?
PNG is too heavy for my taste.
WebP of AVIF support would be nice since they support transparency (I guess)

[RedDevilus]

#5547

DDS and PNG.

[gabriele2000]

#5547

DDS and PNG.

Oh, thanks!

[Pandavenom]

Wildcards really could help us to complete some texture pack. Many games, ex. Shadow Hearts series, Wild Arms series, Tales of Destiny/Rebirth, etc etc etc. just dumps the same textures over and over endlessly. And the hach code is the same just first or second part is different. If we can rename one of those texture with wildcards like this 0000000000000000-4af041be9842aa32-00005a13.png or cc69247b428a8c6b-00000000000000000-00005a13.png (manualy ofc), it would solve most of the problem.
(And no, we don't need ini :) )

[stenzek]

If we can rename one of those texture with wildcards like this 0000000000000000-4af041be9842aa32-00005a13.png

This won't work. The second part of the filename is the CLUT, replacing all textures that share a CLUT will have tons of conflicts.

Some games, particularly those which use large texture atlases with many different CLUTs within, just are not friendly to replacement.

And then you have the games that only partially overwrite the bounds of the texture, leaving whatever junk was in local memory previously around. For example, Snowblind games (Baldur's Gate etc).

[Pandavenom]

So there is nothing we can do with those games? :(

[Mrlinkwii]

@Ammacca we dont support individual texture replacement projects please ask the replacement maker

[Ammacca]

@Ammacca we dont support individual texture replacement projects please ask the replacement maker

sorry, my mistake

[superostrich]

Wildcards really could help us to complete some texture pack. Many games, ex. Shadow Hearts series, Wild Arms series, Tales of Destiny/Rebirth, etc etc etc. just dumps the same textures over and over endlessly. And the hach code is the same just first or second part is different. If we can rename one of those texture with wildcards like this 0000000000000000-4af041be9842aa32-00005a13.png or cc69247b428a8c6b-00000000000000000-00005a13.png (manualy ofc), it would solve most of the problem. (And no, we don't need ini :) )

Ace Combat Zero does the same thing, it'll write out the same identical textures over and over until you run out of storage.

(And no, we don't need ini :) )

I wouldn't say an INI is a need, but as someone who has published a couple packs for PPSSPP it sure is nice to have!

[stenzek]

I explained why this is a bad idea above. If someone wants to implement this and PR it, they're free to do so, but bumping the issue accomplishes nothing.

[F0bes]

The thought of turning texture lookups from O(1) to O(n) is frightening. It would be nice to have but for technical reasons outlined above this is below our low priority threshold.