Adding PhysFS Library to support playing episodes from zip/7z/iso

[0lhi]

A feature that @Wohlstand and I have spoken about in private is to give TheXTech the option to play episodes from archives and image files, which can be done through the PhysFS Library. This would create the following benefits:

• Reduce disc space usage.
• Encourage users to keep unaltered copies of episodes on their harddrives, making preservation easier.
• Potentially, in combination with some modified conversion kit code, allow to play Custom Set Episodes without any prior preparations.
• Allow TheXTech to get the MD5-Checksum of a whole Episode, which is important for the Netplay feature that is currently in the talks (gonna open a GitHub Discussion on this later today).

For the sake of keeping the Archives unchanged, I suggest that TheXTech should ignore any .sav files that may be stored inside of them.

[ChristianSilvermoon]

Hmm, archive formats can also de-dedupe data afaik, which would be great for episodes that repeatedly use the same assets across several, but not all levels.

[Wohlstand]

Hedgewars does use PhysFS a lot, and all custom add-ons were shipped as .hwp files (basically a renamed ZIP archive) which contain some resources that the game can see and re-use

[ChristianSilvermoon]

Hedgewars does use PhysFS a lot, and all custom add-ons were shipped as .hwp files (basically a renamed ZIP archive) which contain some resources that the game can see and re-use

Could we use .smbxe for Episodes? :P

[Wohlstand]

Could we use .smbxe for Episodes? :P

Technically yes, but maybe something also would appear in a head, so, the time will solve this once we'll start this task

[LoveBodhi]

This reminds me "MarioPacker" in SMBX-38A, but they're cracked.

[Wohlstand]

Anyway, main goal to just simplify the distribution and make episodes and packages being more portable and the game being more compact. Absolutely no purpose to craft any sort of DRM for episodes which limits the freedom a lot and annoys. However, if episode maker really wants the protection, then the sort of obfuscation would be made that will confuse all guts of episode data making it hard to maintain by human. Manly it's a noob proof protection, hackers will develop the way to deobfucste that easily.

[MECHDRAGON777]

While I do not currently use TheXTech, I am in support of this as not only does it start on the possibility to play co-op, but also lowers the file size used by episodes and custom levels.

[0lhi]

@ChristianSilvermoon Could we use .smbxe for Episodes? :P

@Wohlstand Technically yes, but maybe something also would appear in a head, so, the time will solve this once we'll start this task

Instead of anything that directly references Nintendo, I'd rather suggest:

.xte (X-Tech Episode(s))

.xtl (X-Tech Level(s))

.xta (X-Tech Asset Pack)

These formats should be used for stuff that is exclusive for TheXTech, content that works on other branches like Vanilla or X2 should stick with 7z or zip to avoid an artificial access barrier. Maybe we could encourage/enforce a naming convention like:

.wld.7z / .wld.zip (Multi-Branch Episode(s))

.lvl.7z / .lvl.zip (Multi-Branch Level(s))

.pck.7z / .pck.zip (Packed game like Talking Time/Yoshi's Archipelago).

[ChristianSilvermoon]

@ChristianSilvermoon Could we use .smbxe for Episodes? :P

@Wohlstand Technically yes, but maybe something also would appear in a head, so, the time will solve this once we'll start this task

Instead of anything that directly references Nintendo, I'd rather suggest:

.xte (X-Tech Episode(s))

.xtl (X-Tech Level(s))

.xta (X-Tech Asset Pack)

These formats should be used for stuff that is exclusive for TheXTech, content that works on other branches like Vanilla or X2 should stick with 7z or zip to avoid an artificial access barrier. Maybe we could encourage/enforce a naming convention like:

.wld.7z / .wld.zip (Multi-Branch Episode(s))

.lvl.7z / .lvl.zip (Multi-Branch Level(s))

.pck.7z / .pck.zip (Packed game like Talking Time/Yoshi's Archipelago).

Yeahhh, now that I think about it you're right using .smbx* is not ideal, and your suggested alternatives feel great ^^

I can imagine:

worlds/
├── other-episode.zip
├── princess-cliche.xte
├── super-talking-time-bros-series.xte
├── the-invsasion-2.xte
└── wip-episode

[LoveBodhi]

This make me think ".love" format (LÖVE2D Program)!!

[ChristianSilvermoon]

Would this also support .tar and .tar.gz files?

[Wohlstand]

Would this also support .tar and .tar.gz files?

Would (just need to check out the specification), but tar.gz (not tar) is less effective for random access as this is a seamless compression that requires to decompress of the whole file until it can be used

[ChristianSilvermoon]

Would this also support .tar and .tar.gz files?

Would (just need to check out the specification), but tar.gz (not tar) is less effective for random access as this is a seamless compression that requires to decompress of the whole file until it can be used

I guess I should've thought of that but I didn't x3
I guess .tar.gz would be rather inefficient for this purpose :P

[LoveBodhi]

Does work if including password for the archive??

[Wohlstand]

Does work if including password for the archive??

That will also mean, password should be provided in certain config file or typed manually on load

[ChristianSilvermoon]

Does work if including password for the archive??

That will also mean, password should be provided in certain config file or typed manually on load

We do have an onscreen keyboard in game now :P

[0lhi]

Uhh, does PhysFS support that? It would surprise me a lot.

If it does, I guess, we could make an MD5 based database for Episodes that are encrypted for some reason, so that TheXTech will know the password by default. But, idk how many episodes are affected by that. It seems kinda pointless.

[ChristianSilvermoon]

Uhh, does PhysFS support that? It would surprise me a lot.

If it does, I guess, we could make an MD5 based database for Episodes that are encrypted for some reason, so that TheXTech will know the password by default. But, idk how many episodes are affected by that. It seems kinda pointless.

ZIP files do support password protection. IDK if PhysFS supports it.

It effects no episodes currently because no episode is played from a password protected ZIP file.

I don't see much utility in password protecting episodes tho, unless we want to keep other people on the same PC from snooping (why would we need to with episodes?) or in case you wanted to keep the episode for certain people only, IE having a previous episode give you the password for its sequel on completion or something.

[poudink]

I would be very happy to have any sort of archive support, regardless of compression. SMBX episodes being packaged as folders with thousands of files makes them very cumbersome. Transferring them between different drives and devices takes a very long time for one, but really the biggest problem is that with some less efficient file systems like exfat and fat32, episodes end up using a lot more space than they should. This mostly affects platforms that don't support better file systems, like Android, the 3DS or the Switch. As far as I can tell, this is because of cluster size shenanigans, like files only being allowed to be a multiple of 100kb large. When an episode consists of thousands of very tiny files, you can see where the problem lies. An episode that only takes 200MB on my laptop (ext4) can easily take up 700MB on my Switch (fat32) just because of that. Being able to simply load episodes from a tar or an uncompressed zip file would completely fix the problem.