Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-add 'pack mode' to 3.0 export templates #14471

Closed
insomniacUNDERSCORElemon opened this issue Dec 9, 2017 · 9 comments
Closed

Re-add 'pack mode' to 3.0 export templates #14471

insomniacUNDERSCORElemon opened this issue Dec 9, 2017 · 9 comments

Comments

@insomniacUNDERSCORElemon
Copy link

@insomniacUNDERSCORElemon insomniacUNDERSCORElemon commented Dec 9, 2017

I think that this was important to allow executable-packed-with-assets exports, especially useful for allowing small games (like game jam games) to be downloadable directly by users rather than requiring unzipping a file (zipping being necessitated because of an executable with .pck file and 1 file per download).

The only reasoning I can think of for this being removed is that someone thought that exporting inside of a zip was a useless feature (because let's face it, it is) because of dedicated archive managers, and therefore pack mode was removed altogether.

It'd be nice to see some actual compression options, such as per-world/per-chapter groups (for a large amount of content that is spaced out), per-group (compression levels based on size and need), on-startup/as needed/self-extraction options etc.

I also think that .pck files aren't the best option for restricting file access OR allowing players to access/edit files. I think it'd make sense for language files to be exposed by default (for language visibility+installation), but every game will have different needs beyond that. When it comes to everything being accessible to create asset packs (maybe even mods) for a game, enabling self-extraction through the game options or even a separate game download would be the best thing (this overlaps with faster loading speeds if sufficient drive space is available).

@reduz

This comment has been minimized.

Copy link
Member

@reduz reduz commented Dec 9, 2017

There were many bugs with this, executables not being recognized, etc. It was removed because it was too much of a pain

@insomniacUNDERSCORElemon

This comment has been minimized.

Copy link
Author

@insomniacUNDERSCORElemon insomniacUNDERSCORElemon commented Dec 9, 2017

Was this an OS version/distro-specific, setup specific (libraries), or 3.0 issue? I've never had an issue with it on 64-bit Arch Linux and dozens of 2.X jam games. And it works for me if I export something with 2.X.

Unless the issue occurs when you have a larger amount of data...

@Zylann

This comment has been minimized.

Copy link
Contributor

@Zylann Zylann commented Dec 10, 2017

It is an issue when you have players' antiviruses flagging downloaded games as virus just because their executable is not known to them (but I never had issues with executables themselves).

@Calinou

This comment has been minimized.

Copy link
Member

@Calinou Calinou commented Dec 10, 2017

You should not be distributing bare program executables – this is valid for any program, not just games. Instead, pack the game files into a ZIP archive (which provides compression for all game files, not just the game data, and makes loading faster as the data sits uncompressed on the user's disk once the archive has been extracted), or create a Windows installer using something like Inno Setup.

Another upside of PCK files is that they can be used with several export template binaries (for different platforms and architectures), which also allows the user to replace the engine binary, even if they do not have the project sources (provided no in-house C++ modules are required).

@insomniacUNDERSCORElemon

This comment has been minimized.

Copy link
Author

@insomniacUNDERSCORElemon insomniacUNDERSCORElemon commented Dec 11, 2017

@Zylann @Calinou If the only issue here is anti-virus, could this option be re-added at least for Linux? Most people on Linux don't run antivirus, and with the performance impacts even if they do they likely only do manual scans with it. And something like ClamAV might not put false positives just because something is 'unknown'. Also it could be non-default and explained in the docs that users might run into antivirus putting it into quarantine.

I still also think compression may be desired on an installed game that takes up a large chunk of data uncompressed. With an SSD (where data is loaded faster but price limits storage available) and highly-parallelized decompression, loading times might not be an issue. Especially if data is properly split into different archives where only what is needed at a specific time is decompressed. So really it depends on each game and user.


If it would be possible to choose to use something other than a .pck file... I'd rather use a .zip, .7z, or even just an exposed data folder (if I wanted a user-editable game--as I describe in the opening post--at least) if since I'd be zipping the export anyways. Self-extraction from the .pck would work in this situation as well, though.

I'm not sure if the current .pck setups would allow it, but being able to download assets (game updates like how Minecraft does it, also for allowing multiple playable versions so users don't have to cautiously update) basically turning the executable into a launcher (even though it would still run the game) would also be useful. That way users wouldn't have to manually re-download the game for each update.

There probably isn't a clean way to use a different executable (without waiting for it to be recognized by false-positive AV software, at least) but hopefully options get sorted out down the line.

@Calinou

This comment has been minimized.

Copy link
Member

@Calinou Calinou commented Dec 11, 2017

If the only issue here is anti-virus, could this option be re-added at least for Linux?

I would rather not see this re-added for any platform, because of the second point I have mentioned (separation of concerns). Besides, features shouldn't be needlessly tied to specific platforms.

I still also think compression may be desired on an installed game that takes up a large chunk of data uncompressed.

You can use GPU-compressed/PNG/WebP textures and Ogg Vorbis sounds in 3.0; these will sit as-is in your PCK file, so you will still benefit from compression, even after installing the game.

If it would be possible to choose to use something other than a .pck file... I'd rather use a .zip, .7z, or even just an exposed data folder (if I wanted a user-editable game--as I describe in the opening post--at least) if since I'd be zipping the export anyways.

Godot 2.1 supported ZIP for data files, but this isn't supported in 3.0 anymore because it's less efficient – you cannot seek directly to a location in a ZIP archive, unlike in a PCK file. Exposed data folders are a bad idea for performance (especially while installing or uninstalling a game), as copying a single large file is much faster than copying a lot of small files (this is true on both HDDs and SSDs).

Update (2018-07-27): Exporting to ZIP is still supported in 3.0, but it's still less optimal than using PCK files.

I'm not sure if the current .pck setups would allow it, but being able to download assets (game updates like how Minecraft does it, also for allowing multiple playable versions so users don't have to cautiously update) basically turning the executable into a launcher (even though it would still run the game) would also be useful. That way users wouldn't have to manually re-download the game for each update.

This is probably not something a game engine should tackle, as launchers often have highly-specific requirements such as user authentication, displaying news, and stuff like that.

@insomniacUNDERSCORElemon

This comment has been minimized.

Copy link
Author

@insomniacUNDERSCORElemon insomniacUNDERSCORElemon commented Dec 12, 2017

@Calinou

I understand the desire for allowing users to get native execution when the author didn't bother to export it, however a few times I've reached out to game jammers without support (who usually didn't know what Linux was) and they promptly released a working Linux download.

Kinda disappointing to need inconvenience (even if minor) to allow user workarounds like this. I mean I understand it, but I could also see a tool to extract files and create the .pck from executables being a thing. Or even lone executables not having compression at all (if it wasn't like that already) so it was really only worth it on small games (<200 MiB), incentivizing .pck use for sold/large-project games.

this isn't supported in 3.0 anymore because it's less efficient – you cannot seek directly to a location in a ZIP archive

That's sort of what I was getting at with archive splitting. Like one per DLC, one per map, one per character faction (characters, vehicles, related textures), one (or a few) for items, etc. It doesn't even have to be .zip, just SOMETHING that common tools can extract. Or even simply archive splits by directory (archive inside a folder acting as contained files, multiple splits, indexed locations etc).

If zips aren't supported for user-made resource packs that's even worse, because users wanting to replace textures/models shouldn't need to use Godot (or some unknown shareware) to compress the files.

Although solving this issue from the other side would have someone contribute code to 7zip and file-roller to compress/decompress .pck files. Or ignoring the issue would be repacking user zips into a .pck file (or multiple) when a new stack is loaded (no slowdown after .pck is created, including subsequent game launches).

This is probably not something a game engine should tackle, as launchers often have highly-specific requirements such as user authentication, displaying news, and stuff like that.

I'm not saying that Godot would need to handle that directly, just wondering if that sort of thing is viable or could be in the future. I'm actually asking as someone who'd like to release free games, so authentication isn't even needed for that.

Exposed data folders are a bad idea for performance (especially while installing or uninstalling a game), as copying a single large file is much faster than copying a lot of small files (this is true on both HDDs and SSDs).

But is it supported? I notice deleting the .pck file in a project folder will cause the executable to use source instead if it's available.

Assuming the assets are modest+well optimized (especially for a game that doesn't have tons of assets), could I allow users to have folder resource packs from a custom directory rather than a .pck or overwriting default assets?

@Calinou

This comment has been minimized.

Copy link
Member

@Calinou Calinou commented Dec 12, 2017

Assuming the assets are modest+well optimized (especially for a game that doesn't have tons of assets), could I allow users to have folder resource packs from a custom directory rather than a .pck or overwriting default assets?

The asset workflow in 2.1 allowed for this, but while this is still possible in 3.0, it's really inefficient because you also have to distribute the .import directory (which can be really large, as it can contain assets that aren't needed for running the game) and the source assets (which can also be really large).

@Calinou

This comment has been minimized.

Copy link
Member

@Calinou Calinou commented Aug 15, 2019

This was implemented in #24086, closing 🙂

@Calinou Calinou closed this Aug 15, 2019
@akien-mga akien-mga added this to the 3.2 milestone Aug 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.