build-release: allow do build dpk or merged dpk+vm as a mod

[illwieckz]

We already have build-release vm command, it produced a vm_0.dpk only containing the .nexe files.

Now we can do build-release base to build unvanquished_0.dpk from pkg/unvanquished_src.dpkdir. It only contains built files from the dpkdir.

We can also do build-release mod to buils unvanquished_0.dpk with both files from pkg/unvanquished_src.dpkdir and the .nexe files.

It uses Urcheon if available, otherwise it fallbacks on a script mimicking what does Urcheon (but less complete and with no file conversion). It may be good enough to build a mod dpk with a single command.

For debugging purposes the --fake-urcheon option makes possible to use the script mimicking Urcheon even if Urcheon is available.

The --vmpak-name <name> and --basepak-name <name> options can
be used to build dpk paks with non-default names.

As usual the build folders are stored in build/target and the produced files are stored in build/release.

[illwieckz]

So one can do that (assuming being called from within Unvanquished/ repository):

build-release mod --reference v0.52.0 --pak-name nightly

And get a delta pak named build/release/nightly_0.dpk, containing both data crunched from unvanquished_src.dpkdir and nexe files.

To get a versionned dpk, one needs #16 to be merged as well.

[illwieckz]

OK, now that #16 is merged, one can do:

build-release mod --reference v0.52.0 --pak-name nightly -v

And get:

build/release/nightly_0.53.2-20220909-225244-265fe4834.dpk

This with crunched assets, nexe debug symbols, etc. Just like our release builds.

[slipher]

I'm skeptical that build-release is a good entry point for modding. The VMs are comparatively easy to build - the default options should work fine. There aren't any extra optimization options or static linking stuff which makes building the engine complicated. So going through this script is a pointless extra layer.

The only exception would be if modders want to have Breakpad symbols. Seems unlikely to me though. Is that the use case you have in mind?

Being opposed to complex shell scripts, I would rather see an Urcheon option that takes the path to a build directory containing VMs to include in the pak.

[illwieckz]

I remind the whole purpose of this script (and doing things like embedding a symbols.7z we don't use as is in releases) was to make sure we can distribute updates and fixes for the game while not loosing the ability to debug them. This happened many time that we distributed undebuggable hot fixes and this is what motivated me to start the writing of this script. I then extended the script to also be usable for releasing the game and build engines to drop the other existing script. The initial purpose of the script was to produce release-like dpk hotfixes (so with symbols, etc.). So in fact this is a modding/hotfix script extended for release (“eating own dog food”, etc.)

As said in #21 “People are complaining for long time doing a mod isn't easy, build-release being part of Unvanquished/Dæmon and [this] being merged would make it much more convenient for people.”

Not doing this here would make modders second-class citizens and would end in code duplication and discrepancy.

I want modders not being second class citizens, and in fact modders are us when we do hotfixes, nightly servers or host gameplay or feature experiments. This can benefit the nightly server, the tutorial effort, the lua effort, etc. For example, right now our nightly server is broken when we release the game because of the dpk is inexistent to get a nexe merged until we also modify the dpkdir.

Even if we go the road of splitting base dpk and nexe dpk, the ability to build the basepak from this script is more than welcome because everyone wants the unvanquished dpkdir to be moved back in Unvanquished repository and then we need an easy way to build it outside of UnvanquishedAssets.

Those enhancements are also purposed to reduce the amount of manual steps done at release time. I'm also thinking about extending build-release to also build UnvanquishedAssets at some point, so the Docker file can produce everything but the macos engine build.

Why I find this script a good entry point is that:

• most of the things are already wired-up there (I first wrote this script to produce dpk hotfixes!), that's why I went extending this script instead of restarting such implementation from scratch in cmake or something else.
• I want modders to get the same convenience, ease, automation, optimizations, etc. we use for releasing. So by providing modders our release tools we make sure everyone can get the state of the art.

Being opposed to complex shell scripts, I would rather see an Urcheon option that takes the path to a build directory containing VMs to include in the pak.

I also have that idea but even if that gets implemented in Urcheon at some point, it would just mean to me I'll make build-release rely on Urcheon more. So to me it's just an implementation detail for a feature that is wanted in all cases. Having this script telling urcheon to merge the nexe or zipping the nexe into the dpk provided by Urcheon is a matter of modifying 5 lines or so. This is really a detail on that script scope.

I still want build-release to build the base pak and provide a merged dpk with base pak and nexe in one run, whatever the underlying implementations.

[illwieckz]

Most of the diff size here is the fake Urcheon implementation. I thought it would be the most controversial part but I want this because it makes making a mod very easy and not crunching a mod is OK if that allows more modders to use this script out of the box without having to set-up the complete Urcheon environment and deps.

Here is some file sizes comparison, first is a crunched delta pak (from 0.52.0 to master), then full pak (master), crunched then not crunched:

8,6M	nightly-delta-urcheon_0.dpk
18M	nightly-full-urcheon_0.dpk
22M	nightly-full-copy_0.dpk

[slipher]

Thanks for explaining your thinking.

Most of the diff size here is the fake Urcheon implementation.

How hard would it be to make fake-urcheon a separate script?

Also I would want --fake-urcheon to be required to use the fake version. I hate that paradigm of building software with different features depending on what random stuff is found on the system. In this case it's a silent downgrade to second class :)

Finally, producing a pak named unvanquished by default seems very dangerous. I saw someone in IRC recently asking if the right way to distribute a mod is to make a pak named unvanquished_something... A different pak name should be a required argument with mod (and also data I guess).

[illwieckz]

How hard would it be to make fake-urcheon a separate script?

The real obstacle here is that the fake-urcheon relies on the package function, so externalizing fake-urcheon would require to either duplicate that package function, eitther externalize this one too. But my main concern is the script to not be self contained. Making fake-urcheon a separate script is probably only an option if the build-release script is part of Unvanquished or Dæmon repository.

Also I would want --fake-urcheon to be required to use the fake version. I hate that paradigm of building software with different features depending on what random stuff is found on the system. In this case it's a silent downgrade to second class :)

Good idea. It's OK to me, I can turn the lack of Urcheon into an error with an advice to use --fake-urcheon as a workaround.

[illwieckz]

• I renamed data target to base to build unvanquished_src.dpkdir (without nexe) as it is the basepak and I reserve data target to build UnvanquishedAssets in the future.
• Missing Urcheon now raises an error telling to use --fake-urcheon instead of printing a warning and using the fake urcheon script in the back of the user.

[slipher]

The real obstacle here is that the fake-urcheon relies on the package function, so externalizing fake-urcheon would require to either duplicate that package function, eitther externalize this one too.

Duplicating is fine. It's a small function.

But my main concern is the script to not be self contained.

How is shelling out to Fake Urcheon any less self-contained than shelling out to Real Urcheon?

Making fake-urcheon a separate script is probably only an option if the build-release script is part of Unvanquished or Dæmon repository.

I don't follow.

[illwieckz]

How is shelling out to Fake Urcheon any less self-contained than shelling out to Real Urcheon?

The idea is to have a self-contained build-release with a working alternative for those who don't have strong enough need for setting-up the full Urcheon and its environment.

Right now, someone already able to build the game code can get a simple mod dpk just by relying on the shell and 7z. I can understand getting the full featured Urcheon may look like a huge step (and be discouraging) just to distribute a simple mod. That's the motivation behind the fake urcheon. If Urcheon was available in common distros or other things making it easy to get working I would not even consider an alternative. :D

Making fake-urcheon a separate script is probably only an option if the build-release script is part of Unvanquished or Dæmon repository.

I don't follow.

As long as build-release lives in a separate repository that is not Unvanquished, people may just want to donwload the script without cloning build-release repository. That's why I like it being self-contained. Once this script and its possible dependencies like fake-urcheon are in Unvanquished or in Unvanquished/daemon, the best way to use the script would be to use it from the Unvanquished repository itself, meaning sidecar files like fake-urcheon would be guaranteed to be there as well. Moving build-release to Unvanquished or Unvanquished/daemon makes script splitting less risky.

[slipher]

The self-containment consideration seems bogus to me, as if people will be able to mod Unvanquished without knowing how to clone a Git repository. But I'm not against moving all the release scripts to Unvanquished.

[necessarily-equal]

+1 on that, that's a step towards making modding simpler. I agree with the idea of making Urcheon optional too, because people would get scared by the perceived complexity and the dependency list otherwise (q3map, crunch, python, sloth, etc).

Long term, if we want to have a make dpk target, we'll need to temperate this script a bit and keep only the packaging part. So maybe it would be best to have "fake-urcheon" as a separate script. It doesn't have to do argument parsing and all, as it wouldn't be called manually. You could also pass variables from the env: USE_CRUNCH=1 ./fake-urcheon "$src" "$dest". Note I don't think it's a blocker for merging.