Improve ease of packaging for Non Linux/Windows/OS X (and esp. OpenBSD)

[Izder456]

Goal

Help better support system lwjgl(,3) on OpenBSD, to let us use this as our minecraft launcher. As our current games/blockgame port is unmaintained.

Motivation

I want to package your software for OpenBSD via the ports tree. By default, PrismLauncher tries to download the Linux version of lwjgl(,3) which will not work for us. We have lwjgl(,3) in our ports tree, as upstream doesn't package for OpenBSD.

Specification

I took some patches from the FreeBSD guys (who got this functional) to get it to take the system lwjgl installs, but i cant get versions 1.13.0+ to work cos of a failed load of liblwjgl.so. we do have liblwjgl64.so though, so i tried symlinking but get the same issue.

See the mailing list thread for more information: https://marc.info/?l=openbsd-ports&m=173800868401324&w=2

prismlauncher-9.2.tar.gz

I have attached the port in this issue. Any help would be appreciated.

Drawbacks

None. Prism Launcher should be more portable to non Linux/Windows/OS X

Unresolved Questions

Do people want Prism Launcher on OpenBSD? Do the devs of Prism Launcher want to work with the ports@ mailing list? Any help would be appreciated.

Alternatives Considered

Maybe add a customizeable configure-time cmake flag to use system lwjgl(,3) instead.

This suggestion is unique

• I have searched the issue tracker and did not find an issue describing my suggestion, especially not one that has been rejected.

You may use the editor below to elaborate further.

No response

[Ryex]

The main issue here is that there are at least three different mutually incompatible lwjgl versions that are required to run all our currently supported version of Minecraft. So it's usually never as simple as pulling in the system lwjgl, there are all sorts of strange incompatibilities if your not on exactly the right version. That's why we have the current system of supplying those known working versions via our meta server.

one of our maintainers (@getchoo ) is currently working on a side project set up a build pipeline for the versions of lwjgl that we don't build ourselves and there is at least some work to pull in some community work on RiscV lwjgl versions.

if you are willing to put in some work helping set up a BSD based pipeline and perhaps help patch Prism to detect the BSD variants necessary I think we would be happy to put those versions into the meta server to fully support the BSDs.

[getchoo]

I think a quick solution in the meantime would be introducing the build option mentioned here, as well as add more options for other natives. Alpine Linux for example allows configuring libjemalloc here and patches out default instance options for natives here; the former could be easily upstreamed from a quick look, and the latter should be simple to implement properly

one of our maintainers (@getchoo ) is currently working on a side project set up a build pipeline for the versions of lwjgl that we don't build ourselves and there is at least some work to pull in some community work on RiscV lwjgl versions.

This is the long term goal though, definitely. I don't think BSD should be that hard to add onto it as there's support in Nixpkgs

we do have liblwjgl64.so though, so i tried symlinking but get the same issue.

Weird that symlinking doesn't work...but I'm also a little curious about why OpenBSD ships a library of that name. I've never seen it on other platforms, and even the official lwjgl binaries for FreeBSD don't refer to it as that -- maybe it's hardcoded in the jars themselves rather than in the launcher?

[Izder456]

I think a quick solution in the meantime would be introducing the build option mentioned here, as well as add more options for other natives. Alpine Linux for example allows configuring libjemalloc here and patches out default instance options for natives here; the former could be easily upstreamed from a quick look, and the latter should be simple to implement properly

one of our maintainers (@getchoo ) is currently working on a side project set up a build pipeline for the versions of lwjgl that we don't build ourselves and there is at least some work to pull in some community work on RiscV lwjgl versions.

This is the long term goal though, definitely. I don't think BSD should be that hard to add onto it as there's support in Nixpkgs

Please no.

Nix is not a portable solution to the *BSDs. Some sort of automation that is truly cross platform would be far more optimal. Whatever you see fit and is properly compatible is IMHO the correct solution.

Also, what I presume to be effectively cross-compiling for the unstable ABI of OpenBSD is unwise. The OpenBSD ports tree is the way we do things. When the base system changes something that breaks a lot of ports it shouldn't, porters like myself work hard with upstream to make sure that the software works for everyone (regardless of platform) correctly. Usually this is as simple as removals of unneeded GNUisms and alike, but it can look more complex such as trying to build a more portable solution like this RFC is attempting to kickstart.

By automating this via some sort of Nix-based pipeline, you completely ignore the regression testing of real humans on the ports@ mailing list, and will likely unintentionally break compat with the systems you want to support better. I can gladly work with you on giving feedback on something to better support us on OpenBSD, but I'm not too knowledgeable on the Java-side of things here. Sorry ahead of time if this is too blunt.

we do have liblwjgl64.so though, so i tried symlinking but get the same issue.

Weird that symlinking doesn't work...but I'm also a little curious about why OpenBSD ships a library of that name. I've never seen it on other platforms, and even the official lwjgl binaries for FreeBSD don't refer to it as that -- maybe it's hardcoded in the jars themselves rather than in the launcher?

I am not sure on the reason, but the do-install target of the games/lwjgl port does this:

do-install:
	${INSTALL_DATA_DIR} ${LWJGL_HOME}
	for x in ${WRKSRC}/libs/*.jar; do \
		${INSTALL_DATA} $$x ${LWJGL_HOME}/; \
	done
	${INSTALL_DATA} ${WRKSRC}/libs/openbsd/liblwjgl${M_ARCH}.so ${LWJGL_HOME}/

This appears to install the .so according to the machine architecture. In the case of a 64-bit system, it names it according to that. I presume on a 32-bit machine, the 64 would become 32. I'd ask MAINTAINER why this is, as I personally have no clue. It is very strange.

Our games/lwjgl3 port is a tad out of date, and is built against a patched version of lwjgl3 for OpenBSD as upstream dropped support for OpenBSD. Fortunately, a recent commit seems to update the fork to a newer version. Hopefully the port will be updated to incorporate this commit/release.

Hopefully this helps.

Appreciated-

[getchoo]

Nix is not a portable solution to the *BSDs. Some sort of automation that is truly cross platform would be far more optimal. Whatever you see fit and is properly compatible is IMHO the correct solution.

I think there might be a misunderstanding here. Nix will not be required on any end user's or packagers system, and remain a completely separate (optional) way of building the launcher through the Flake. This is only replacing the build system for the binaries we distribute remotely in our meta, mainly to

a) Ensure the binaries we distribute come directly from us or Mojang (currently some are sourced from community members who do not have transparent build processes)
b) Allow users to reproduce the builds locally to independently verify they're coming from the sources we claim
c) As Ryex said above, support more exotic architectures that Mojang does not support (i.e., riscv64)

This shouldn't introduce any more portability issues than we currently have, with random binaries obviously not working on all Unix platforms

Also, what I presume to be effectively cross-compiling for the unstable ABI of OpenBSD is unwise. The OpenBSD ports tree is the way we do things.

I'm not super familiar with OpenBSD and it's binary compatibility, but this does sound similar to what we already deal with on many non-GNU/FHS/etc Linux distributions. In those cases, we do actively encourage the use of native packages built against the system over the binaries we distribute. This is why our built in Java manager for example is meant to be "opt-in" on Unix platforms when compiling from source -- as again, binaries compiled against Ubuntu aren't going to work everywhere. Packagers using a build option similar to our Java manager for all native libraries would definitely make more sense in these cases

I'll make sure to keep this in mind once it's implemented; OpenBSD will most likely be excluded entirely similar to Alpine Linux, etc. so that the maintainers of those systems can take better care of it

Sorry ahead of time if this is too blunt.

Don't worry about it. Binary compatibility is a mess, naive upstream assumptions suck, and that very commonly ends up confusing your end users until you pick up the slack. I deal with similar things in Nixpkgs all the time lol

This appears to install the .so according to the machine architecture. In the case of a 64-bit system, it names it according to that. I presume on a 32-bit machine, the 64 would become 32. I'd ask MAINTAINER why this is, as I personally have no clue. It is very strange.

Yeah I'd definitely try reaching out, as it'd be a bit surprising if this naming doesn't affect other projects linking against it. Maybe there are other ports that depend on it with patches?

upstream dropped support for OpenBSD

Yikes, TIL. Guess that's another reason for us to not try and make binaries 😆

[Izder456]

I'm typing this from a mobile so apologies for poor formatting. Assuming this meta build server is reliable enough, it might be an option to rebase the lwjgl ports to piggy back off of them, maybe? I dunno, just an idea. @getchoo

[getchoo]

it might be an option to rebase the lwjgl ports to piggy back off of them

For some builds we probably could (and currently do IIRC), but older versions of LWJGL for older versions of Minecraft don't support many architectures, which is where we fallback onto external sources. riscv64-linux for example wasn't built upstream until 3.3.2, and FreeBSD until 3.3.4

[Ryex]

So if I'm following getchoo we're thinking a nix based pipeline to build binaries that will hopefully work and some patching of the launcher to allow pulling in system library for all relavent lwjgl versions. and possibly allow spesifying the path to those if they are in none standard paths/names.
But,in the end @ports may need to do some work to provide some kind of lwjgl-minecraft package with the relavent versions needed?

[Izder456]

why would we need both? couldn't the ports tree just build the patched up sources that the meta server creates and call it a day?

is the meta build server also going to be used for regression testing or something?

I don't see reason in having both here.

[Ryex]

well, we'll still need the nix pipelines for our arm linux, macos, and riscv64 binaries. but it would be a good prototyping stage for the *BSDs. @ports could take that work to build the lwjgl versions needed.

[getchoo]

but it would be a good prototyping stage for the *BSDs

If OpenBSD's ABI is as unstable as described above (I have no reason to believe it's not), I think we'd just stick with targeting only FreeBSD there. It's supported by upstream and better supported in Nixpkgs anyways

Would probably be a good idea to still organizes patches and stuff in a way that's easily usable for downstreams like the ports tree, though

[Trial97]

This is related to #3282