Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FreeBSD support is missing #421

Closed
graemeg opened this issue Oct 14, 2018 · 108 comments
Closed

FreeBSD support is missing #421

graemeg opened this issue Oct 14, 2018 · 108 comments

Comments

@graemeg
Copy link

graemeg commented Oct 14, 2018

Environment

  • LWJGL version: 3.2
  • LWJGL build #: unknown
  • Java version: openjdk 8
  • Platform: FreeBSD 11.1
  • Module: core

Description

The Minecraft 1.12.x game worked just fine under FreeBSD with lwjgl 2.9.x but then when Minecraft 1.13.x came out, they moved up to lwjgl 3.x. Currently the lwjgl 3.x doesn't have support for FreeBSD.

Seeing as v2.9.x did, is there any changes somebody here could try and add v3.x support for FreeBSD, so we can get the wonderful Minecraft game working again. :-)

Somebody in the FreeBSD web forums have tried to get lwjgl 3.2 to build under FreeBSD 11.1, and here is how far they can. Hopefully this is a good start for anybody with more knowledge, to add the missing piece, or understand the compilation errors. I don't.

https://forums.freebsd.org/threads/minecraft-11-2-failed-to-locate-library-liblwjgl-so.66923/#post-402189

Here is a copy of that forum message:

With some help on #freebsd freenode, I am attempting to compile lwjgl 3.2 on FreeBSD 11.1-STABLE. I have modified lwjgl's build.xml file to compile using gcc8.

https://github.com/LWJGL/lwjgl3/blob/2359eb7b5a9e7e629beae95d3cdab8e997c59039/config/linux/build.xml

And here is my modified build.xml, to work with gcc8:

https://paste.pound-python.org/show/Akuv2tbz32OTgSyqZKfJ/

Using the build instruction found here:
https://www.lwjgl.org/guide#build-instructions

I am getting the following output:
https://paste.pound-python.org/show/8GO6dhwjNrQFu1ddtx2a/

@Spasi
Copy link
Member

Spasi commented Oct 14, 2018

You can find an attempt to port LWJGL (at least the parts Minecraft needs) to FreeBSD here.

Official support won't be possible without a CI service (like Travis, AppVeyor, etc.) that supports FreeBSD. Afaik, such a service does not exist and cross-compilation from Linux is painful. If someone would like to host a solution (Jenkins or whatever) for us, please let me know.

@nkfilis
Copy link

nkfilis commented Nov 25, 2018

@Spasi we could be willing to provide that, what would you need there?

@Spasi
Copy link
Member

Spasi commented Nov 26, 2018

Hey @nkfilis,

A FreeBSD environment with the tools (GCC, git, CMake, etc.) and development headers necessary to build the projects listed here. You can check the .travis.yml file in each repository to see what's involved. Currently the build workflow is:

  • I push latest version to an LWJGL-CI repository.
  • This triggers a new build at Travis CI (Linux & macOS) and AppVeyor (Windows).
  • When the build is done, the new artifacts are uploaded to LWJGL's S3 bucket.

Ideally, the FreeBSD solution would work as simply as that.

@graemeg
Copy link
Author

graemeg commented Nov 26, 2018

Just to let you know, I followed those instructions and managed to compile LWJGL3 and what was needed for Minecraft 1.13.2. It worked perfectly fine on my FreeBSD 11.2 64-bit system.

@graemeg
Copy link
Author

graemeg commented Nov 26, 2018

If @nkfilis can't manage the CI setup for FreeBSD, I'll happily give it a go too. My FreeBSD server already runs 24/7 for other open source projects.

@lwhsu
Copy link

lwhsu commented Sep 14, 2019

I think this page could help a little bit about FreeBSD CI: https://wiki.freebsd.org/HostedCI

@kmosiejczuk
Copy link

Official support won't be possible without a CI service (like Travis, AppVeyor, etc.) that supports FreeBSD. Afaik, such a service does not exist and cross-compilation from Linux is painful. If someone would like to host a solution (Jenkins or whatever) for us, please let me know.

FYI, Sourcehut supports FreeBSD and OpenBSD CI: https://man.sr.ht/builds.sr.ht/compatibility.md

I haven't used it, but I know other OpenBSD developers who said it works quite well.

@stormchaser3000
Copy link

would anyone be willing to layout what it would take to configure a CI system to support FreeBSD? I would be willing to learn and help with it, but my knowledge at the moment is rather low.

@lwhsu
Copy link

lwhsu commented Aug 4, 2020

Please check https://wiki.freebsd.org/HostedCI , there is some examples, and TravisCI also has experimental FreeBSD support now.

@Spasi
Copy link
Member

Spasi commented Mar 7, 2021

LWJGL-CI has migrated to GitHub Actions. Anyone still interested in FreeBSD support, you may want to try https://github.com/marketplace/actions/freebsd-vm. Let me know if it works out please.

@ivan-volnov
Copy link

I'm looking forward to seeing freebsd support added

@ivan-volnov
Copy link

@Spasi Added a PR for FreeBSD Github Actions to LWJGL-CI

@yurivict
Copy link

yurivict commented Jun 4, 2021

CI aside, does Gephi's master branch build/work under FreeBSD?

@grahamperrin
Copy link

@grahamperrin
Copy link

@ivan-volnov
Copy link

@grahamperrin great work! If everything is ok, post a pull request to the upstream

@grahamperrin
Copy link

Not my work, I'm just linking

@Spasi Spasi added this to Low Priority in Bindings Nov 20, 2021
@Spasi Spasi moved this from Backlog to Build Targets in Bindings Nov 20, 2021
@Spasi Spasi added this to Build Targets in Maintenance/CI Nov 20, 2021
@Irgendwer01
Copy link

@ivan-volnov that's the github of them who worked on LWJGL3 support for FreeBSD https://github.com/CRKatri/lwjgl3

@qolarnix
Copy link

Hello, what is the status of FreeBSD support? Has there been any progress?

I have tried to compile using forks that claim they work but no luck. I assume either my environment is not setup correctly or the previous methods no longer work.

@graemeg
Copy link
Author

graemeg commented Feb 23, 2023

I'm on FreeBSD 13.1 and the forks work for Minecraft up to MC v1.19.2 without issue. I'm using the [https://github.com/patrickSpaceSurfer/lwjgl3-port] fork.

The Minecraft 1.19.3 release uses a new LWJGL API call org.lwjgl.glfw.GLFWImage.malloc(int intArg, org.lwjgl.system.MemoryStack memStack) that isn't in the forks. I tried to backport that API call, but the code that auto-generates the JNI java code, doesn't seem to see my change for some reason. To be honest, I haven't tried very hard to get it working, as there isn't a big change in 1.19.2 and 1.19.3 (regarding game features). I do want to look at it again though, before the 1.20 Minecraft release comes out. :-)

On a side note: I am working on implementing full FreeBSD support in LWJGL though (not using the "linux" workaround that the forks do) - but my motivation is low, seeing as the LWJGL developers aren't really interested in supporting FreeBSD at all. Multiple people (including myself) offered to supply a CI build environment etc, but still nothing. So us FreeBSD folks will have to keep at it the hard way. :-(

@Spasi
Copy link
Member

Spasi commented Feb 23, 2023

my motivation is low, seeing as the LWJGL developers aren't really interested in supporting FreeBSD at all

This is not accurate. We obviously want LWJGL to be usable on more platforms, but every platform/architecture added means increased maintenance burden. Every platform has its quirks (e.g. the libc nightmare on Linux) and LWJGL has a lot of dependencies with all kinds of weirdness, build systems, etc. It has to be worth the effort. Please, don't forget that LWJGL is maintained by a single person.

But, hey, good news! LWJGL-CI accepts contributions! We already have LWJGL-CI#3 and I'm sure we can get it to work. The FreeBSD-VM-on-macOS-runner solution seems to be viable. Doing the same for some critical dependencies (e.g. libffi, GLFW, OpenAL-Soft) would be a great step towards adding official support for FreeBSD.

@VVD
Copy link

VVD commented Mar 6, 2024

Build without errors with system libc.so instead of the libjemalloc.so:

WRKSRC=/tmp/work/usr/ports/games/lwjgl3/work/lwjgl3-de8bd86
mkdir -p ${WRKSRC}/bin/libs/native/freebsd/x64/org/lwjgl/jemalloc
cp /lib/libc.so.7 ${WRKSRC}/bin/libs/native/freebsd/x64/org/lwjgl/jemalloc/libjemalloc.so

after apply this patch:

--- modules/lwjgl/jemalloc/src/generated/java/org/lwjgl/system/jemalloc/JEmalloc.java.orig      2023-12-18 14:22:59 UTC
+++ modules/lwjgl/jemalloc/src/generated/java/org/lwjgl/system/jemalloc/JEmalloc.java
@@ -40,27 +40,27 @@ public class JEmalloc {

         /** Function address. */
         public static final long
-            malloc_message     = apiGetFunctionAddress(JEMALLOC, "je_malloc_message"),
-            malloc             = apiGetFunctionAddress(JEMALLOC, "je_malloc"),
-            calloc             = apiGetFunctionAddress(JEMALLOC, "je_calloc"),
-            posix_memalign     = apiGetFunctionAddress(JEMALLOC, "je_posix_memalign"),
-            aligned_alloc      = apiGetFunctionAddress(JEMALLOC, "je_aligned_alloc"),
-            realloc            = apiGetFunctionAddress(JEMALLOC, "je_realloc"),
-            free               = apiGetFunctionAddress(JEMALLOC, "je_free"),
-            free_sized         = apiGetFunctionAddress(JEMALLOC, "je_free_sized"),
-            free_aligned_sized = apiGetFunctionAddress(JEMALLOC, "je_free_aligned_sized"),
-            mallocx            = apiGetFunctionAddress(JEMALLOC, "je_mallocx"),
-            rallocx            = apiGetFunctionAddress(JEMALLOC, "je_rallocx"),
-            xallocx            = apiGetFunctionAddress(JEMALLOC, "je_xallocx"),
-            sallocx            = apiGetFunctionAddress(JEMALLOC, "je_sallocx"),
-            dallocx            = apiGetFunctionAddress(JEMALLOC, "je_dallocx"),
-            sdallocx           = apiGetFunctionAddress(JEMALLOC, "je_sdallocx"),
-            nallocx            = apiGetFunctionAddress(JEMALLOC, "je_nallocx"),
-            mallctl            = apiGetFunctionAddress(JEMALLOC, "je_mallctl"),
-            mallctlnametomib   = apiGetFunctionAddress(JEMALLOC, "je_mallctlnametomib"),
-            mallctlbymib       = apiGetFunctionAddress(JEMALLOC, "je_mallctlbymib"),
-            malloc_stats_print = apiGetFunctionAddress(JEMALLOC, "je_malloc_stats_print"),
-            malloc_usable_size = apiGetFunctionAddress(JEMALLOC, "je_malloc_usable_size");
+            malloc_message     = apiGetFunctionAddress(JEMALLOC, "malloc_message"),
+            malloc             = apiGetFunctionAddress(JEMALLOC, "malloc"),
+            calloc             = apiGetFunctionAddress(JEMALLOC, "calloc"),
+            posix_memalign     = apiGetFunctionAddress(JEMALLOC, "posix_memalign"),
+            aligned_alloc      = apiGetFunctionAddress(JEMALLOC, "aligned_alloc"),
+            realloc            = apiGetFunctionAddress(JEMALLOC, "realloc"),
+            free               = apiGetFunctionAddress(JEMALLOC, "free"),
+            free_sized         = apiGetFunctionAddress(JEMALLOC, "free_sized"),
+            free_aligned_sized = apiGetFunctionAddress(JEMALLOC, "free_aligned_sized"),
+            mallocx            = apiGetFunctionAddress(JEMALLOC, "mallocx"),
+            rallocx            = apiGetFunctionAddress(JEMALLOC, "rallocx"),
+            xallocx            = apiGetFunctionAddress(JEMALLOC, "xallocx"),
+            sallocx            = apiGetFunctionAddress(JEMALLOC, "sallocx"),
+            dallocx            = apiGetFunctionAddress(JEMALLOC, "dallocx"),
+            sdallocx           = apiGetFunctionAddress(JEMALLOC, "sdallocx"),
+            nallocx            = apiGetFunctionAddress(JEMALLOC, "nallocx"),
+            mallctl            = apiGetFunctionAddress(JEMALLOC, "mallctl"),
+            mallctlnametomib   = apiGetFunctionAddress(JEMALLOC, "mallctlnametomib"),
+            mallctlbymib       = apiGetFunctionAddress(JEMALLOC, "mallctlbymib"),
+            malloc_stats_print = apiGetFunctionAddress(JEMALLOC, "malloc_stats_print"),
+            malloc_usable_size = apiGetFunctionAddress(JEMALLOC, "malloc_usable_size");

     }

@Spasi
Copy link
Member

Spasi commented Mar 7, 2024

Hey @VVD,

You could probably avoid the libc copy. Drop the jemalloc native artifact/module and use -Dorg.lwjgl.system.jemalloc.libname=libc.so.7 (or Configuration.JEMALLOC_LIBRARY_NAME.set("libc.so.7");) on FreeBSD.

I'll see if I can also make the je_ prefix customizable with another option.

@VVD
Copy link

VVD commented Mar 7, 2024

@Spasi, thanks!
But ant release -Dbuild.offline=yes -Djavadoc.skip=true -Dlocal.kotlin=${LOCALBASE}/share/kotlin -Dorg.lwjgl.system.jemalloc.libname=/lib/libc.so.7 fails.
Maybe I misunderstood your suggestion.

@VVD
Copy link

VVD commented Mar 7, 2024

Review with port: https://reviews.freebsd.org/D44263

@Spasi
Copy link
Member

Spasi commented Mar 7, 2024

Configuration options are runtime flags, they don't do anything for the build. What I meant is, build LWJGL normally (with the custom jemalloc), but when deploying an application on FreeBSD, simply skip the jemalloc native module and use the option to make the jemalloc bindings use libc.so instead.

@VVD
Copy link

VVD commented Mar 8, 2024

But I need to create port LWJGL - "rules" for build it. For build automatically in clean and "closed" environment too (tool poudriere).

@Spasi
Copy link
Member

Spasi commented Mar 8, 2024

I would personally just leave jemalloc as is. There's no serious drawback to using an non-libc allocator, even if it's basically the same implementation.

However, if you have no control of the application that's going to use your port AND you really want to use libc, then try adding Configuration.JEMALLOC_LIBRARY_NAME.set("libc.so.7"); to org.lwjgl.system.MemoryManage, above line 36.

@er2off
Copy link

er2off commented Mar 9, 2024

Review with port: https://reviews.freebsd.org/D44263

why dependencies are in EXTRACT_DEPENDS?

@VVD
Copy link

VVD commented Mar 9, 2024

I would personally just leave jemalloc as is.

I can use software from ports or from base system only - we don't have jemalloc in ports, but have it in base embedded in libc.

However, if you have no control of the application that's going to use your port AND you really want to use libc, then try adding Configuration.JEMALLOC_LIBRARY_NAME.set("libc.so.7"); to org.lwjgl.system.MemoryManage, above line 36.

lwjgl-jemalloc.jar build without errors, lwjgl-jemalloc-natives-freebsd.jar not build (as expected - use external lib instead packed in this jar).

why dependencies are in EXTRACT_DEPENDS?

Need files for post-extract:, but packed into jars during build - not required for run.

@VVD
Copy link

VVD commented Mar 12, 2024

I would personally just leave jemalloc as is. There's no serious drawback to using an non-libc allocator, even if it's basically the same implementation.

I don't have other jamalloc - only in base system.

However, if you have no control of the application that's going to use your port AND you really want to use libc,

I need to build lwjgl correctly before using it.

then try adding Configuration.JEMALLOC_LIBRARY_NAME.set("libc.so.7"); to org.lwjgl.system.MemoryManage, above line 36.

Check plz patches:

--- modules/lwjgl/jemalloc/src/generated/java/org/lwjgl/system/jemalloc/JEmalloc.java.orig      2023-12-18 14:22:59 UTC
+++ modules/lwjgl/jemalloc/src/generated/java/org/lwjgl/system/jemalloc/JEmalloc.java
@@ -40,27 +40,27 @@ public class JEmalloc {

         /** Function address. */
         public static final long
-            malloc_message     = apiGetFunctionAddress(JEMALLOC, "je_malloc_message"),
-            malloc             = apiGetFunctionAddress(JEMALLOC, "je_malloc"),
-            calloc             = apiGetFunctionAddress(JEMALLOC, "je_calloc"),
-            posix_memalign     = apiGetFunctionAddress(JEMALLOC, "je_posix_memalign"),
-            aligned_alloc      = apiGetFunctionAddress(JEMALLOC, "je_aligned_alloc"),
-            realloc            = apiGetFunctionAddress(JEMALLOC, "je_realloc"),
-            free               = apiGetFunctionAddress(JEMALLOC, "je_free"),
-            free_sized         = apiGetFunctionAddress(JEMALLOC, "je_free_sized"),
-            free_aligned_sized = apiGetFunctionAddress(JEMALLOC, "je_free_aligned_sized"),
-            mallocx            = apiGetFunctionAddress(JEMALLOC, "je_mallocx"),
-            rallocx            = apiGetFunctionAddress(JEMALLOC, "je_rallocx"),
-            xallocx            = apiGetFunctionAddress(JEMALLOC, "je_xallocx"),
-            sallocx            = apiGetFunctionAddress(JEMALLOC, "je_sallocx"),
-            dallocx            = apiGetFunctionAddress(JEMALLOC, "je_dallocx"),
-            sdallocx           = apiGetFunctionAddress(JEMALLOC, "je_sdallocx"),
-            nallocx            = apiGetFunctionAddress(JEMALLOC, "je_nallocx"),
-            mallctl            = apiGetFunctionAddress(JEMALLOC, "je_mallctl"),
-            mallctlnametomib   = apiGetFunctionAddress(JEMALLOC, "je_mallctlnametomib"),
-            mallctlbymib       = apiGetFunctionAddress(JEMALLOC, "je_mallctlbymib"),
-            malloc_stats_print = apiGetFunctionAddress(JEMALLOC, "je_malloc_stats_print"),
-            malloc_usable_size = apiGetFunctionAddress(JEMALLOC, "je_malloc_usable_size");
+            malloc_message     = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "malloc_message"),
+            malloc             = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "malloc"),
+            calloc             = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "calloc"),
+            posix_memalign     = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "posix_memalign"),
+            aligned_alloc      = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "aligned_alloc"),
+            realloc            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "realloc"),
+            free               = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "free"),
+            free_sized         = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "free_sized"),
+            free_aligned_sized = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "free_aligned_sized"),
+            mallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "mallocx"),
+            rallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "rallocx"),
+            xallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "xallocx"),
+            sallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "sallocx"),
+            dallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "dallocx"),
+            sdallocx           = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "sdallocx"),
+            nallocx            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "nallocx"),
+            mallctl            = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "mallctl"),
+            mallctlnametomib   = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "mallctlnametomib"),
+            mallctlbymib       = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "mallctlbymib"),
+            malloc_stats_print = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "malloc_stats_print"),
+            malloc_usable_size = apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "malloc_usable_size");

     }

@@ -105,11 +105,16 @@ public class JEmalloc {
     /** Use as arena index in "stats.arenas.<i>.*" mallctl interfaces to select destroyed arenas. */
     public static final int MALLCTL_ARENAS_DESTROYED = 0x1001;

+    private static String functionNamePrefix = "je_";
+
     static {
+        if (Platform.get() == Platform.FREEBSD) {
+            functionNamePrefix = "";
+        }
         // Force jemalloc to initialize before anyone else uses it.
         // This avoids a dangerous race when the first jemalloc functions are called concurrently.
         if (Platform.get() == Platform.WINDOWS) {
-            invokePV(invokePP(8L, apiGetFunctionAddress(JEMALLOC, "je_malloc")), apiGetFunctionAddress(JEMALLOC, "je_free"));
+            invokePV(invokePP(8L, apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "malloc")), apiGetFunctionAddress(JEMALLOC, functionNamePrefix + "free"));
         }
     }
--- modules/lwjgl/core/src/main/java/org/lwjgl/system/MemoryManage.java.orig    2023-12-18 14:22:59 UTC
+++ modules/lwjgl/core/src/main/java/org/lwjgl/system/MemoryManage.java
@@ -34,6 +34,9 @@ final class MemoryManage {
             String className;
             if (allocator == null || "jemalloc".equals(allocator)) {
                 className = "org.lwjgl.system.jemalloc.JEmallocAllocator";
+                if (Platform.get() == Platform.FREEBSD) {
+                    Configuration.JEMALLOC_LIBRARY_NAME.set("libc.so.7");
+                }
             } else if ("rpmalloc".equals(allocator)) {
                 className = "org.lwjgl.system.rpmalloc.RPmallocAllocator";
             } else {

How to do same for other libraries?
Where to add for example Configuration.ASSIMP_LIBRARY_NAME.set("libassimp.so");?

Look like for openxr this patch work:

--- modules/lwjgl/openxr/src/main/java/org/lwjgl/openxr/XR.java.orig    2023-12-18 14:22:59 UTC
+++ modules/lwjgl/openxr/src/main/java/org/lwjgl/openxr/XR.java
@@ -44,6 +44,9 @@ public final class XR {
      * @see #create(String)
      */
     public static void create() {
+        if (Platform.get() == Platform.FREEBSD) {
+            Configuration.OPENXR_LIBRARY_NAME.set("libopenxr_loader.so");
+        }
         create(Library.loadNative(
             XR.class,
             "org.lwjgl.openxr",

But create patch for every library is … IMHO … bad way.

Also why do you link statically with libffi? Why not dynamically as with other libraries?

Thanks!

@VVD
Copy link

VVD commented Mar 12, 2024

Need patches for libraries with other names only:

--- modules/lwjgl/shaderc/src/generated/java/org/lwjgl/util/shaderc/Shaderc.java.orig   2023-12-18 14:22:59 UTC
+++ modules/lwjgl/shaderc/src/generated/java/org/lwjgl/util/shaderc/Shaderc.java
@@ -20,7 +20,7 @@ import static org.lwjgl.system.MemoryUtil.*;
 /** Native bindings to the libshaderc C API of the <a href="https://github.com/google/shaderc/">shaderc</a> library. */
 public class Shaderc {

-    private static final SharedLibrary SHADERC = Library.loadNative(Shaderc.class, "org.lwjgl.shaderc", Configuration.SHADERC_LIBRARY_NAME.get(Platform.mapLibraryNameBundled("shaderc")), true);
+    private static final SharedLibrary SHADERC = Library.loadNative(Shaderc.class, "org.lwjgl.shaderc", Configuration.SHADERC_LIBRARY_NAME.get(Platform.mapLibraryNameBundled(Platform.get() == Platform.FREEBSD ? "shaderc_shared" : "shaderc")), true);

     /** Contains the function pointers loaded from the shaderc {@link SharedLibrary}. */
     public static final class Functions {
--- modules/lwjgl/spvc/src/generated/java/org/lwjgl/util/spvc/Spvc.java.orig    2023-12-18 14:22:59 UTC
+++ modules/lwjgl/spvc/src/generated/java/org/lwjgl/util/spvc/Spvc.java
@@ -40,7 +40,7 @@ import static org.lwjgl.system.MemoryUtil.*;
  */
 public class Spvc {

-    private static final SharedLibrary SPVC = Library.loadNative(Spvc.class, "org.lwjgl.spvc", Configuration.SPVC_LIBRARY_NAME.get(Platform.mapLibraryNameBundled("spirv-cross")), true);
+    private static final SharedLibrary SPVC = Library.loadNative(Spvc.class, "org.lwjgl.spvc", Configuration.SPVC_LIBRARY_NAME.get(Platform.mapLibraryNameBundled(Platform.get() == Platform.FREEBSD ? "spirv-cross-c-shared" : "spirv-cross")), true);

     /** Contains the function pointers loaded from the spvc {@link SharedLibrary}. */
     public static final class Functions {

@Spasi
Copy link
Member

Spasi commented Mar 12, 2024

Hey @VVD,

I don't understand what you're trying to do. The whole point of this issue was to add official support for FreeBSD to LWJGL, which is what happened. Afaict, FreeBSD ports are useful for a) building software from source and b) patching software that does not support or has issues on FreeBSD, so that it works on FreeBSD. There should be no need to patch anything in LWJGL, it already supports FreeBSD.

However, if your goal is to build LWJGL from source, then that's no simple undertaking at all. The LWJGL core is simple enough, but LWJGL is nothing without the bindings. Each binding is a whole project by itself, written in C and/or C++, with a variety of build systems and dependencies of its own. This is what we have LWJGL-CI for and keeping it healthy is a major part of my work as LWJGL maintainer. It's difficult and time-consuming.

So, you can't really claim that your LWJGL port is building from source, without also building all these dependencies. Which is something that I would find pointless and wouldn't wish for anyone to have to do. LWJGL already provides prebuilt binaries for all its dependencies, in sync with every release/build of the LWJGL core.

If on the other hand, your goal is to make an LWJGL port that does not use its own dependencies, but uses natives from other FreeBSD ports instead... I don't know, maybe I see some value there, but I also find it weird too. You're opening yourself to a world of pain with versioning issues. LWJGL bindings are developed in sync with the corresponding native libraries, i.e. updated always together, which is a guarantee that leaves little room for backward compatibility. Native libraries often break API and LWJGL breaks it at the same time. Or maybe I'm busy irl for a while and can't update LWJGL, so other ports are more up-to-date and LWJGL is behind. In any case, LWJGL has been designed to work with its own binaries. I mean, that's the whole point of LWJGL, saving Java developers from having to build native code.

Anyway, statements like:

I don't have other jamalloc - only in base system.

Make little sense to me, exactly because LWJGL comes with its own jemalloc build. I don't see how you can claim you don't have other jemalloc, a FreeBSD build comes out-of-the-box with LWJGL.

Also why do you link statically with libffi? Why not dynamically as with other libraries?

Because libffi is a critical part of LWJGL core (used to implement native-to-Java callbacks) and wanted the core to be a single native binary. It's also an implementation detail. At some point LWJGL switched to dyncall, now it's back to libffi. Note that any API/binding under the org.lwjgl.system package (e.g. libffi) is considered unstable/unsupported/subject-to-change.

@arrowd
Copy link

arrowd commented Mar 13, 2024

Afaict, FreeBSD ports are useful for a) building software from source and b) patching software that does not support or has issues on FreeBSD, so that it works on FreeBSD. There should be no need to patch anything in LWJGL, it already supports FreeBSD.

A port is a build recipe that produces a package. For user to be able to do pkg install lwjgl a corresponding port should exist.
Usually ports download sources and build them, but there are some ports that download pre-compiled stuff and just repackage it.

However, if your goal is to build LWJGL from source, then that's no simple undertaking at all. The LWJGL core is simple enough, but LWJGL is nothing without the bindings. Each binding is a whole project by itself, written in C and/or C++, with a variety of build systems and dependencies of its own. This is what we have LWJGL-CI for and keeping it healthy is a major part of my work as LWJGL maintainer. It's difficult and time-consuming.

Yes, it is a common problem for all the open source software and meta-build systems like FreeBSD Ports, Gentoo Portage, etc.

So, you can't really claim that your LWJGL port is building from source, without also building all these dependencies. Which is something that I would find pointless and wouldn't wish for anyone to have to do.

Now, this is arguable. From my experience as downstream packager I'd say that most of the software tries to use dependencies from the system. Build systems like Autotools, Meson and CMake are built around this idea - they all provide a way to look for some other library or package and link your project to it.

A bit less common scenario is when the project gives user a choice where to look for dependencies - either system or bundled ones. This is achieved with FetchContent and ExternalProject in CMake or wrap machinery in Meson.

Finally, on the rarest occasions, the project just declares that it is not intended to be built with external dependencies and just bundle everything.

From what I gather, lwjgl follows the "bundle-everything" path. In this case there is indeed no sense in fighting it on the ports side.

If on the other hand, your goal is to make an LWJGL port that does not use its own dependencies, but uses natives from other FreeBSD ports instead... I don't know, maybe I see some value there, but I also find it weird too. You're opening yourself to a world of pain with versioning issues.

Yes, versioning issues is a thing, but so far the world keeps up with it. We have more than 30k ports in our tree and only a miniscule of them are bundling their stuff. This approach also has its own benefits:

  • Libraries are really shared in the memory.
  • libfoo provided by our ports may have additional patches critical for FreeBSD, which we haven't upstreamed yet. If you build your own libfoo, it won't have these patches.
  • The weight of managing all these deps falls on the project developer rather than downstream packagers. I regard as a wrong thing. Developers should develop and downstream packagers' job is to tie all things up into a working combination.

Of course I don't request lwjgl to change the way it handles deps, just sharing my experience from the other side of borderline.

@Spasi
Copy link
Member

Spasi commented Mar 13, 2024

Hey @arrowd,

Thanks for the interesting input, it honestly makes a lot of sense for native software. However, doing

pkg install lwjgl

is not something a Java developer would want to do. Java dependencies are managed via Maven/Gradle, there's no point in installing a Java library outside the local Maven repo. However, a software product that uses LWJGL may want to be installable that way. Which includes NOT using some of LWJGL's prebuilt binaries and overriding them with other ports. I.e. this is something that the product developer should decide and control, it should not be enforced by an LWJGL port.

Also, note that incompatibilities may also arise in the opposite direction. There are some LWJGL dependencies with patches that are Java/LWJGL-specific and will never be included in the upstream project. For example, GLFW's async implementation for macOS.

@yurivict
Copy link

yurivict commented Mar 13, 2024

@Spasi

However, doing
pkg install lwjgl
is not something a Java developer would want to do.

Regular users want to use pkg install xx for all packages. It's too difficult for them to know separate recipes for Java software, for Python software, for Rust software, for Haskel software, etc, etc. Using a single pkg install xx command is a lot easier and should be available for as much software as possible.

@VVD
Copy link

VVD commented Mar 14, 2024

Added ant build options -Duse.libffi.so=true and -Dlibffi.path=/usr/local/lib to allow link with libffi dynamically:

--- config/freebsd/build.xml.orig       2023-12-18 14:22:59 UTC
+++ config/freebsd/build.xml
@@ -159,16 +159,18 @@
                     <include name="${module.lwjgl}/jawt/src/generated/c/*.c" if:true="${binding.jawt}"/>
                 </fileset>
             </source>
-            <beforeLink>
+            <beforeLink unless:true="${use.libffi.so}">
                 <parallel threadsPerProcessor="2" failonany="true" unless:set="lib-dependencies-uptodate">
                     <update-dependency module="core" artifact="core/libffi.a"/>
                 </parallel>
             </beforeLink>
             <link>
-                <fileset dir="${lib.native}/org/lwjgl">
+                <fileset dir="${lib.native}/org/lwjgl" unless:true="${use.libffi.so}">
                     <include name="libffi.a"/>
                 </fileset>
                 <arg value="-ldl"/>
+                <arg value="-lffi" if:true="${use.libffi.so}"/>
+                <arg value="-L${libffi.path}" if:set="libffi.path"/>
             </link>
         </build>
 

Look like it's last patch for now. :-D

@VVD
Copy link

VVD commented Mar 24, 2024

Time to test run!

I got error during run:
Caused by: java.lang.NullPointerException: A required function is missing: glfwGetPreeditCursorRectangle
It can't find one function glfwGetPreeditCursorRectangle?
libglfw.so from FreeBSD ports doesn't have glfwGetPreeditCursorRectangle:

$ strings /usr/local/lib/libglfw.so | grep glfwGetPreeditCursorRectangle

libglfw.so from bundle of the Minecraft 1.20.4 for Linux with lwjgl 3.3.2:

$ strings versions/1.20.4/natives/libglfw.so | grep glfwGetPreeditCursorRectangle

Sources of the glfw from https://github.com/glfw/glfw:

$ git clone https://github.com/glfw/glfw.git
$ grep -R glfwGetPreeditCursorRectangle glfw

File libglfw.so downloaded if build lwjgl3 without -Dbuild.offline=true:

$ strings bin/libs/native/freebsd/x64/org/lwjgl/glfw/libglfw.so | grep glfwGetPreeditCursorRectangle
glfwGetPreeditCursorRectangle

With this libglfw.so software run without errors!

Do you use this set of patches: glfw/glfw#2130?

More logs:

Time: 2024-03-24 18:47:36
Description: Initializing game

java.lang.ExceptionInInitializerError
        at org.lwjgl.glfw.GLFW.nglfwGetError(GLFW.java:1267)
        at org.lwjgl.glfw.GLFW.glfwGetError(GLFW.java:1296)
        at epf.a(Window.java:176)
        at com.mojang.blaze3d.platform.GLX._initGlfw(GLX.java:69)
        at com.mojang.blaze3d.systems.RenderSystem.initBackendSystem(RenderSystem.java:707)
        at evi.<init>(SourceFile:512)
        at net.minecraft.client.main.Main.main(SourceFile:223)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at net.minecraft.launchwrapper.Launch.launch(Launch.java:159)
        at net.minecraft.launchwrapper.Launch.main(Launch.java:30)
Caused by: java.lang.NullPointerException: A required function is missing: glfwGetPreeditCursorRectangle
        at org.lwjgl.system.APIUtil.requiredFunctionMissing(APIUtil.java:147)
        at org.lwjgl.system.APIUtil.apiGetFunctionAddress(APIUtil.java:141)
        at org.lwjgl.glfw.GLFW$Functions.<clinit>(GLFW.java:123)
        ... 11 more


A detailed walkthrough of the error, its code path and all known details is as follows:
---------------------------------------------------------------------------------------

-- Head --
Thread: Render thread
Stacktrace:
        at org.lwjgl.glfw.GLFW.nglfwGetError(GLFW.java:1267)
        at org.lwjgl.glfw.GLFW.glfwGetError(GLFW.java:1296)
        at epf.a(Window.java:176)
        at com.mojang.blaze3d.platform.GLX._initGlfw(GLX.java:69)
        at com.mojang.blaze3d.systems.RenderSystem.initBackendSystem(RenderSystem.java:707)
        at evi.<init>(SourceFile:512)

-- Initialization --
Details:
        Modules:
Stacktrace:
        at net.minecraft.client.main.Main.main(SourceFile:223)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at net.minecraft.launchwrapper.Launch.launch(Launch.java:159)
        at net.minecraft.launchwrapper.Launch.main(Launch.java:30)

@VVD
Copy link

VVD commented Mar 24, 2024

Build glfw with patches allow to run software.

@VVD
Copy link

VVD commented Mar 24, 2024

It use llvmpipe (software rendering) if file /usr/local/lib/libGLX.so.0.0.0 exist.
After mv /usr/local/lib/libGLX.so.0.0.0 /usr/local/lib/libGLX.so.0.0.0_ it use hardware rendering from proprietary NVIDIA driver.
Look same as here: #842 (comment)

In the first 3.3.3 snapshot, I could add a command line flag that makes LWJGL call GLFWNativeGLX.setPath(GL.getFunctionProvider()) internally on glfwInit. This would let you run the unmodified application by forcing GLFW to load libGL.so.1 directly.

@Spasi, did you added this option?

Patch didn't help:

--- modules/lwjgl/glfw/src/generated/java/org/lwjgl/glfw/GLFWNativeGLX.java.orig        2023-12-18 14:22:59 UTC
+++ modules/lwjgl/glfw/src/generated/java/org/lwjgl/glfw/GLFWNativeGLX.java
@@ -116,6 +116,7 @@ public class GLFWNativeGLX {
      * @param sharedLibrary a {@code FunctionProvider} instance that will be cast to {@code SharedLibrary}
      */
     public static void setPath(FunctionProvider sharedLibrary) {
+        sharedLibrary = GL.getFunctionProvider();
         if (!(sharedLibrary instanceof SharedLibrary)) {
             apiLog("GLFW OpenGL path override not set: Function provider is not a shared library.");
             return;

-Dorg.lwjgl.opengl.libname=/usr/local/lib/libGL.so.1, -Dorg.lwjgl.opengl.libname=/usr/local/lib/libGL-NVIDIA.so.1 and patch:

--- modules/lwjgl/opengl/src/main/java/org/lwjgl/opengl/GL.java.orig    2023-12-18 14:22:59 UTC
+++ modules/lwjgl/opengl/src/main/java/org/lwjgl/opengl/GL.java
@@ -99,7 +99,9 @@ public final class GL {
         switch (Platform.get()) {
             case FREEBSD:
+                GL = Library.loadNative(GL.class, "org.lwjgl.opengl", Configuration.OPENGL_LIBRARY_NAME, "libGL-NVIDIA.so.1", "libGL.so", "libGL.so.1");
+                break;
             case LINUX:
                 GL = Library.loadNative(GL.class, "org.lwjgl.opengl", Configuration.OPENGL_LIBRARY_NAME, "libGLX.so.0", "libGL.so.1", "libGL.so");
                 break;
             case MACOSX:
                 GL = Library.loadNative(GL.class, "org.lwjgl.opengl", Configuration.OPENGL_LIBRARY_NAME, "/System/Library/Frameworks/OpenGL.framework/Versions/Current/OpenGL");

didn't help too.

Only mv /usr/local/lib/libGLX.so.0.0.0 /usr/local/lib/libGLX.so.0.0.0_ help.

@arrowd
Copy link

arrowd commented Mar 25, 2024

IIRC, this problem is specific to NVIDIA proprietary driver, although it arranges a libmap.conf file to make applications load proper library. Anyways, maybe you should try nvidia-drm-kmod to hook NVIDIA driver into DRM infrastructure.

@VVD
Copy link

VVD commented Mar 25, 2024

IIRC, this problem is specific to NVIDIA proprietary driver, although it arranges a libmap.conf file to make applications load proper library. Anyways, maybe you should try nvidia-drm-kmod to hook NVIDIA driver into DRM infrastructure.

Don't think nvidia-drm-kmod support my GeForce 9800 GT.

@VVD
Copy link

VVD commented Mar 25, 2024

Found other bug (maybe offtopic here): can't set multiple paths in -Djava.library.path and -Dorg.lwjgl.librarypath.
Tried -Dorg.lwjgl.librarypath=/usr/local/lib/lwjgl3:/lib or -Djava.library.path=/usr/local/lib/lwjgl3:/lib - found only libs in /lib. If -Dorg.lwjgl.librarypath=/usr/local/lib/lwjgl3 or -Djava.library.path=/usr/local/lib/lwjgl3 can't find /lib/libc.so.7.

@Spasi
Copy link
Member

Spasi commented May 26, 2024

Hey all,

What's the ABI compatibility story on FreeBSD?

Current binaries are built on FreeBSD 13.2 with Clang 14. 13.3 comes with Clang 17 and there's also 14.0 (released earlier than 13.3?) with Clang 16.

I'm guessing that a build on 13.3 will work on 14.0, but will it also be compatible with 13.0-13.2?

@VVD
Copy link

VVD commented May 26, 2024

I'm guessing that a build on 13.3 will work on 14.0

Yes, with COMPAT_FREEBSD13 compatibility layer (default ON).

but will it also be compatible with 13.0-13.2?

It can work, but no warranty.
13.0 and 13.1 reached EOL. 13.2 will EOL in ~1 month (June 30, 2024): https://www.freebsd.org/security/#sup
After this date binary packages for 13 will build on 13.3.

there's also 14.0 (released earlier than 13.3?) with Clang 16.

14.1 release in 3 weeks: https://www.freebsd.org/releases/14.1R/schedule/
14.0 will EOL ~Sep 30, 2024.
After this date binary packages for 14 will build on 14.1.

@er2off
Copy link

er2off commented May 26, 2024

released earlier than 13.3?

Yes, just like Linux kernel.

About binary compatibility, current FreeBSD config is compatible with binaries from FreeBSD 4 (I think it needs to rebuild or pack libraries into package to make old binaries workable)

@VVD
Copy link

VVD commented May 26, 2024

I think it needs to rebuild or pack libraries into package to make old binaries workable

All necessary system libraries are in ports:

[/usr/ports]$ ls -d misc/compat*x
misc/compat10x
misc/compat11x
misc/compat12x
misc/compat13x
misc/compat4x
misc/compat5x
misc/compat6x
misc/compat7x
misc/compat8x
misc/compat9x

@Spasi
Copy link
Member

Spasi commented May 27, 2024

Thank you @VVD and @er2off,

I'll try to follow FreeBSD's release schedule and will be dropping support for EOL versions regularly, maybe 3-6 months after the EOL date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Maintenance/CI
Build Targets
Development

No branches or pull requests