Gradle build

[Arcnor]

Is this considered/going to be considered?

I think it's the fastest way of getting LWJGL3 to be used by other projects without the current LWJGL2 hassle of generating POMs separated from the main build. It can be used to compile without ant at all, or it can be mixed.

I started adding support and got the kotlin plugin to compile all the sources, but got confused about the Generator.kt

If you're not planning on adding this, please close as wont fix so I don't spend any more time on it :D

[Spasi]

Yes, building with gradle would be interesting. Let me know if you still have trouble with the Generator. If you got it to compile, then it's a simple Java application, see the "generate" Ant target.

[Arcnor]

Ok, I'll continue then :).

Yeah, I know it's just a program, but it was strange that the Kotlin code was compiled using a Java tool. I need to look a bit more into it.

Also, do you mind if I move the code around? For ease of integration with gradle, it will be nice to have a module per "unit", i.e. "core" for the core library, "templates" for the templates library, and "natives" for the native library (or maybe "natives-linux", "natives-macosx" and "natives-windows", I need to check what needs to be generated). That means the code for each module will be inside /src/main/[java|kotlin]

[Spasi]

I understand the src/main/java thing is the Maven convention (over configuration), but it's not really compatible with how this project is set up. Especially, if you consider the generated code that's written in a different folder hierarchy (and I'd like to keep it this way).

I believe you can break the convention and configure everything manually in Gradle.

[Arcnor]

Yeah, I can break the convention, although some things like the Kotlin plugin might not like it. Also, it will be way more verbose.

I'll try to see what I can do.

Also, if you change where the generated code goes, IDEA will pick that up automatically and will detect the code as generated. I like that, but this it's your project ;)

[Spasi]

Thanks, I didn't know that feature. I didn't have to move the generated code though, just marked the root as generated. Noticed that IDEA disables inspections for the generated classes, do you know if it affects anything else?

[Arcnor]

You can either mark it as generated all the time, or put it on target/generated. Then IDEA will mark it automatically.

It doesn't affect much, although if you try to modify a generated class, IDEA will warn you.

[Arcnor]

Ok, an update on this.

I've managed to compile everything, although hardcoding some paths for now to make it simpler. Also, I managed to do it moving the code around like I said before.

I'm now trying to convert my code to use your non standard paths. However, it seems that the Kotlin plugin chokes on that, it compiles the Kotlin code fine, except for Types.kt (that depends on Java code, PointBuffer, that's going to be compiled at the same time). When I move everything into src/main/java and src/main/kotlin (it compiles Kotlin and Java code separatedly) everything works again.

I'm going to continue trying to make it work, but if I can't, will you accept that module (src/templates) being converted to some more standard? I'll also ask on the Kotlin forums in case I'm doing something wrong.

[Spasi]

The PointerBuffer dependency is resolved with classpath += "src/templates" in config/Templates.ktm. You should make sure that kotlinc is configured with Templates.ktm, because in the future it will be used for filtering the supported bindings. You'll be able to do custom builds, for example without OpenCL or with only the core OpenGL functionality, filter out old/obsolete extensions, etc.

[Arcnor]

Ok, I'll try to use the Templates.ktm file, although so far no luck with that one. Also, what that script is doing can be done with Gradle as well, and using Gradle flavors it will be even more elegant. But I'll try to make it work.

[big-guy]

hey @Arcnor, have you done anything with this yet that's available publically?

[virtuoushub]

Hi @Arcnor , like @big-guy, I am wondering if you have any code for the Gradle build I can take a look at.

[Arcnor]

Hi,

Sorry, but I don't have anything on the intertubes yet. I got it to compile, more or less, but never got it to use that Templates.ktm file (asked in their forums about the .ktm file, but got no response other than "use the mechanism native to Gradle: https://devnet.jetbrains.com/message/5527371)

I really want this to get with some project of mine (or even LibGDX) sooner or later, so if you can help, that'd be great :)

I'll try uploading what I had as soon as I get the time.

[dhild]

@Arcnor I think I figured out the kotlin configuration; I ended up just adding 'config' as a source directory for the templates compile. https://github.com/dhild/lwjgl3/tree/gradle_project if anyone is interested. Currently the java & kotlin files compile (haven't verified that they're functionally identical), and I've started looking at using gradle's native compilation mechanisms. Still a work in progress, but might give @big-guy or @virtuoushub a reference.

[dhild]

I've spent a little more time figuring out the native compilation, and got it working on windows. Using entirely the gradle build for compilation, I was able to get the WGLDemo and UniformArrayDemo classes to run.
Still remaining:

• Setting up the Linux build (I'll probably do this later)
• Setting up the Mac build (I can write the script, but I don't have a Mac to test with)
• Consolidating the existing config (there's a lot of "if windows ..." statements, this could be handled better)
• Extending with a few more tasks, etc. to fully replicate the ant scripts:
  • Javadoc needs some tweaking
  • Release task has not been started
  • demo task (or tasks) could be created
  • I'm still figuring out exactly how the generated & wiki pieces fit into the build / workflow

I'll keep updating the fork I made as I progress.

[octylFractal]

@dhild I've got a Mac.

[dhild]

@kenzierocks Cool - would you want to tackle adding it into what I have, or would you like me to get it started first?

[octylFractal]

I think I only have time to test right now, so you should probably get started.

[Arcnor]

@dhild if @kenzierocks doesn't have time to test it, I can also do it after you've submitted what you need to submit.

Awesome that you got this to work. I stopped at the native libraries compilation for lack of time.

[dhild]

@kenzierocks @Arcnor Sounds great. I've been doing a lot of gradle configuration at work lately so it flows pretty quickly for me. From the gradle docs, not too sure how the Mac config even works; it looks like it uses the Clang configurations, so it could be pretty easy after getting the linux build down.

[dhild]

Okay - haven't tried it out on my linux environment yet, but I have an initial configuration for mac & linux uploaded. There was one hiccup in the mac setup, some sort of config relating to the sysroot setting and another related to the version in use. I left those out for the moment, so you might have to manually set them up. I'm betting that gradle does some of that configuration for us, but I'd have to try it out to find out....

Look for the TODO statement in the build.gradle file to see what I'm talking about.

[Arcnor]

I did ./gradlew build and that didn't go well:

/tmp/lwjgl3@gradle_project✗✗✗✗✗✗% ./gradlew build
:compileTemplatesKotlin
:compileTemplatesJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processTemplatesResources UP-TO-DATE
:templatesClasses
:compileKotlin
No Kotlin files found, skipping Kotlin compiler task
:generate
Exception in thread "main" java.lang.IllegalArgumentException: Invalid generation target path: /private/tmp/lwjgl3/generated
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4$main$validateDirectory$1.invoke(Generator.kt:37)
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4$main$validateDirectory$1.invoke(Generator.kt)
    at org.lwjgl.generator.GeneratorPackage$Generator$4582b1a4.main(Generator.kt:41)
    at org.lwjgl.generator.GeneratorPackage.main(Unknown Source)
:generate FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generate'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 36.119 secs

[octylFractal]

Your latest commit (878664b) should use it.options, not options.

Never mind, I don't know what's failing it.

[Arcnor]

@kenzierocks you can add comments directly to commit lines if you like 😄

[octylFractal]

I do know, it's just quicker for me to use what I've already got.

[dhild]

@kenzierocks Whoops. I've always just used the implicit reference to the delegate object though, mostly because I think it reads easier.

@Arcnor Okay, I'm guessing that the generate failed because that directory didn't exist. I added a quick fix to create the directories first. I'm a little nervous about that because of the symlinks in the original script (don't yet know about those), but it might fix it.

[Arcnor]

@dhild ok, that got me further. Now:

 warnings
:processResources UP-TO-DATE
:classes
:jar
:compileX64DebugLwjglSharedLibraryLwjglC
In file included from /private/tmp/lwjgl3/src/native/system/common_tools.c:5:
In file included from /private/tmp/lwjgl3/src/native/system/common_tools.h:26:
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/../include/jni.h:45:10: fatal error: 'jni_md.h' file not found
#include "jni_md.h"
         ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

jni_md.h is on /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/include/darwin/jni_md.h, not sure if you can automate that

[Arcnor]

Ok, commented too quickly. I edited build.gradle and added the include for /darwin, but now:

:compileX64DebugLwjglSharedLibraryLwjglC
/private/tmp/lwjgl3/src/native/system/org_lwjgl_system_MemoryUtil.c:24:19: fatal error: use of undeclared identifier 'intptr_t'
        return (jobject)(intptr_t)globalRef;
                         ^
1 error generated.
:compileX64DebugLwjglSharedLibraryLwjglC FAILED

[dhild]

@Arcnor That's weird. stdint.h should have intptr_t defined (or at least include its definition). Not sure where to go with that one.

https://github.com/LWJGL/lwjgl3/blob/master/src/native/system/macosx/MacOSXConfig.h#L7

[Arcnor]

If I add #include <stdint.h> at the top of the file, I get past that error, so it seems stdint.h is not being included...

[Spasi]

I will of course also consider changing the source layout, as required by Gradle.

[Arcnor]

@Spasi if you don't mind changing the layout, @dhild and me can take care of the whole building stuff & doing the porting if you want, even if you don't have the time to learn Gradle, and even for 3.0a.

[dhild]

@Spasi I completely understand limited time. Sounds like @Arcnor and I will keep working on the port in parallel for now. Thanks for the quick response!

[big-guy]

@dhild @Arcnor I'm one of the native support maintainers for Gradle, would you mind if I pitch in a bit?

[Arcnor]

@big-guy even if you were not somebody that useful (that's awesome btw 😄) no need to ask for permission, all the help we can get is welcome!

[big-guy]

Is https://github.com/dhild/lwjgl3/blob/gradle_project/build.gradle the latest?

[Arcnor]

Yes. Main problem was passing the parameters for OSX to run on the first thread, but fix what you think is wrong :P. Also, not sure if it works after the modules work @Spasi did, but @dhild knows more.

[Spasi]

I didn't want to waste any more time, so did the 3.0.0a release yesterday. Today I started working on the Gradle build, using @dhild's script as reference, but writing everything from scratch. I'd like to take the time to understand every little detail, to avoid wasting time later when I'll have to maintain it. Using Gradle 2.4-rc-1 for the parallel native compilation, LWJGL has tons of native files and I personally can't work without parallel builds.

From the reading I've done, it looks like the best approach would be to refactor the existing modules (templates, generated, core, util) as Gradle (sub)projects. Would that be a good idea?

Also, I'd like to keep the project root clean. All modules, except generated, are currently in /src. Would it be a problem if I made a /modules folder and put all sub-projects under there? Or do they really have to be directly under the root project?

[Arcnor]

That's awesome, congrats for the release.

As for your question, you can really put things wherever you want, so putting everything under modules will work perfectly fine. You will just need to reference it by doing modules/myModule and that's it. You can also have multiple build.gradle, so having one inside the modules directory will also be a good idea to keep everything more clean.

You can check any LibGDX project for a simple example if you want

[dhild]

@Spasi Very cool, release done!

The modules subdirectory sounds like a great idea.

There was one other idea I'd like to at least throw out for you to consider: splitting the modules along functionality lines, rather than by core, native, util. I tried this out, got it mostly working (just needed some more native build attention): https://github.com/dhild/lwjgl3/tree/source_reorganization

[Spasi]

I have just pushed the gradle branch. For now, the only changes are the new folder structure and the updated Ant scripts & IDEA project. Also clarified terminology a bit; the old "binding modules" are now just the "bindings" and "modules" are the compilation modules (as understood in Gradle or IDEA).

I would really appreciate any comments or suggestions on the new folder structure. It's a bit painful to refactor, so I'd like to make sure I get it right before pushing to master. If anyone has time to review the changes, please do let me know if you can think of any improvements.

I'll try to add the actual gradle scripts soon.

[huhlig]

If you are going to stick with Ant, why not just use ivy.

http://stackoverflow.com/questions/5111831/how-to-publish-3rdparty-artifacts-with-ivy-and-nexus

[upsonp]

I know this has been dead for quite some time other than tangentially mentioned from other topics, but...

Has anyone considered using Kotlin to create the Gradle build script. I was brushing up on my technologies because I'd like to make more use of LWJGL but there are a lot of things I don't understand and I get a lot of errors, which I don't know if they're me, my set up or the LWJGL itself. One of my biggest issues at the moment is just getting through the Ant build on my Windows machine. I had no issues on my personal Linux machine and had a few minutes today so I thought I'd check out my eclipse .classpath update on my work machine #373. Complete failure on the compile-templates over a Java Stack Heap issue.

I was doing a lot of reading into Gradle, which is what some people in my shop are using and I stumbled on Gradle with Kotlin. I know Kotlin is being used in LWJGL pretty heavily at the moment and at first I was like, "Oh great yet another language", but after spending some time watching conferences and tutorials on it, I'm quite excited. One of the conferences I watched was on how Gradle will be supporting Kotlin (that was in 2016, seems to be a done deal now). I pointed the Devs in my shop doing Gradle builds to it and they seem pretty interested in it as well.

Not promising I'm going to do it, mainly because I'm new to Kotlin and only vaguely acquainted with Gradle, but I might consider doing more research into and playing around more with it if I/it might be of use somewhere.

[Spasi]

Complete failure on the compile-templates over a Java Stack Heap issue.

I've only seen kotlinc failing when there is not enough memory available. Could you post more details about the error you're seeing?

Has anyone considered using Kotlin to create the Gradle build script.

I've been waiting for two features: a) incremental Kotlin compilation in Gradle builds and b) mature build.gradle.kts support. The former has been available for some time, the latter is getting there.

This remains a low priority issue though. The project won't gain anything from moving to Gradle and there are more pressing matters (e.g. ARM/Android support).

[upsonp]

From Running Ant Init

Buildfile: E:\_JBuilder\Personal\lwjgl3\build.xml
bindings:
init:
check-dependencies:
  [kotlinc] E:\_JBuilder\Personal\lwjgl3\bin\libs\kotlinc\build.txt doesn't exist
update-dependencies:
    [mkdir] Created dir: E:\_JBuilder\Personal\lwjgl3\bin\libs
    [mkdir] Created dir: E:\_JBuilder\Personal\lwjgl3\bin\libs\java
-lib-download:
-kotlinc-download:
  [kotlinc] Getting: https://github.com/JetBrains/kotlin/releases/download/v1.2.20/kotlin-compiler-1.2.20.zip
  [kotlinc] To: E:\_JBuilder\Personal\lwjgl3\bin\libs\kotlin-compiler-1.2.20.zip
  [kotlinc] https://github.com/JetBrains/kotlin/releases/download/v1.2.20/kotlin-compiler-1.2.20.zip moved to https://github-production-release-asset-2e65be.s3.amazonaws.com/3432266/83bc653a-fb8b-11e7-86d0-cb2dc1dc4b30?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180215T193417Z&X-Amz-Expires=300&X-Amz-Signature=3ca28f5affa86bd389a68657c87c1765b1adf410907e5277d30f1addddbcdf36&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkotlin-compiler-1.2.20.zip&response-content-type=application%2Foctet-stream
  [kotlinc] ....................................................
.
.
.
  [kotlinc] ..........................
  [kotlinc] Expanding: E:\_JBuilder\Personal\lwjgl3\bin\libs\kotlin-compiler-1.2.20.zip into E:\_JBuilder\Personal\lwjgl3\bin\libs
  [kotlinc] Deleting: E:\_JBuilder\Personal\lwjgl3\bin\libs\kotlin-compiler-1.2.20.zip
  [kotlinc] The Kotlin compiler was updated to build: 1.2.20
BUILD SUCCESSFUL
Total time: 1 minute 25 seconds

From running compile-templates

Buildfile: E:\_JBuilder\Personal\lwjgl3\build.xml
bindings:
init:
check-dependencies:
-compile-generator:
[Generator] Compiling Kotlin generator...
  [kotlinc] Compiling [E:\_JBuilder\Personal\lwjgl3\modules\generator\src\main\kotlin] => [E:\_JBuilder\Personal\lwjgl3\bin\classes\generator]
  [kotlinc] info: kotlinc-jvm 1.2.20 (JRE 1.8.0_111-b14)
  [kotlinc] info: PERF: INIT: Compiler initialized in 1145 ms
  [kotlinc] info: PERF: ANALYZE: 18 files (8561 lines) in 8258 ms - 1036.692 loc/s
  [kotlinc] info: PERF: GENERATE: 18 files (8561 lines) in 8143 ms - 1051.332 loc/s
  [kotlinc] info: PERF: GC time for Copy is 592 ms
  [kotlinc] info: PERF: GC time for MarkSweepCompact is 218 ms
  [kotlinc] info: PERF: JIT time is 2102 ms
  [kotlinc] info: PERF: Find Java class performed 17 times, total time 3 ms
  [kotlinc] info: PERF: Type info performed 28505 times, total time 5967 ms
  [kotlinc] info: PERF: Call resolve performed 16207 times, total time 4556 ms
  [kotlinc] info: PERF: Binary class from Kotlin file performed 263 times, total time 171 ms
    [touch] Creating E:\_JBuilder\Personal\lwjgl3\bin\classes\generator\touch.txt
[javac: Generator Tools & Doclets] Compiling 7 source files to E:\_JBuilder\Personal\lwjgl3\bin\classes\generator
compile-templates:
[Templates] Compiling Kotlin templates. This will take 1-2 minutes...
    [mkdir] Created dir: E:\_JBuilder\Personal\lwjgl3\bin\classes\templates
  [kotlinc] Compiling [E:\_JBuilder\Personal\lwjgl3\modules\lwjgl\core\src\main\kotlin\core\dyncall, ...] => [E:\_JBuilder\Personal\lwjgl3\bin\classes\templates]
  [kotlinc] info: kotlinc-jvm 1.2.20 (JRE 1.8.0_111-b14)
  [kotlinc] ERROR: Exception while analyzing expression at (371,11) in E:/_JBuilder/Personal/lwjgl3/modules/lwjgl/opengl/src/main/kotlin/opengl/templates/GL45.kt
.
.
.
  [kotlinc] Caused by: java.lang.OutOfMemoryError: Java heap space
.
.
.
  [kotlinc] Caused by: java.lang.OutOfMemoryError: Java heap space
.
.
.

BUILD FAILED
E:\_JBuilder\Personal\lwjgl3\build.xml:255: Compile failed; see the compiler error output for details.

Total time: 2 minutes 1 second

Edit: cut out the stack trace

[Spasi]

From running compile-templates
...
Caused by: java.lang.OutOfMemoryError: Java heap space

Try set ANT_OPTS=-Xmx1G before running ant. Or use a 64bit JVM.

[upsonp]

I gave up trying to run Ant from the command line.

1. I checked out the project fresh and started by copying the eclipse .classpath file to the root.
2. Added lwjgl3 as a project.
3. Then I created a "Run External Tool" under the "External Tools Configuration" dialog
   From there it was a piece of cake.
4. On the Targets tab I selected (Clean, init, compile-templates, compile-tests) and made sure that was the order they'd execute in.
5. On the JRE tab I added '-Xmx1G', at @Spasi suggestion.
6. Clicked Run and got a full build, no issues.
7. I updated the Class path for the tools.jar file
8. Clicked on the project and hit F5 for a refresh. (Then waited for a workspace rebuild)

It looks like it completed, but I'm missing the lwjgl.dll file though. I assume that was suppose to go in the bin/libs folder the same as the Linux *.so files.

Edit: Should also mention I did set ANT_OPTS and switched my default JDK to the 64-bit 1.8.0_161 install before trying the eclipse ant interface.

I got a bunch of errors like these

Buildfile: E:\_JBuilder\Personal\lwjgl3\build.xml

bindings:

init:

check-dependencies:

-compile-generator:
[javac: Generator Tools & Doclets] Compiling 7 source files to E:\_JBuilder\Personal\lwjgl3\bin\classes\generator
[javac: Generator Tools & Doclets] E:\_JBuilder\Personal\lwjgl3\modules\generator\src\main\java\org\lwjgl\system\ExcludeDoclet.java:7: error: package com.sun.javadoc does not exist
[javac: Generator Tools & Doclets] import com.sun.javadoc.*;
[javac: Generator Tools & Doclets] ^

Edit 2: I started looking at issue #349, just because, and I now realize the errors being generated on my Windows machine from the command line may be related to the deprecation of com.sun.javadoc.* Which if I'd actually read the error I might have made the connection sooner. No idea why I'd get them from the command line and not the Eclipse Ant plug-in. I made sure both were using the same JVM and Ant version.

[Spasi]

I got a bunch of errors like these

Is the JAVA_HOME environment variable set to the JDK 8 folder?

but I'm missing the lwjgl.dll file

Native compilation happens with ant compile-native or (demo or tests that depend on it). You'll need Visual Studio Community and a properly configured toolchain (usually running vcvarsall amd64_x86 is all you need).

[upsonp]

JAVA_HOME is set, I think it might be pointing to an Oracle DB or ArcGIS or [something else] install of the JDK though.

My work machine is sometimes locked down in terms of what I can install (depends on what my group policy permissions has decided is ok that day) and I don't have access to another windows box to develop on. So I'll have to see if I have permissions to install Visual Studio.

I appreciate the help, sometimes it takes me awhile to get familiarized with project and it's tools, but if there's some way I can give back, I'd like to help out.

[Spasi]

Gradle 5 has been released and the Kotlin DSL is now considered production-ready. The LWJGL codebase structure has also stabilized, so now is a good time to attempt a migration to Gradle.

[TheMrMilchmann]

I have already experimented with porting the build process to Gradle in the past and had a look at it again quite recently. I might publish my progress later this week to give you an idea of what it could look like.

However, one issue that's stopping me from cleanly transitioning to Gradle remains: There are circular dependencies between templates of some modules (e.g. OpenGL and OpenCL). While it is possible to compile all templates together this eliminates some of Gradle's benefits and I'd rather see those circular dependencies removed (if this is reasonable without too much duplication).

[Spasi]

There are circular dependencies between templates of some modules (e.g. OpenGL and OpenCL)

The cycle is now broken with 7657b20. There are still one-way dependencies between modules (e.g. GLFW depends on the EGL, GL and Vulkan modules), but that shouldn't be a problem, right?

[elect86]

So, I picked up this (again) a couple of weeks ago and ported the most important ant task to gradle tasks. I didn't use Leon's work because of outdate (also Gradle gets developed super fast), but I took inspiration nonetheless.

The current work is here and right now only Linux is considered somehow complete. I'm working on Windows at the moment, but for MacOS I'd need someone helps..

Features:

• Tasks are fully integrated in the Gradle model, with outputs and inputs
• Is there additionally a way to extend this optimization (skipping on specific conditions) even further but has to be polished
• mutable states are shared via Build Service
• DSL for compiling and interacting with java, javac and kotlinc and the native build process

There are still tons of stuff to polish and tune (move to multithreading/parallel execution/coroutines), but I'd like to start gathering whatever feedback/hint/ideas I can, since there is a lot of time ahead yet

Ps: I guess the best would have been something more "Gradlesque" way, by moving all the templates in a module, the generator as well and having Gradle build them while setting up all the proper dependencies, but Spasi is used to have all the relevant code in one close space (c, java and generated) and so he would like to continue. I'm not sure if that design would have worked with the Spasi wish, but that's something to play with for the next time :)

[elect86]

I pushed another commit, the whole linux workflow can be considered complete now