Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a GUI solution to LWJGL #101

Open
Spasi opened this Issue Oct 8, 2015 · 65 comments

Comments

Projects
None yet
@Spasi
Copy link
Member

Spasi commented Oct 8, 2015

This is meant to be a meta-issue in which discussion can take place about potential solutions to GUI layout & rendering. Currently LWJGL only supports text rendering (via stb) and may soon support antialiased 2D graphics (via nanovg).

Topics to discuss:

  • Rendering GUI components on top of 3D content (e.g. game menus, HUDs, etc).
  • Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).
  • Embedding 3D content in a standard application/tool. The 3D content should be lightweight, i.e. standard menus should be renderable on top of it.

The ideal solution should be hardware-accelerated, i.e. there shouldn't be any framebuffer copies from the CPU to the GPU or vice versa.

Approaches that have been explored already:

  • JavaFX integration via PixelWriter & PixelReader. See the LWJGL-FX project (uses LWJGL 2 but the same issues apply to LWJGL 3). Not hardware-accelerated.
  • Adding bindings to CEF and using the offscreen rendering API. Not hardware-accelerated.
@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 8, 2015

Also I have a running version of Themable Widget Library, which works really awesome with LWJGL 3 and kappa's lwjglx library.
I would really love for someone to do the porting to native LWJGL3 without needing an adaptation layer.
The render code of TWL is really nicely decoupled from everything else and can be swapped once a new LWJGL3 implementation is available.

@kappaOne

This comment has been minimized.

Copy link
Member

kappaOne commented Oct 8, 2015

Do agree that TWL is a great option for rendering GUI on top of 3d content. It only supports LWJGL2 atm, though looking at the code base it doesn't seem all that much work to port to LWJGL3.

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Interestingly, l33tlabs (who host TWL), work on a really interesting project called Bling. They've somehow integrated SWT with TWL! I think they also use LWJGL2. They already have a full IDE running on the same, so pretty far along. There is a pretty long video here explaining how it works and theres a quick preview of what it looks like here.

I'm not sure what l33tlabs plans are but if they released the underlying SWT/TWL library or if something similar could be written, it could probably be the ultimate and complete solution for LWJGL3 i.e. a small lightweight API that can render ui inside OpenGL applications, create native platform widgets (like dialog's, system tray integration, native controls, etc) or even enable full blown native applications that can render opengl content inside it (e.g. particle editors, game engine editors, etc).

@renannprado

This comment has been minimized.

Copy link

renannprado commented Oct 8, 2015

I wonder if it would be possible to somehow render HTML+CSS (and possible code the UI logic in javascript via nashorn maybe) for HUD?
I've researched quite a bit some time ago but couldn't find anything like that or even a reference.
It would also be nice to check how games such as Heroes of Storm and Starcraft 2 (which, IMO, have nice HUD) are doing their HUD.

Another thing that maybe could help is the project dukescript.

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 8, 2015

I think using Nashorn and JavaScript would not be the problem; the library implementation and its API would be, as it sounds rather like the tasks of browser engines like WebKit or Blink and the domain of the Web platform in general. However it would sure be great. :)
IMHO when layouting with HTML/CSS, I would then switch to WebGL (like with three.js), Canvas "2d" based 3D frameworks such as phoria.js or CSS 3D Transformations, instead of developing a desktop application with Java/LWJGL.
dukescript however sounds fascinating! :)

@renannprado

This comment has been minimized.

Copy link

renannprado commented Oct 8, 2015

Yeah, that's the thing. I didn't mean to create something like WebKit, but if we could create a binding for it and use it to somehow render the HUD, that is what, IMO, would be awesome.
(I don't have any clue if it's possible, maybe I'm saying some non-sense here. I'm just trying to think from the library end user POV and, for me, HTML+CSS (and possible javascript) is the most flexible way to write UI today)

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 9, 2015

This "awesomium" seems to be what you were looking for:

It also allows to render into an OpenGL surface (one can define its own surface factory).
That could be an option.

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Oct 10, 2015

In my humble opinion its more of an issue if a GUI lib doesn't supports core OpenGL than not supporting LWJGL 3 (TWL is really nice but it's renderer is GL 1.1 exclusively).

NiftyGUI is there, has a core OpenGL backend, but doesn't supports LWJGL 3 (mainly Display/Keyboard/Mouse related changes and some function name changes).

Also another concern of mine is the state in which the GUI lib leaves OpenGL. This should be very well defined, ie, what state OpenGL needs to be before the GUI lib can do whatever it needs to do, and what state is OpenGL left with once it finishes. This is vital for efficient state tracking.

@JustGregory-zz

This comment has been minimized.

Copy link

JustGregory-zz commented Oct 10, 2015

Unlike @dustContributor I'm not concerned with supporting "core OpenGL" standing as much as if it fits in the "lwjgl wheelhouse" of libraries and functionality, and if the general API fits thematically with other, existing lwjgl API calls. I don't have a specific recommendation on how to do that, so long as I would be able to pass in some UI component hierarchy, and just tell it to "show the gui".

What would be wrong with basically "copying" the AWT and making it render for lwjgl?

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 10, 2015

Supporting OpenGL 1.1 and not >= 3.2 core is an issue if you yourself want to use a "modern" OpenGL context and functions therein, which would both not be possible in a single context. Either TWL would not function correctly because it used deprecated OpenGL functions or you would not be able to use "new" OpenGL functions. This is an issue with Mac OS X, which either supports OpenGL <= 2.1 if you don't ask for a new context specifically, or you would only get a "core" context if specifically asked for, but then TWL would not work.

One solution to this problem, and also to the problem of touched OpenGL state, could be multiple sharing contexts. One "deprecated" OpenGL context for TWL and another one for your own rendering.
TWL could be issued to render into a texture using a Framebuffer Object and then that texture can be shared between both contexts to simply blit the texture onto your own "new" context's window.

Sharing between an "old" deprecated context and a "new" core/forward compatible context does work at least under Windows with Nvidia.

However, only "requiring" OpenGL 1.1 is the smartest thing one can do for such a widespread library like TWL, in my opinion. Whenever you require anything higher than OpenGL 2.0 it will not work on some person's machine. Compatibility is a very big thing here!

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Oct 10, 2015

That sounds both slow and bound to hit driver specific issues. No, I'd rather handle the state change myself.

At the very least it should have a renderer interface that doesn't assumes fixed-function pipeline working underneath, TWL has that issue in various places last time I checked, which makes it a pain in the butt to code a core renderer for.

I still propose NiftyGUI as a better alternative, right now it just needs a couple of classes using a LWJGL 3 backend. I opened an issue about it a while ago nifty-gui/nifty-gui#304

There is literally no other GUI library that has a core renderer that I know of (for Java that is), thus why I think its an important matter. OpenGL 3.0 has been out there for 7 years, with hardware supporting it going as back as 2006 (GeForce 8xxx, Radeon HD2xxx), that was almost ten years ago.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Oct 18, 2015

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Do we even need anything special for SWT integration? Looks like @httpdigest already has a sample here.

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 18, 2015

The only downside of SWT is that it still (even in the very latest nightly build) cannot produce a GL >=3.2 core context. I had a look at the sources and found that the WGL solution at least is devised but commented out... But this problem can be solved by letting LWJGL3 do the GL context creation. So we would need a cross-platform way to create GL contexts in LWJGL3 based on a window handle. This handle is easily API-accessible in SWT.
Another solution would of course be to hack SWT, but I heard that their OS X backend is really heavily outdated, so it would be quite some hacking.
I also dug out an old LWJGL forum entry about this: http://forum.lwjgl.org/index.php?topic=5531.0

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Oct 18, 2015

I recall digging through the sources and finding no clear way to hijack the SWT's component so I could use it with my core context (although my experience with window management APIs is limited).

Both JavaFX and SWT (since 4.2 I believe) are theme-able with CSS, which is a pretty cool thing to have.

I wonder if we could nag SWT developers about this...

EDIT: Still no interest in NiftyGUI?

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Oct 19, 2015

I am going to build an own GLCanvas, something like a org.lwjgl.opengl.swt.GLCanvas or org.lwjgl.swt.opengl.GLCanvas, based on the platform API exposed by LWJGL3. The native/JNI code of SWT is really horrible. They do like a thousand (*env)->GetField(...) invocations and other slow/nasty hacks.
Personally I think, an own custom SWT Canvas would be the best solution, and I don't think it will require much maintenance, since it is only about creating the GL context and that should be common knowledge how to do that.

@crisp00

This comment has been minimized.

Copy link

crisp00 commented Nov 23, 2015

So any update on lwjgl native gui rendering methods?
Any news on nanovg implementation?
Please keep us updated!

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Nov 23, 2015

@httpdigest is making good progress with SWT integration, see his latest update.

I experimented a bit more with JavaFX and have a working OpenGL backend that runs on top of LWJGL. This may seem like a huge step forward, but it's not enough. It would be easy to implement context sharing with the JavaFX context and access the textures directly. The problem is, JavaFX is architected in a such way that the rendering layer has no idea which textures correspond to which GUI elements. I don't have a solution to this yet, I'll explore some more ideas when I have time.

I don't know if nanovg bindings will make it to the final LWJGL 3.0 release, but I'll try.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Jan 10, 2016

The latest nightly build has jawt bindings, which in theory allows for the AWT integration in LWJGL 2 to be implemented in terms of LWJGL 3. Sample code can be found here (simplest possible implementation, Windows-only).

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Jan 11, 2016

I've been thinking. Now that NanoVG bindings are up, it might be possible to use it as backend for other GUI libs that lack an OpenGL core context, like TWL for example.

I'm looking at TWL's sources, seems a lot of work though -_-

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Jan 15, 2016

I completely forgot about libRocket: http://librocket.com/

Its C++ though.

@ShadowLordAlpha

This comment has been minimized.

Copy link

ShadowLordAlpha commented Mar 7, 2016

This is kind of late but a while ago I did manage to make TWL run with LWJGL 3 though it wasn't good by any means and TWL does use old OpenGL functions.

I should also bring up that one of the main problems I was having was figuring out a good way to get the event data from GLFW and processes it. As there can only be one listener it would be easy to make a version that just splits that up however it gets more complicated for the end user and taking over isn't a very good option as its not very easy to detect if your listener has been removed and could lead to problems with the end users code.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Apr 18, 2016

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Apr 20, 2016

Sweet, it even leaves rendering up to the user by just giving you commands to execute, perfect.

@AdMaioremDeiGloriam

This comment has been minimized.

Copy link

AdMaioremDeiGloriam commented May 5, 2016

Hey, if at all interested, I'm making a library for lwjgl, but for a broader spectrum*, from scratch. It'll allow direct support for GLFW in LWJGL3, support for the latest OpenGL (4.5+), and a way to build shaders on the fly during runtime (in GLSL). The window component system will be made to support XHTML and XML based layouts for quick building of GUIs and easy management, so you could simply build it in a browser or other development environment.

(I don't have a repository yet, and I have a(n empty) repositoryI have a repository, but don't have internet. ~~ but I'll get one up as soon as possible. It's pretty bare bones right now anyways...)~~

*By which I mean it's for more than just lwjgl. It'll support GLFW, but it'll support other GUI systems as well: AWT, SWT, Swing, etc.

@ShchAlexander

This comment has been minimized.

Copy link

ShchAlexander commented May 13, 2016

Hey! It would be great if you add bindings to this library NanoGUI
Can you please take a look on it? :)

@arisona

This comment has been minimized.

Copy link

arisona commented May 13, 2016

NanoGUI is C++ and probably hard to wrap. But it could make sense to port NanoGUI to Java on top of lwjgl's NanoVG.
Edit: NanoGUI is quite decent, btw. IMHO would be a good choice.

@AdMaioremDeiGloriam

This comment has been minimized.

Copy link

AdMaioremDeiGloriam commented May 24, 2016

SGL now has code in it

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Jun 1, 2016

https://github.com/glegris/nuklear4j

Dafuq, never saw what O_o

EDIT: Oh, SWIG : /

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Jun 1, 2016

LWJGL will very soon have Nuklear bindings, see the nuklear branch.

There's already support for everything, but I'm working on a Nuklear+GLFW+stb_truetype demo to understand and properly annotate the API (nullability, autosize parameters, etc).

@J-Ibarra

This comment has been minimized.

Copy link

J-Ibarra commented Jun 4, 2016

i have a problem is the code of the demos,

import static org.lwjgl.nuklear.Nuklear.*;

in the version of lwjgl 3.0.0 build 90 no have the package

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Jun 4, 2016

Like @Spasi said:

The first 3.0.1 builds will have Nuklear built-in

You are using the 3.0.0 Release, which does not have Nuklear included.
The sources of the Nuklear module, as well as the demo you are referring to, live in the nuklear branch, and you could build LWJGL with Nuklear from that branch manually.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Jun 5, 2016

The first LWJGL 3.0.1 nightly build is now available and it includes the Nuklear bindings.

@pierr3

This comment has been minimized.

Copy link
Contributor

pierr3 commented Jun 5, 2016

Great job, thanks!

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Jun 9, 2016

Awesome man, thanks a lot for your work!

@jediminer543

This comment has been minimized.

Copy link

jediminer543 commented Nov 21, 2016

Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).

Is this something that can be done with nuklear?

@sriharshachilakapati

This comment has been minimized.

Copy link
Contributor

sriharshachilakapati commented Nov 21, 2016

@jediminer543

You can render the GUI to a texture using a framebuffer and then use it to render the monitor panel triangles in the 3D space. I never worked with Nuklear, but this is how you will be doing normally.

@codemuse

This comment has been minimized.

Copy link

codemuse commented Dec 7, 2016

Are there still plans to further explore JavaFX integration "natively" in lwjgl3, or is LWJGL-FX the current best alternative?

Most discussion in this ticket is about adding UI in a native GL display, what I have is an SWT based GUI that embeds lwjgl through an SWT GLCanvas to display a 3D representation that can be inspected using the SWT UI components around it, but we are looking at rewriting the GUI with JavaFX.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Dec 8, 2016

Hey @codemuse,

Have you tried JavaFX Interoperability with SWT?

As for a pure JavaFX solution, I will probably try again when I have time, but failure is the most likely outcome.

@kappaOne

This comment has been minimized.

Copy link
Member

kappaOne commented Dec 8, 2016

IMO any proper solution is really going to need help from the JavaFX team at Oracle, such as them adding an API for this sort of stuff or for them to expose certain internals of JavaFX which we can use to implement it properly.

Without the above, any solution is just going to be a hack, likely non-optimal and subject to breakage. JavaFX at present just isn't designed to allow this sort of integration.

@kappaOne

This comment has been minimized.

Copy link
Member

kappaOne commented Dec 9, 2016

Interestingly, planning for JavaFX.next (Java 10+) is currently taking place over at http://mail.openjdk.java.net/pipermail/openjfx-dev/2016-December/020025.html and Oracle are asking for suggestions.

Might be a good opportunity to chime in and request support for OpenGL/Vulkan via LWJGL.

@httpdigest

This comment has been minimized.

Copy link
Member

httpdigest commented Dec 9, 2016

It seems that has already been requested:

Also for JDK8:

But I fear, because one of JavaFX's design goals being "abstract all and everything away" so that JavaFX is able to run on phones, on the Desktop, on the Browser platform and everywhere while being accelerated by either D3D, OpenGL or Vulkan (or what happens to be the hardware acceleration on that particular device), that Oracle will not break with this abstraction to actually expose an "OpenGLNode" (or even just a proposed "3D Canvas Node" telling the client that everything is currently being accelerated by D3D, OpenGL or Vulkan) on the API layer.

@codemuse

This comment has been minimized.

Copy link

codemuse commented Dec 12, 2016

@Spasi, @kappaOne and @httpdigest: thanks for all your quick replies!

Your comments are similar to what I could find myself as I was investigating this. I'm looking for a "pure" JavaFX solution, i.e. embedding it in SWT wouldn't really make sense, since the idea is to move our application away from SWT.

@httpdigest thanks for the pointers to the OpenJDK mailinglist, I'll follow that to see what comes out of it.

In the meantime, I'll toy around what sort of hack would -- although suboptimal -- work for our specific situation.

@sriharshachilakapati

This comment has been minimized.

Copy link
Contributor

sriharshachilakapati commented Dec 12, 2016

Now that we have JAWT bindings, can't we have it embedded into Swing and embed it into JavaFX using SwingNode?

@JustGregory-zz

This comment has been minimized.

Copy link

JustGregory-zz commented Dec 13, 2016

I will second what sriharsha said above, except I fear that Swing/JavaFX/SwingNode adds altogether more of the abstraction-complexity to the mix. If it could be done easily, it would have been already; but maybe it will take a complex implementation to simplify it somewhat.

@ShchAlexander

This comment has been minimized.

Copy link

ShchAlexander commented Jan 22, 2017

Hi all, currently I working on GUI oop solution with api little bit similar to swing...
Here is my repository: legui

Here is result:

Radio button and text input Widgets TextArea with horizontal and vertical alignment
Button, togglebutton, checkbox, selectbox Sliders ScrollPanel
@Barteks2x

This comment has been minimized.

Copy link

Barteks2x commented Mar 17, 2017

Someone actually made JavaFX work with lwjgl3, but it seems like it doesn't really use opengl core profile and it uses opengl 3.2 core profile: https://bitbucket.org/cuchaz/jfxgl

As it says in readme, it's a giant hack but it works and has good performance.

@ShchAlexander

This comment has been minimized.

Copy link

ShchAlexander commented Mar 18, 2017

@Barteks2x it looks like it's hard to make it work with Oracle java as standalone library, am I right?

@Barteks2x

This comment has been minimized.

Copy link

Barteks2x commented Mar 18, 2017

@ShchAlexander I'm not the author so beyond what it says in the readme and what I see I can't tell you much. Based on my testing, it works as standalone library, but it uses implementation-specific hacks, including modifying the bytecode using asm library.

Obviously when using JFXGL you can't use normal JavaFX without lwjgl at the same time (and why would you?)

@cuchaz

This comment has been minimized.

Copy link

cuchaz commented Mar 20, 2017

Hi There. I'm the JFXGL author. It's still very much a work in progress, but it's getting better all the time. I've been talking to Spasi about it too, so it's already on the radar.

If you have any questions about JFXGL, ask away. Although maybe don't spam this issue thread?

@J-Ibarra

This comment has been minimized.

Copy link

J-Ibarra commented Mar 28, 2017

Doing some research I got with this post!

https://engineering.riotgames.com/news/under-hood-league-client%E2%80%99s-hextech-ui

Here mention the use of javascript to create the client of this game, it would be interesting to be able to create GUI with HTML

OpenGL GUI using Chromium Embedded Framework (CEF)

@jjYBdx4IL

This comment has been minimized.

Copy link

jjYBdx4IL commented Jun 18, 2017

SWT uses native OS widgets, why would one want to use that?

Did anyone evaluate whether it is possible to extend Graphics2D class/override repaint() methods and thereby redirect Java Swing UI painting to OpenGL/NanoVG?

Besides that, one could set JFrames as transparent and place them on top of the OpenGL window... :)

However, the best solution would be some generic Java UI lib that can be plugged into NanoVG, ie. a rewrite of Nuklear. CSSBOX provides html/css rendering, though I'm not sure how efficient animations would be. And interactions? In general, I'm not sure how efficient GUI animations would be with pre-written GUI libs. It would be an interesting task to think about the required structure of such a library to efficiently support all sorts of GUI animations.

BTW: because Vulkan support is now quite widespread, is NanoVG compatible with it?

@elect86

This comment has been minimized.

Copy link

elect86 commented Nov 21, 2017

Hi guys,

I'd like to propose also another possibility..

I'm doing a full jvm port of imgui here (nuklear is based on its concept), it's almost complete, text input/handling needs still to be addressed but all the rest is done

plus, it's also libgdx compatible :)

See the Screenshots Thread for some user creations. Also see the Mega screenshots for an idea of the available features.

Sample

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Aug 21, 2018

I was reading some Java articles and it popped up that Gluon published release/support plans for OpenJFX. For those who don't know, JavaFX is being removed from core JDK, and it will be released separately. So to use it you'd need it to import it like any other dependency (maven, raw jars, etc). And now it will be called OpenJFX.

https://gluonhq.com/javafx-11-release-and-support-plans/

They have a GitHub mirror of the OpenJFX repository and they say they pay attention to the issues there. https://github.com/javafxports/openjdk-jfx

Moreover, they say they have their bug tracking on the JDK thingy (although the site works like ass for me and I cant find it) https://bugs.openjdk.java.net/secure/Dashboard.jspa

Anyway, I'm wondering if that given the changes JFX is going through, if it would be possible to push for direct access to the GL internals from it? As a means to integration with other GL rendering.

All the stack is open source now so you got in that repo their "Prism" rendering engine and all, plus the HLSL/GLSL shader transpiler (Decora I think?) and some other nifty things.

In any case, they went a different route for it so instead of making bindings and "lower" the layer of native interop, they instead "raised" it by making the rendering part in C/C++ (doing all their multi platform context management on their own and all) and exposing higher level operations to Java. That doesn't sounds like it'd interact well with how LWJGL works but I wouldn't know.

It gets complicated because OpenJFX uses three backends, OpenGL ES 2, Direct3D and software. So what happens if you want to use OpenGL Core? Or Vulkan?

It's pretty complex, and I personally believe that The One True Way™ to do an UI lib for games is doing it like Nuklear does, handing out buffers of vertices/uvs/indices at rendering stage and have the user be responsible of displaying that to the screen, but I thought someone might be interested.

@Spasi

This comment has been minimized.

Copy link
Member Author

Spasi commented Aug 22, 2018

@dustContributor Yes, OpenJFX being decoupled from the JDK opens up opportunities, you can read some details in this JGO post.

(No work has been done yet, I grossly overestimated the free time I'd have this summer)

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Aug 26, 2018

I see you know already how to tackle it.

I inspected a bit OpenJFX code and it seems to have a rather direct rendering, in the sense that it has fairly hard codes operations in a bunch of classes that do specific things.

I'm guessing that achieving Nuklear-style of integration (ie, making it spit out vertices and texture bindings, API agnostic) would be pretty hard.

@ShchAlexander

This comment has been minimized.

Copy link

ShchAlexander commented Aug 29, 2018

Hello everyone again.
I have several changes since last comment here.
Here is image of my gui library's current state:

preview

If anyone want to contribute, request any changes - you are welcome.
GitHub repo - https://github.com/LiquidEngine/legui/

@dustContributor

This comment has been minimized.

Copy link

dustContributor commented Sep 8, 2018

If anyone want to contribute, request any changes - you are welcome.

I'll repeat what I said above (except longer!). I dont think having your own rendering backends is a good idea. You don't know how I, library user, do my rendering, you don't know what API I run, you don't know what version I run, and you really shouldn't.

Things that did their own drawing like Scaleform on big AAA games were phased out, because you have a complete subsystem trying its best to group rendering commands, group state and shader changes, and drawing all of that efficiently, and then you stuff your code with some other renderer that has some other API with some other concepts that just doesn't fits on how your engine does its drawing.

You have libraries like NanoVG, Nuklear and (now defunct) libRocket that are really really easy to integrate because they have a clear distinction between what is UI handling, what is rendering and what is input handling.

Without the library setup code, this is my rendering loop for Nuklear:

try ( MemoryStack mem = MemoryStack.stackPush() ) {
	// setup buffers to load vertices and elements
	final NkBuffer vbuf = NkBuffer.mallocStack( mem );
	final NkBuffer ebuf = NkBuffer.mallocStack( mem );
	nk_buffer_init_fixed( vbuf, tmpVertices/* , max_vertex_buffer */ );
	nk_buffer_init_fixed( ebuf, tmpIndices/* , max_element_buffer */ );
        // converts buffers to the layout specified in the config
	nk_convert( context, commands, vbuf, ebuf, bufferLayoutConfig );
}
// Upload data to the GPU
glAttribBuffer.bind().updateUnsafe( tmpVertices );
glIndicesBuffer.bind().updateUnsafe( tmpIndices );
// Process task commands.
int offset = 0;
// Compose render tasks out of nuklear's draw commands.
for ( NkDrawCommand cmd = nk__draw_begin( context, commands ); cmd != null; cmd = nk__draw_next( cmd, commands,
		context ) ) {
	final int elmCount = cmd.elem_count();
	if ( elmCount < 1 ) {
		continue;
	}
	// Submit task to the queue.
	// Matrices computed in shader.
	renderer.addToQueue( taskBuilder.reset()
			.type( RenderTask.Type.NUKLEAR )
			.meshId( glVao.id )
			.material( materialsById.get( cmd.texture().id() ) )
			.vertexCount( elmCount )
			.indicesOffset( offset )
			.build() );
	offset += elmCount * 2;
}

I just ask for Nuklear to fill vertex/index buffers, upload them to the GPU, then compose render tasks that use them and that's it.

I know that will best fit my rendering scheme because it is my rendering scheme. It will use the API I use, it will use the version I use, it will track state changes like I do, it will batch UBO updates like I do, and run without switching shaders unnecessarily like I do, and if I wanted I could make it use my instanced rendering by grouping by material first.

I dropped things like NiftyGUI because literally the first time I use them it turns out that there was some GL context interaction that they didn't account for so I had to spend time debugging what state my renderer was leaving GL with, what state the UI lib really needed (that may or may not be what the documentation says) and what state the GL context ended being in that made things not work correctly.

The GPU is a big fucking thing, wouldn't it be nice if you (lib author) didn't have to account for these things? Wouldn't it be a concern better suited for myself the library user since I know what API I target and how I want to use it?

I'd say: Write an UI lib, or a renderer, but not both in the same thing. They're separate concerns, and big enough to warrant their own separate libraries.

The same argument can be made for input handling but that'd make this too long. I have an input system that translates GLFW's stuff for my own scheme, and those events get pushed to Nuklear, Nuklear doesn't cares how do I map my keys, devices or anything, it just cares for being fed input events in a format it understands.

At least that's my POV after trying out integrating JavaFX, Swing, TWL and NiftyGUI multiple times over time. Nuklear, being the pain in the ass of a library it is to use, was way, way easier to integrate. And since the API is so clean cut on that regard, it was easier to debug. I know when it renders because I render it. I know when it processes input because I tell it to.

That said, I know UI is hard, that's why I'm always in the lookout for a lib that does it for me. I understand making such a simple integration point isn't easy, and may require rethinking a lot of abstractions. For example, it's really hard to move TWL as-is to such scheme, or even to using a core GL context (nevermind Vulkan) with it because not only believes it should draw itself, it believes it should draw itself like they did with OpenGL 1.1 in 1998, which for any kind of recent project is a problem. The right abstractions just aren't there.

Moreover, I will argue that moving to such scheme not only opens up the integration possibilities, it has no drawbacks. Nothing prevents you from implementing a GL 1.1 renderer for Nuklear, or a Vulkan one, or a software renderer, nothing prevents you from supplying separately your own backends to draw it if you really really want to code an UI renderer. Hell, it wouldn't surprise me if data-driving it that way made the whole thing faster via handling flat buffers of draw commands and vertices. It just makes the whole ordeal way easier for everyone involved.

I'm not writing this for you specifically anyway, so dont take it as a big ass critique of your code, I've only skimmed through your library and it probably does a lot of things right that aren't even on my mind right now. I'm just writing it for anyone who is interested in doing/using an UI library in Java for videogame (or any real-time graphics) purposes.

@ShchAlexander

This comment has been minimized.

Copy link

ShchAlexander commented Sep 9, 2018

I understand what you mean, @dustContributor, and of course big projects often use their own decisions that fit better to the project and work faster.

I'm not saying that my decision is good, but I tried to make it as much extensible as it could be.
Renderers could be implemented using OpenGL, Vulkan or and any other technology.

Any way every solution is good for it's purposes. If you need best performance - you implement something in your own way, if you need fast development without pain - you can choose existing solution. 👌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.