Emscripten building issue

[a2kas]

Anyone can say whats the problem is here?

[JakeTrock]

Seems inolen has left this place to rot.. better off trying to recreate the tutorial he originally created here:
https://web.archive.org/web/20140301184841/http://www.inolen.com:80/articles/compiling-quake-3-virtual-machines-to-javascript/

I'd be interested in grouping a few people together to make a functioning fork. just respond asking for my discord handle, and we can take this farther.

[inolen]

I wouldn't say "left this place to rot." The issue is, the project was originally compiled with what is now a very old version of Emscripten (before it had an LLVM backend) and no one has made the necessary changes to get it compiling on a modern version of Emscripten.

If you'd like to make those changes, by all means I'm here to accept the pull request.

[JakeTrock]

I'm sorry to be harsh, I was just a bit frusturated with myself that I couldnt fet this to work after trying for nearly half a year. I think it would beneficial to replicate your original dev environment, as I'm not sure what should be changed to make this llvm compatible, and I could release a compressed virtualbox file for anyone who wants to make

[inolen]

There's no value in trying to compile this with a 5 year old version of Emscripten - it should just be upgraded to work with current day Emscripten.

[JakeTrock]

alright, What can I do, what is to be revised(the makefile?)?

[JakeTrock]

Also, if making is done to better target a CPU, why does the js need to be recompiled? shouldn't it just work out of the box, one size fits all?

P.S. Just was looking at your blog for more context as to how the qvm works and saw your MobyGames profile, props to you, you made one of my favorite games ever(gh3) thank you so much!

[inolen]

These are great questions to dig into more and learn about it you want to get this updated.

I'd suggest working with Emscripten on some smaller projects if you want to understand this better.

[JakeTrock]

I'm going to try to bring over some old demoscene stuff made by a friend (nothing too complex), what changes typically have to be made before export
(thanks for the patience btw, I'm only 16 and it means a lot)

[a2kas]

@hokuco can you give your contact? I will help help you with this

[NTT123]

I've managed to successfully compile with the latest emscripten. The hack is to hardcode the offsets of structures instead of using Runtime.genStructureInfo which is no longer supported.

However, it currently has performance problems and noticeably slower than the version at quakejs.com. Checkout the project at https://github.com/NTT123/quakejs

[inolen]

@NTT123 fantastic!

I'd love to update the website with a modern version, and have no problem merging your changes if you want to keep them rolling.

The performance problems however are kind of a bummer - any idea where the major regression is?

[a2kas]

really nice, I have been trying to build it over 2 months :D

[NTT123]

@inolen, openal function calls somehow runs slower than the old version.

Unfortunately, when I replace openal library with an older version from emscripten git commits, the whole game still consumes more %CPU than the older version running at quakejs.com, especially when running on complex maps.

[ghost]

I have this building against the latest ioq3, I've repatched all the emscripten specific stuff.
I need to go back through and add #ifdef EMSCRIPTEN.

https://github.com/megamindbrian/planet-quake/commits/repairing-quakejs-changes

I am stuck on loading .map files for the menu system. ui.qvm and ui.map. I don't know why these are excluded in the asset graph and I don't totally understand the asset graph.

I might try to use just the ioq3 functions that load assets in game, and try to make a graph that downloads individual files out of the zip on a server. I.e. start read stream and use the header to only grab the requested file.

Here's my latest console, I've added messages to debug the most minuscule changes it doesn't look like these other patches have, for example in tr_image, additional calls to TextureImage2D was causing errors because of the same RGBA8 -> RGBA change in the last branch from inolen in Feb or 2014.

ioquake3.js:3006 Sound initialization successful.
ioquake3.js:3006 --------------------------------
ioquake3.js:3006 Loading file /base/baseq3/vm/ui.qvm
ioquake3.js:3006 Loading vm file vm/ui.qvm...
ioquake3.js:3006 FS_FOpenFileRead: vm/ui.qvm (found in '/base/baseq3/pak8.pk3')
ioquake3.js:3006 File "vm/ui.qvm" found in "/base/baseq3/pak8.pk3"
ioquake3.js:3006 VM file ui compiled in 232 milliseconds
ioquake3.js:3006 Reading file vm/ui.map
ioquake3.js:3006 Loading file /base/baseq3/vm/ui.map
ioquake3.js:3006 Couldn't load symbol file: vm/ui.map
ioquake3.js:3006 ui loaded in 1368608 bytes on the hunk
ioquake3.js:3006 ----- Client Shutdown (Client fatal crashed: program stack corrupted, is 1031716, expected 1048516

I think once that is fixed the game should load.

EDIT:pipeline, I forgot I intentionally excluded pak0 because it's large.
I just have to merge my changes with NTT123. It might show better performance and work with WebGL2.

[ghost]

Working in this branch:
https://github.com/briancullinan/planet-quake/commits/repairing-images

I should have a PR pretty soon for ioq3. There is probably still some work to get new environments working conveniently. Make could install emsdk (kind of a pain) for example.

The gist of this is, ioq3 has reorganized a lot since 2014, but the changes are pretty simple.

• Backed out the alpha shader stuff, and the &trs struct for shaders, the most recent commit. Maybe the most recent commit didn't work and that's why inolen stopped, not knowing which commit was working made it more difficult to merge.
• There are some bugs in emscripten's library_memfs.js loading files remotely, I have to PR but until then I added new overloads locally.
• tr_extensions.c and sdl_glimp.c use fancy new GLE compiler templates, so that was a pain to figure out what code actually needed to map to qgl functions (some of these functions are mapped directly to gl* in the code already, it isn't consistent).
• tr_extensions.c changed a lot so I had to figure out which ones still apply. Logs from working quakejs came in handy and seeing old commits when it was overhauled.

[ghost]

Turns out some of the hardship in debugging this, in case anyone is still following. emscripten acts differently when it is compiled to WASM=1 versus JS. I assume this is because assembler instructions are run inside some sort of For-loop. The javascript thread won't break out of a for loop, but running in JS, calling other functions makes them async. This is a problem because the HEAP is managed by one thread at a time. Got network multiplayer working with the UDP proxy I copied from another issue here. Will post to a quakejs backup AWS instance before I am done.

It appears everything is working properly. I still want to back out of some of the file system/download changes, and reimplement the CDN in C because it just seems like a good idea. Does the server transfer files to every client currently? That would be crazy, no reason we shouldn't be able to direct clients to make an HTTP request somewhere else and verify checksum.

Speaking of checksum, that's another thing I have to back out. I like the CDN/manifest/ breaking maps and game data out of the paks so they can be loaded individually. If we keep that pattern, I have to get checksums working normally. Using additional CRC in javascript is silly when the engine already does it. Sorry to be so critical of the code.

Adding function declarations to the header helps a lot with the commit because it doesn't look like much code moved. This is the branch I am currently on and will likely be the branch I PR to ioq3 as a build mode.

https://github.com/briancullinan/ioq3/tree/ioq3-quakejs

[ghost]

@a2kas @inolen @NTT123 I think the slowness comes from playing background level music.

It kept logging "Restarted OpenAL music" so I disabled it in snd_openal.c in the build and now it appears to stay open for +10 minutes. Before it would get to 10 minutes, sounds would start skipping, and dropping frames.

https://quake.games/?set%20name%20Name

EDIT: nvm, still seeing that 100% CPU after 10 minutes.

[ghost]

This looks absolutely perfect for the JS build:

https://github.com/cdev-tux/q3lite

@cdev-tux How much work would it be to combine the ES version upstream as a build option? I will try a cherry pick from my branch anyways.

[cdev-tux]

Q3lite uses a modified GPLv3 license (DOOM 3), and the upstream ioquake3 project uses a GPLv2 license so they can’t use GPLv3 code. They would need to find a GPLv2 GLES renderer I believe. It would be straightforward for one of their programmers to add it.

[ghost]

@cdev-tux Oh. I am not sure what licensing has to do with progress... Could ioq3 upgrade their license?

https://quake.games/?set%20com_maxfps%2035

Setting the frame rate at 35 (slightly higher than what I feel like the eye notices because a few frames are dropped) instead of the default 85. Maybe the garbage collector catches up with the emscripten GL stuff? It returns to 30-40% CPU usage after only a few seconds of play.

OpenArena adds a maxInactiveFPS setting, I'll look into getting ported, maybe when the tab is inactive the requestAnimationFrame rate is lowered in the JS client.

Is WebAssembly supposed to be as fast as bare metal?

[ghost]

Didn't mean to hijack the thread. Still working on the engine if anyone needs anything. Not sure where to take the whole manifest.json and pk3 graphing, but it desperately needs improvement since @ebbesand123 reached out to me. We tried to convert Defrag mod using repak with no luck. My plan was to use the dedicated server build as a command line tool, and generate the graph using existing/appended C code.

Here's the article on loading multiple maps at the same time, if anyone is interested:
https://blog.briancullinan.com/article/performing-game-engine-surgery

[ghost]

I tried to reapply the fast vid_restart hack, but it doesn't seem to work :(

Not sure how to fix this performance stuff, but running with -s SAFE_HEAP=1 causes lots of weird errors, I am going to try to find changes in other engines that improve performance.

For example, this might help if the HEAP is leaking:
ec-/Quake3e@bef917a

Instead of the alpha and shader stuff, I might try out this renderer from vitaQuake or q3lite:
https://github.com/briancullinan/q3lite/commit/c26300e278a5782f2ae3982d35e951561c968167

It's GLES1 instead of something new like ES2 or Vulkan, but it might run at a better FPS, there is also this BGFX renderer that looks nice, but I don't know if it will work on emscripten:
https://github.com/jpcy/ioq3-renderer-bgfx

[ghost]

@JakeTrock @NTT123 @a2kas I can say with certainty the slowness is caused by S_Respatialize() called in CGame. It might be calling too often for Emscripten OpenAL (audio layer), it's only partially supported and the respatializing function is slowing things down. I've run the Performance calculator in Chrome and it takes up 99% of the CPU. Turning audio off completely using +set s_initsound 0 makes the problem go away instantly. Runs consistently at 60+ FPS no problem. No slowing down from leaving it idling. I personally, really like a game with sound. Any ideas on how to proceed? I don't know if only calculating it 10 times per second will affect it. Maybe it is affecting the HEAP improperly. I am going to try to pull in this change from Quake3e to see if it helps.

EDIT: Looks like adding a little time buffer to limit the call to emscripten alListenerfv() in S_AL_Respatialize() is good enough not to drop the frame-rates. I set this to 1000/35 so a little more than 25x per second because the audio rate is also set at a static 25x per second. I think it is noticeably different, but if it was my first time playing this 20 years ago, I am not sure I could tell a difference.
https://emscripten.org/docs/porting/Audio.html

Very strange indeed. I'm just glad I can proceed to other, more visual, features.