documentation confusion for Draco javascript decoder build

[Gassepouille]

Hi,

I've been trying to load 50+ millions points (200MB+) point cloud with DracoLoader for threejs,
But it gives me an error on loading:
Aborted(). Build with -s ASSERTIONS=1 for more info.

So I tried to build the decoder,
Cloning the emscripten repo,
But in your documentation, you have this step:
export EMSCRIPTEN=../../emscripten/tools/parent
The problem being I do not have this folder on the emscripten repo (I tried multiple version of it but I couldn't find that folder)
And when I run make -s ASSERTIONS=1, I can't see any .js files in the build_dir repo.

So maybe I'm doing something wrong.
Forgive me if that's not the right place to ask.

edit: added some more info

[tomfinegan]

export EMSCRIPTEN=../../emscripten/tools/parent

The above means the parent directory of the tools directory in the currently active Emscripten installation. It will also contain a cmake directory that contains the subpath /cmake/Modules/Platform where the Emscripten CMake toolchain file Emscripten.cmake lives.

The problem being I do not have this folder on the emscripten repo (I tried multiple version of it but I couldn't find that folder)
And when I run make -s ASSERTIONS=1, I can't see any .js files in the build_dir repo.

The Draco CMake build won't generate any javascript without an installed and activated Emscripten SDK. If you have not already, follow the Emscripten installation documentation.

Note that if you have multiple python versions installed on your system that Emscripten builds can fail in weird and interesting ways due to picking up multiple python versions during builds. In this situation it's best to be explicit when running CMake, so add this argument when configuring Draco: -DPYTHON_EXECUTABLE:FILEPATH=python3.7 (adjust accordingly for the python version you have installed).

Edit to add more about the ASSERTIONS setting:

And when I run make -s ASSERTIONS=1, I can't see any .js files in the build_dir repo.

I don't think the above will do any good even if you configure everything correctly. If you really want an assertions enabled Emscripten build of Draco your CMake configuration command needs to look something like the following:

cmake  \
  -DCMAKE_TOOLCHAIN_FILE=/path/to/activated/emscripten/cmake/Modules/Platform/Emscripten.cmake \
  -DPYTHON_EXECUTABLE:FILEPATH=python3.7 \
  -DDRACO_CXX_FLAGS=-sASSERTIONS=1 \
  -DDRACO_EXE_LINKER_FLAGS=-sASSERTIONS=1

Note the omission of the space between the -s argument its value. CMake will punish you if you add a space as shown in your make command: if lucky configuration will fail, if unlucky the build will appear successful yet produce completely broken output that will only throw strange looking exceptions.

[Gassepouille]

Thanks for all the information!
I'm way less confused now :D

I'm not gonna be able to try it this weekend, but on Monday should be good ! I'm quite confident about this though !

[Gassepouille]

I'm so sorry, I can't seem to make it work...

So far I have 3 folders

/parentFolder/emscripten/
/parentFolder/emsdk/
/parentFolder/draco/

I'm on WSL,
Python is installed (python3 --version : v. Python 3.8.5)

I start with (from parentFolder/emsdk/) :

# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes .emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

then I run the following commands (from parentFolder/draco/build_dir/):

export EMSCRIPTEN=../../emscripten/
cmake ../ -DCMAKE_TOOLCHAIN_FILE=../../emscripten/cmake/Modules/Platform/Emscripten.cmake -DPYTHON_EXECUTABLE:FILEPATH=python3 -DDRACO_CXX_FLAGS=-sASSERTIONS=1 -DDRACO_EXE_LINKER_FLAGS=-sASSERTIONS=1

I get this output :

--- Running combined CXX flags test, flags: -sASSERTIONS=1;-sASSERTIONS=1
--- Passed combined CXX flags test
-- Configuring done
-- Generating done
-- Build files have been written to: parentFolder/draco/build_dir

But the build_dir/ doesn't contain any js files (following is the content of the build_dir/) :

-rwxrwxrwx 1 gassepouille gassepouille  19K Jan 10 12:00 CMakeCache.txt*
drwxrwxrwx 1 gassepouille gassepouille 4.0K Jan 10 12:05 CMakeFiles/
-rwxrwxrwx 1 gassepouille gassepouille 212K Jan 10 12:05 Makefile*
-rwxrwxrwx 1 gassepouille gassepouille 165K Dec 27 11:13 cmake_install.cmake*
drwxrwxrwx 1 gassepouille gassepouille 4.0K Jan 10 12:05 draco/
-rwxrwxrwx 1 gassepouille gassepouille 1019 Dec 27 11:13 draco-config.cmake*
-rwxrwxrwx 1 gassepouille gassepouille 1.4K Dec 27 11:13 draco-version.cmake*
-rwxrwxrwx 1 gassepouille gassepouille  155 Dec 27 11:13 draco.pc*
lrwxrwxrwx 1 gassepouille gassepouille   19 Dec 27 11:30 draco_decoder -> draco_decoder-1.5.0*
-rwxrwxrwx 1 gassepouille gassepouille 2.8M Dec 27 11:30 draco_decoder-1.5.0*
lrwxrwxrwx 1 gassepouille gassepouille   19 Dec 27 11:29 draco_encoder -> draco_encoder-1.5.0*
-rwxrwxrwx 1 gassepouille gassepouille 2.8M Dec 27 11:29 draco_encoder-1.5.0*
lrwxrwxrwx 1 gassepouille gassepouille   22 Dec 27 11:31 draco_transcoder -> draco_transcoder-1.5.0*
-rwxrwxrwx 1 gassepouille gassepouille 2.8M Dec 27 11:31 draco_transcoder-1.5.0*
-rwxrwxrwx 1 gassepouille gassepouille 6.2M Dec 27 11:28 libdraco.a*

The previous files are, I guess , the files from the classic build :

$ mkdir build_dir && cd build_dir
$ cmake ../

Maybe there is something obvious that I'm missing...

[tomfinegan]

I think you need to clean your build directory. It appears that CMake is ignoring the toolchain argument and updating the existing configuration when you run it-- delete the contents of build_dir and start with the cmake command you included above.

[Gassepouille]

Sweet ! It worked!

So now, when loading my 50Million points scan I have this (which I believe is triggering the error):
Cannot enlarge memory, asked to go up to 2362150912 bytes, but the limit is 2147483648 bytes!
Followed by: Aborted(native code called abort())

I was looking into this thread:
#571
specifically this: #571 (comment)

Seems like emscripten can handle a maximum of 4GB.

I'm trying to figure out how to run this :
emcc -s ALLOW_MEMORY_GROWTH -s MAXIMUM_MEMORY=4GB
but I'm not sure how to pass this flag in my current configuration though.

[tomfinegan]

ALLOW_MEMORY_GROWTH is already set for the Draco build. If you watch the CMake output you'll see the Draco build testing it in the combined flag check.

I think this model may simply be too large, but you can try increasing the max memory. It's the same procedure as the assertions. Here's an example with both enabled.

cmake ../ -DCMAKE_TOOLCHAIN_FILE=../../emscripten/cmake/Modules/Platform/Emscripten.cmake \
  -DPYTHON_EXECUTABLE:FILEPATH=python3 \
  -DDRACO_CXX_FLAGS=-sASSERTIONS=1\;-sMAXIMUM_MEMORY=4GB \
  -DDRACO_EXE_LINKER_FLAGS=-sASSERTIONS=1\;-sMAXIMUM_MEMORY=4GB

[Gassepouille]

So, first of all, thanks for the help and for being so quick to respond !

I think this model may simply be too large, but you can try increasing the max memory. It's the same procedure as the assertions. Here's an example with both enabled.

Yes, the model is definitely too large to be handled by the decoder with just 2GB of max memory

So I tried the command you paste (with the build_dir fully cleaned before), it create files (so I guess it worked kinda ?)
But when loading the drc, I have the same warning (more or less) that I had when I only had 2GB of max memory:

emscripten_realloc_buffer: Attempted to grow heap from 1864105984 bytes to 2362179584 bytes, but got error: RangeError: Invalid array buffer length

Failed to grow the heap from 1864105984 bytes to 2362179584 bytes, not enough memory!

Aborted(native code called abort())

Tried with another huge model :

emscripten_realloc_buffer: Attempted to grow heap from 2114387968 bytes to 2697920512 bytes, but got error: RangeError: Invalid array buffer length

and same following errors

So it feels like it didn't take the 4GB max memory ? Or maybe there is something that I'm missing ?
I'm just updating the "draco_decoder.js" and I keep "draco_decoder.wasm" and "draco_wasm_wrapper.js" or should I regenerate them too ?

Edit : just to make sure I had the right new file, I rebuilt it without the assertion and I had indeed the right file.

[tomfinegan]

So it feels like it didn't take the 4GB max memory ? Or maybe there is something that I'm missing ?

I suspect that the browser running your javascript simply will not allow such a large allocation.

[Gassepouille]

Mhhh, I'm running chrome on windows, Version 97.0.4692.71 (Official Build) (64-bit)
It's weird, because I keep reading that the maximum memory per tab is 4GB but somehow,
when I ran tests like this: https://nodachisoft.com/common/en/article/en000005/ (not sure if that's reliable though)
It never goes past 1600974592Bytes.
(when I check at the task manager it seems that it does go up to 4GB., and then it crashes).

Even weirder:
When I check the window.performance.memory.jsHeapSizeLimit
I get : 4294705152.
So around 4GB.

Could that be it though? : chromium bug

I'm so sorry to bother you with this...

[tomfinegan]

Could that be it though? : chromium bug

The referenced bug and the bugs it links to make some pretty clear statements that a tab won't be allowed to allocate over 4GB of memory. The size of the javascript heap is irrelevant if the process/tab hosting it caps out at 4 GB. You might be able to allocate somewhere between 2GB and 4GB, but allocating 4 GB on the JS heap certainly won't be possible if the host environment is limited to 4 GB total. The browser needs at least some memory to manage the tab itself along with whatever else is contained within it.

[tomfinegan]

Closing this-- it's pretty clear the documentation is correct, and we're now going down the path of debugging your environment. If there's an issue here it's the browser limiting the total memory allowed to be allocated by javascript, but everything seems to be working-as-intended on the browser side as well. The file you're attempting to load is simply too large for the javascript decoder to handle due to runtime environment limitations.

[Gassepouille]

That's fair!

Thanks a lot for the help!