Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make basis_transcoder.{js,wasm} substantially smaller #7

Merged
merged 5 commits into from May 23, 2019

Conversation

Projects
None yet
4 participants
@zeux
Copy link
Contributor

commented May 23, 2019

This change substantially reduces the size of Basis transcoder when built for Web; this means that the transcoder itself can download and compile faster.

The changes are as follows:

  • Rebuild transcoder using latest Emscripten (1.38.31)
  • Remove std::iostream (it has large impact on both JS and WASM code, and is only used for one console error message)
  • Make BASISD_SUPPORT macros configurable externally and disable BC7 for web build since BC7 doesn't seem to be currently supported in WebGL in practice (BC7 decoding tables are large)
  • Use emmalloc instead of default dlmalloc (allocations are slightly slower but there are very few of them so it doesn't impact transcoding time)

This change also cleans up the use of assert(x) by adding NDEBUG - this change isn't very significant on the code size, but removes a hack that's not necessary

The cumulative impact of the changes is as follows (individual impact is recorded in each commit message):

Before:
basis_transcoder.js 139216
basis_transcoder.wasm 886294
basis_transcoder.js.gz + basis_transcoder.wasm.gz 366383

After:
basis_transcoder.js 58707
basis_transcoder.wasm 217809
basis_transcoder.js.gz + basis_transcoder.wasm.gz 124984

There are some more gains to be had in wasm and JS code but they seem to come with performance tradeoffs or more complex build pipeline, so I've settled on this for now.

Re: BC7: it technically can be supported via EXT_texture_compression_bptc extension that's currently "community approved". However, neither Chrome nor Firefox implement it on Windows, and the compatibility table is uninspiring: https://developer.mozilla.org/en-US/docs/Web/API/EXT_texture_compression_bptc. Since BC7 tables take ~470 KB of wasm, I think it makes sense to compile this out for now, and add BC7 if and when it becomes a viable option in WebGL.

zeux added some commits May 23, 2019

Rebuild basis_transcoder with new Emscripten (emsdk 1.38.31)
basis_transcoder.js 141235
basis_transcoder.wasm 833219
Remove assert(x) hack in favor of NDEBUG
basis_transcoder.js 140999
basis_transcoder.wasm 832985
Remove std::iostream from the transcoders
The impact on the file size is enormous, and it's used to output one
line into console which isn't exposed through the API anyhow.

basis_transcoder.js 58707
basis_transcoder.wasm 705079
Make codec options externally configurable and remove BC7 support
This change makes it possible to exclude formats using compile time
options. BC7 is *never* exposed on the web and it has an absolutely huge
coding table which means we're just wasting space.

basis_transcoder.js 58707
basis_transcoder.wasm 224404
Use emmalloc instead of dlmalloc to save a bit more space
basis_transcoder.js 58707
basis_transcoder.wasm 217809
@kenrussell
Copy link
Collaborator

left a comment

Thanks @zeux for looking into the file size of the wasm build! These are tremendous improvements!

CC @austinEng , @shrekshao , @kainino0x

@@ -39,7 +37,6 @@ struct basis_file

if (!m_transcoder.validate_header(m_file.data(), m_file.size())) {
m_file.clear();
std::cerr << "Invalid Basis header" << std::endl;

This comment has been minimized.

Copy link
@kenrussell

kenrussell May 23, 2019

Collaborator

Some sort of error reporting is desirable - at least a bool isValid() method returning (m_magic == MAGIC).

This comment has been minimized.

Copy link
@zeux

zeux May 23, 2019

Author Contributor

Note that error handling can/should be performed by checking the result of startTranscoding() call - the extra console output may be valuable for debugging but not as part of the production .wasm binary.

For debugging, you can build the transcoder with BASISU_DEVEL_MESSAGES option which will emit detailed error messages indicating what specifically went wrong with the header.

This comment has been minimized.

Copy link
@zeux

zeux May 23, 2019

Author Contributor

Separately, I think m_magic variable and all associated code can be removed based on the fact that every method of transcoder is doing quick header validation which should be ~as strong as the code in this file; not that big of a deal but that seems cleaner. It will also make it so that JS code can call startTranscoding and other functions without hitting assertions in debug builds for malformed files.

This comment has been minimized.

Copy link
@austinEng

austinEng May 23, 2019

Collaborator

I agree we can remove the magic. This is old code when we weren't using Embind and had a more C-like api. It was possible to pass a raw pointer to any of the functions. Now that we have a proper BasisFile constructor, we could just have the constructor throw a Javascript exception (or have a createBasisFile function return undefined). That way none of the member functions are accessible unless you first make a valid BasisFile. This plus the quick header validation should be sufficient.

@richgel999

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

Thank you, this is awesome. Transcoder download size is very, very important so I will test and merge this in tomorrow (Thursday).

For BC7, I've been planning on reducing the size of the tables, so eventually when BC7 is supported more widely on the Web it can be enabled. BC7 doesn't add much extra value now, but it will once I get higher quality support in there.

@richgel999 richgel999 merged commit 0319b50 into BinomialLLC:master May 23, 2019

@richgel999

This comment has been minimized.

Copy link
Contributor

commented May 23, 2019

It's in, thanks!

@donmccurdy donmccurdy referenced this pull request May 23, 2019

Open

BasisTextureLoader: Next steps #16524

7 of 12 tasks complete

@zeux zeux referenced this pull request Jun 3, 2019

Open

Modularize emscripten build. #26

zeux added a commit to zeux/basis_universal that referenced this pull request Jun 3, 2019

Fix webgl CMakeLists.txt setup
It looks like BinomialLLC#7 didn't update CMakeLists.txt properly - defines
specified with -D have to be added as compile definitions.

Additionally, -O3 must be specified redundantly as a compile option and
as a link option - compile option makes .wasm bits small & fast, and
link option enables JS minifier.

After this fix CMake build results in minimal js/wasm files as well.

@zeux zeux referenced this pull request Jun 3, 2019

Merged

Fix webgl CMakeLists.txt setup #27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.