Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation time, CI time and executable size improvements #293

Open
mosra opened this issue Nov 2, 2018 · 2 comments

Comments

1 participant
@mosra
Copy link
Owner

commented Nov 2, 2018

For the 2018.1d release (and onwards) I'd like to focus on reducing the header and executable size, together with improving compile time (and, as a side effect, runtime performance). This was last done in 2013 (see the blog article) and while current workflow enforces enough rules to prevent worsening of this problem, it's not actively improving it either.

The ultimate goal is being able to ship useful utilities as "single-header" libraries without being laughed at for compile times and having compile times competitive with C header-only libs, yet staying in C++. Which, of course, means much better compile times than other C++ projects (json.hpp and Eigen, I'm looking at you).

Compile time improvements

Executable size reduction / perf improvements (mainly WebAssembly-focused)

  • Remove symbol visibility for static libraries
  • Get rid of stuff reported by -Wglobal-constructors -Wexit-time-destructors on Clang
    • GL::defaultFramebuffer,
    • stuff in GL::Context -- b05c887
    • driver-specific workaround list
      • sort it by hand to have fast lookup
  • Put all separate GL function symbols into a big struct to reduce amount of exported symbols (basically the same way as Vulkan does it now)
  • Investigate gains of -Oz with Emscripten, switch the toolchains to that, fix Emscripten closure compiler (#211) and investigate how much can it shave off the JS code (around 200 kB?) -- mentioned in the docs as of df6b414, not enabling by default since it has compile time impact
  • Some preprocessor hook for Utility::Resource that's able to strip license headers off shader sources
    • should be just a function pointer with user ptr, otherwise we'd need to depend on PluginManager
    • or encrypt/compress them (and another decrypt/decompress hook at runtime)
  • Reduce repetitive strings in debug output literals for enums (print the prefix just once)
  • Try out Emscripten with minified imports/exports
  • Usage of std::sort() and std::unordered_map for extensions during context creation is almost 20% of code size
  • Make it possible to compile for Emscripten with -s FILESYSTEM=0 (compile away parts of Utility::Directory, GL::Shader::addFile(), non-callback-based Trade::AbstractImporter::openFile() etc.
  • There's one entry per every global wasm constructor (affects resources and static plugin imports), reduce that somehow (combining static plugin import with resource import, importing the resource explicitly in the code...). Related: emscripten-core/emscripten#6904 (comment)
  • Investigate using brotli instead of gzip for wasm compression (Qt says all wasm-enabled browsers support it, what about Apache?)

Bigger tasks

  • Remove all usage of C++ iostreams (just using std::cout adds 250 kB to JS + WASM size)
    • Convert iostreams to C I/O + Utility::format() -- app using std::printf() has only 40 kB compared to that
      • the Utility::Debug class
      • Arguments help / usage formatting
      • Resource binary-to-hex conversion
      • std::stringstream debug redirection in many tests -- how else? append to a string? doesn't need to be performant
      • file I/O in Utility::Directory -- mosra/corrade@c1a5eed
    • Reimplement printf-based Utility::format() without printf (float conversion with Ryū, integer conversion using "the fastest ever integer conversion" as claimed by the author of fmt) -- float32 tables in Ryū are 624 B and even float64 tables in Ryū are just 10 kB and with Utility::format() if we don't print doubles, the tables won't even get compiled in
    • Remove all uses of printf() -- a naive copypasted implementation using grisu3 was just 25 kB, shaving > 10 kB compared to printf (and being much faster)
      • Ensure dependencies (plugins) that matter for WASM don't use it (would be hard to ensure for tinygltf, ugh)
      • It gets used by libc++'s abort_handler(), patch emscripten to not do that
    • This all needs a blog post (compare to competing implementations)
  • Create a direct EmscriptenApplication instead of using Sdl2Application -- should trim down at least the generated *.js file size (the library_sdl.js is 137 kB (though unminified)) in progress #300
  • Port away from tiny_gltf and json.hpp (json.hpp alone is a 400 kB header and the recent versions are almost 700 kB)
    • tiny_gltf might be going away from json.hpp on its own (syoyo/tinygltf#141)
    • just the TinyGltfImporter plugin compilation alone takes around 15 seconds -- for a single file -- which is more than all other plugins combined
    • I bet it has some effect on WASM output size as well, just don't know how much
  • Remove hard dependncy on GL from the Text library by creating an abstract API-independent base for glyph cache -- 834c5fe
    • That'll allow the plugins to be built and tested without needing to take care of GL/GLES/WebGL differences -- mosra/magnum-plugins@e6f8792
  • Make it possible to fully disable the debug output (and then define CORRADE_ASSERT to the C assert) -- needed for the single-header libs, done in mosra/corrade@64c56aa and cee5307
    • Similarly for configuration value and tweakable literal parsers -- 64bc7f9 and 77a8c0c
  • Provide STL-interfacing APIs only as an opt-in

CI speedup

  • A separate repository for all dependencies we currently build manually in every CI job (ANGLE, SDL for WinRT, Bullet, GLFW...)
    • download the binaries from ci.magnum.graphics
    • some token authentication so it's not publicly accessible (just restricting to Travis/AppVeyor IPs is not enough, as we want to prevent mainly CI users from abusing the server)
  • Build the ES2/ES3 variants without code that's not API-dependent

Long-term

  • Opt-in resizing APIs for Containers::Array so we can ditch std::vector as a growable storage, eliminating it from headers completely
  • A string view class so we can get rid all the const char* / const char(&)[n] / const std::string& overloads everywhere, again eliminating std::string from headers completely
    • And a String class as well, with small string optimization (https://wg21.link/P1330, chapter 4) and easily convertible from/to the string view

Further work

  • Investigate compiling with a lighter-weight STL implementation (e.g. nanostl, EASTL?) -- most of them have no type_traits, we need type_traits :(
  • Can C++20(?) modules help in any way with compile times? So far I didn't see any experiment that would prove a breakthrough in compile times -- probably not
  • The Utility/Debug.h headers will be still quite heavy after forward-declaring strings and removing <iterator> and since these get used almost everywhere, what to do? no it's not, it's fine since mosra/corrade@89da382 (just <utility> and <type_traits>)

Further read / references:

@mosra mosra added this to the 2018.1d milestone Nov 2, 2018

@mosra mosra self-assigned this Nov 2, 2018

@mosra mosra added this to TODO in Project management via automation Nov 2, 2018

@mosra mosra added the help wanted label Nov 2, 2018

@mosra mosra moved this from TODO to In progress in Project management Nov 27, 2018

@mosra mosra referenced this issue Dec 29, 2018

Closed

2019.01 #302

55 of 55 tasks complete

@mosra mosra modified the milestones: 2019.01, 2019.0b Dec 29, 2018

@mosra mosra pinned this issue Dec 30, 2018

@mosra

This comment has been minimized.

Copy link
Owner Author

commented Mar 7, 2019

Progress!

@mosra

This comment has been minimized.

Copy link
Owner Author

commented Apr 9, 2019

More progress:

  • forward declaration headers for STL containers
  • splitting away STL stuff from Utility::Debug and Utility::Format
  • PIMPLing away things
  • ...

See above for the actual commit references.

@mosra mosra referenced this issue May 28, 2019

Draft

[WIP] Test emscripten build for WebGL deployment #53

6 of 15 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.