Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add -s MINIMAL_RUNTIME=1 option. #7923

Merged
merged 4 commits into from
Feb 5, 2019
Merged

Add -s MINIMAL_RUNTIME=1 option. #7923

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f99066c
Add -s MINIMAL_RUNTIME=1 option.
juj Jan 22, 2019
35ab6c4
Do not attempt to run MINIMAL_RUNTIME tests in SpiderMonkey or v8
juj Feb 5, 2019
a4cd806
Update minimal_runtime to latest incoming
juj Feb 5, 2019
6445e39
Update expected code sizes
juj Feb 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 89 additions & 21 deletions emcc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -663,7 +663,7 @@ def uniquename(name):

specified_target = target
target = specified_target if specified_target is not None else 'a.out.js' # specified_target is the user-specified one, target is what we will generate
target_basename = unsuffixed_basename(target)
shared.Settings.TARGET_BASENAME = target_basename = unsuffixed_basename(target)

final_suffix = suffix(target)

Expand Down Expand Up @@ -1043,6 +1043,7 @@ def check(input_file):
assert not (not shared.Settings.DYNAMIC_EXECUTION and options.use_closure_compiler), 'cannot have both NO_DYNAMIC_EXECUTION and closure compiler enabled at the same time'

if options.emrun:
assert not shared.Settings.MINIMAL_RUNTIME, '--emrun is not compatible with -s MINIMAL_RUNTIME=1'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--emrun support could be added later

shared.Settings.EXPORTED_RUNTIME_METHODS.append('addOnExit')

if options.use_closure_compiler:
Expand Down Expand Up @@ -1122,7 +1123,7 @@ def check(input_file):
shared.Settings.EXPORTED_FUNCTIONS += ['___cxa_demangle']
forced_stdlibs += ['libc++abi']

if not shared.Settings.ONLY_MY_CODE:
if not shared.Settings.ONLY_MY_CODE and not shared.Settings.MINIMAL_RUNTIME:
# Always need malloc and free to be kept alive and exported, for internal use and other modules
shared.Settings.EXPORTED_FUNCTIONS += ['_malloc', '_free']
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unconditionally adding malloc and free should not occur, but it should only happen based on linking decisions dictating that they are needed.

if shared.Settings.WASM_BACKEND:
Expand Down Expand Up @@ -1285,6 +1286,24 @@ def check(input_file):
if not shared.Settings.SEPARATE_ASM_MODULE_NAME:
shared.Settings.SEPARATE_ASM_MODULE_NAME = 'Module["asm"]'

if shared.Settings.MINIMAL_RUNTIME:
# Minimal runtime uses a different default shell file
if options.shell_path == shared.path_from_root('src', 'shell.html'):
options.shell_path = shared.path_from_root('src', 'shell_minimal_runtime.html')
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A new default shell file for MINIMAL_RUNTIME shows a simple example of how the user should XHR in the compiled code files. The model of how asm.js/wasm/mem file glues together with the main JS file stays very much the same.


# Remove the default exported functions 'memcpy', 'memset', 'malloc', 'free', etc. - those should only be linked in if used
shared.Settings.DEFAULT_LIBRARY_FUNCS_TO_INCLUDE = []
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default don't export anything extra, but user should explicitly opt in to whatever is needed.


# Always build with STRICT mode enabled
shared.Settings.STRICT = 1

# Always use the new HTML5 API event target lookup rules (TODO: enable this when the other PR lands)
# shared.Settings.DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR = 1

# In asm.js always use memory init file to get the best code size, other modes are not currently supported.
if not shared.Settings.WASM:
options.memory_init_file = True
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code size suboptimal memory initializer modes are not supported (if someone would like to integrate those back in, would be ok as long as it occurs in a way that does not require Closure to clean up bloat caused by them)


if shared.Settings.WASM:
if shared.Settings.SINGLE_FILE:
# placeholder strings for JS glue, to be replaced with subresource locations in do_binaryen
Expand Down Expand Up @@ -1317,18 +1336,20 @@ def check(input_file):
if any(s.startswith('MEM_INIT_METHOD=') for s in settings_changes):
exit_with_error('MEM_INIT_METHOD is not supported in wasm. Memory will be embedded in the wasm binary if threads are not used, and included in a separate file if threads are used.')
options.memory_init_file = True
if shared.Settings.BINARYEN_ASYNC_COMPILATION == 1:
# async compilation requires a swappable module - we swap it in when it's ready
shared.Settings.SWAPPABLE_ASM_MODULE = 1
else:
# if not wasm-only, we can't do async compilation as the build can run in other
# modes than wasm (like asm.js) which may not support an async step
shared.Settings.BINARYEN_ASYNC_COMPILATION = 0
warning = 'This will reduce performance and compatibility (some browsers limit synchronous compilation), see http://kripken.github.io/emscripten-site/docs/compiling/WebAssembly.html#codegen-effects'
if 'BINARYEN_ASYNC_COMPILATION=1' in settings_changes:
logger.warning('BINARYEN_ASYNC_COMPILATION requested, but disabled because of user options. ' + warning)
elif 'BINARYEN_ASYNC_COMPILATION=0' not in settings_changes:
logger.warning('BINARYEN_ASYNC_COMPILATION disabled due to user options. ' + warning)

if not shared.Settings.MINIMAL_RUNTIME: # BINARYEN_ASYNC_COMPILATION and SWAPPABLE_ASM_MODULE do not have a meaning in MINIMAL_RUNTIME (always async)
if shared.Settings.BINARYEN_ASYNC_COMPILATION == 1:
# async compilation requires a swappable module - we swap it in when it's ready
shared.Settings.SWAPPABLE_ASM_MODULE = 1
else:
# if not wasm-only, we can't do async compilation as the build can run in other
# modes than wasm (like asm.js) which may not support an async step
shared.Settings.BINARYEN_ASYNC_COMPILATION = 0
warning = 'This will reduce performance and compatibility (some browsers limit synchronous compilation), see http://kripken.github.io/emscripten-site/docs/compiling/WebAssembly.html#codegen-effects'
if 'BINARYEN_ASYNC_COMPILATION=1' in settings_changes:
logger.warning('BINARYEN_ASYNC_COMPILATION requested, but disabled because of user options. ' + warning)
elif 'BINARYEN_ASYNC_COMPILATION=0' not in settings_changes:
logger.warning('BINARYEN_ASYNC_COMPILATION disabled due to user options. ' + warning)

if not shared.Settings.DECLARE_ASM_MODULE_EXPORTS:
# Swappable wasm module/asynchronous wasm compilation requires an indirect stub
Expand All @@ -1337,10 +1358,6 @@ def check(input_file):
if shared.Settings.SWAPPABLE_ASM_MODULE == 1:
shared.Settings.DECLARE_ASM_MODULE_EXPORTS = 1
logger.warning('Enabling -s DECLARE_ASM_MODULE_EXPORTS=1 since -s SWAPPABLE_ASM_MODULE=1 is used')
# Wasm -O3 builds use Meta-DCE which is currently not compatible with -s DECLARE_ASM_MODULE_EXPORTS=0 option.
if will_metadce(options):
shared.Settings.DECLARE_ASM_MODULE_EXPORTS = 1
logger.warning('Enabling -s DECLARE_ASM_MODULE_EXPORTS=1 since -O3/-Os build with Wasm meta-DCE is used')

# we will include the mem init data in the wasm, when we don't need the
# mem init file to be loadable by itself
Expand Down Expand Up @@ -1405,6 +1422,16 @@ def check(input_file):
if not shared.Settings.WASM and (shared.Settings.MAIN_MODULE or shared.Settings.SIDE_MODULE):
assert not shared.Settings.ALLOW_MEMORY_GROWTH, 'memory growth is not supported with shared asm.js modules'

if shared.Settings.MINIMAL_RUNTIME:
if shared.Settings.ALLOW_MEMORY_GROWTH:
logging.warning('-s ALLOW_MEMORY_GROWTH=1 is not yet supported with -s MINIMAL_RUNTIME=1')

if shared.Settings.EMTERPRETIFY:
exit_with_error('-s EMTERPRETIFY=1 is not supported with -s MINIMAL_RUNTIME=1')

if shared.Settings.USE_PTHREADS:
exit_with_error('-s USE_PTHREADS=1 is not yet supported with -s MINIMAL_RUNTIME=1')

if shared.Settings.ALLOW_MEMORY_GROWTH and shared.Settings.ASM_JS == 1:
# this is an issue in asm.js, but not wasm
if not shared.Settings.WASM:
Expand Down Expand Up @@ -1437,6 +1464,10 @@ def check(input_file):
options.separate_asm = True
shared.Settings.FINALIZE_ASM_JS = False

# MINIMAL_RUNTIME always use separate .asm.js file for best performance and memory usage
if shared.Settings.MINIMAL_RUNTIME and not shared.Settings.WASM:
options.separate_asm = True

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always use separate .asm.js files, so that it can be asynchronously loaded, debugging the main JS is feasible when the compiled asm.js code is out of sight, and Firefox can unload asm.js script content from memory to disk.

if shared.Settings.GLOBAL_BASE < 0:
shared.Settings.GLOBAL_BASE = 8 # default if nothing else sets it

Expand Down Expand Up @@ -1918,7 +1949,11 @@ def get_final():
with ToolchainProfiler.profile_block('memory initializer'):
memfile = None
if shared.Settings.MEM_INIT_METHOD > 0 or embed_memfile(options):
memfile = target + '.mem'
if shared.Settings.MINIMAL_RUNTIME:
# Independent of whether user is doing -o a.html or -o a.js, generate the mem init file as a.mem (and not as a.html.mem or a.js.mem)
memfile = target.replace('.html', '.mem').replace('.js', '.mem')
else:
memfile = target + '.mem'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a simplification/bugfix to allow not having to deal with two suffixed files, depending on if one compiled to JS or HTML. Also a.mem is shorter than a.html/js.mem

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about doing this in all modes? (in a separate PR) There is a slight annoyance for existing users that would need to look at a different file, but I think that's a one-time annoyance, while if changing modes to MINIMAL_RUNTIME changes a filename, that would be more disruptive in the long term.


if memfile and not shared.Settings.WASM_BACKEND:
# Strip the memory initializer out of the asmjs file
Expand Down Expand Up @@ -2109,6 +2144,15 @@ def get_eliminate():

module_export_name_substitution()

# Run a final regex pass to clean up items that were not possible to optimize by Closure, or unoptimalities that were left behind
# by processing steps that occurred after Closure.
if shared.Settings.MINIMAL_RUNTIME == 2 and shared.Settings.USE_CLOSURE_COMPILER and options.debug_level == 0:
# Process .js runtime file
shared.run_process([shared.PYTHON, shared.path_from_root('tools', 'hacky_postprocess_around_closure_limitations.py'), final])
# Process .asm.js file
if not shared.Settings.WASM:
shared.run_process([shared.PYTHON, shared.path_from_root('tools', 'hacky_postprocess_around_closure_limitations.py'), asm_target])

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are big hacks, but useful since Closure has limits on what it can do. They make the minimal console log application considerably smaller, so there is benefit.

# The JS is now final. Move it to its final location
shutil.move(final, js_target)

Expand Down Expand Up @@ -2690,15 +2734,39 @@ def module_export_name_substitution():
logger.debug('Private module export name substitution with ' + shared.Settings.EXPORT_NAME)
src = open(final).read()
final = final + '.module_export_name_substitution.js'
replacement = "typeof %(EXPORT_NAME)s !== 'undefined' ? %(EXPORT_NAME)s : {}" % {"EXPORT_NAME": shared.Settings.EXPORT_NAME}
if shared.Settings.MINIMAL_RUNTIME:
# In MINIMAL_RUNTIME the Module object is always present to provide the .asm.js/.wasm content
replacement = shared.Settings.EXPORT_NAME
else:
replacement = "typeof %(EXPORT_NAME)s !== 'undefined' ? %(EXPORT_NAME)s : {}" % {"EXPORT_NAME": shared.Settings.EXPORT_NAME}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In MINIMAL_RUNTIME, the Module object is always present to provide the .asm.js/.wasm content, so pulling it in is simpler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add that in a comment in the source.

with open(final, 'w') as f:
f.write(src.replace(shared.JS.module_export_name_substitution_pattern, replacement))
src = src.replace(shared.JS.module_export_name_substitution_pattern, replacement)
# For Node.js, create an unminified Module object so that loading external .asm.js file that assigns to Module['asm'] works
# even when Closure is used.
if shared.Settings.MINIMAL_RUNTIME and shared.Settings.target_environment_may_be('node'):
src = 'if(typeof process!=="undefined"){var Module={};}' + src
f.write(src)
save_intermediate('module_export_name_substitution')


def generate_minimal_runtime_html(target, options, js_target, target_basename,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this needed? doesn't the other code path emit the same (all the emtepreter etc. code paths are skipped)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default path requires the presence of {{{ SCRIPT }}} in the shell file, which minimal runtime shell file does not have. It also appends PROXY_TO_WORKER, and injects things like Emterpreter, Wasm Memory initializer, inline separate-asm, precise_f32=2 script replace mods, inject arrayutils.js, base64utils.js, uriutils.js, do single_file, etc. none of which are items that are supported in MINIMAL_RUNTIME, so it seems cleanest to use a different function, rather than go through all of that and wrangle build flags and options so that none of the cases would be triggered.

asm_target, wasm_binary_target,
memfile, optimizer):
logger.debug('generating HTML for minimal runtime')
shell = read_and_preprocess(options.shell_path)
html_contents = shell.replace('{{{ TARGET_BASENAME }}}', target_basename)
html_contents = tools.line_endings.convert_line_endings(html_contents, '\n', options.output_eol)
with open(target, 'wb') as f:
f.write(asbytes(html_contents))


def generate_html(target, options, js_target, target_basename,
asm_target, wasm_binary_target,
memfile, optimizer):
if shared.Settings.MINIMAL_RUNTIME:
return generate_minimal_runtime_html(target, options, js_target, target_basename, asm_target,
wasm_binary_target, memfile, optimizer)

script = ScriptSource()

logger.debug('generating HTML')
Expand Down
Loading