Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use official emsdk bazel toolchain #4769

Merged
merged 30 commits into from
Mar 30, 2021
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
53ca3fe
Compile a simple hello world example using emsdk's bazel toolchain
mattsoulanille Feb 25, 2021
35761df
xnnpack hello world not working with emsdk toolchain
mattsoulanille Feb 25, 2021
2e719ca
Remove unused variables
mattsoulanille Feb 26, 2021
9785982
WASM simd compiling locally with -copt='-msimd128'
mattsoulanille Mar 1, 2021
e9530cf
Add msimd128 copt when simd is true
mattsoulanille Mar 1, 2021
6438cd4
Compile wasm bundles from the same cc target
mattsoulanille Mar 1, 2021
602e2c5
Use a remote repo instead of local for patched XNNPACK
mattsoulanille Mar 1, 2021
b434002
Enable incompatible_strict_action_env
mattsoulanille Mar 2, 2021
3eb59d4
Enable remote cache in CI
mattsoulanille Mar 2, 2021
ee64690
Cleanup and format
mattsoulanille Mar 2, 2021
927ec38
Apply buildifier lints
mattsoulanille Mar 2, 2021
30b0ac9
Remove hello world test cc files
mattsoulanille Mar 2, 2021
f9dc556
Remove old emscripten toolchain
mattsoulanille Mar 2, 2021
d2ae7b6
Update dockerfiles
mattsoulanille Mar 2, 2021
a81863f
Add trailing slash to shell command
mattsoulanille Mar 2, 2021
f643328
Revert "Add trailing slash to shell command"
mattsoulanille Mar 2, 2021
7dfdb99
Revert "Update dockerfiles"
mattsoulanille Mar 2, 2021
98ec22a
Merge branch 'master' into wasm_toolchain_2.0.14
mattsoulanille Mar 3, 2021
e09b523
Use local spawn strategy for bazel build performance
mattsoulanille Mar 3, 2021
a01ee3c
Update XNNPACK
mattsoulanille Mar 4, 2021
fab5bc6
Merge branch 'master' into wasm_toolchain_2.0.14
mattsoulanille Mar 4, 2021
95e0a46
Stop accidentally building twice
mattsoulanille Mar 4, 2021
d81c077
Merge remote-tracking branch 'upstream/master' into wasm_toolchain_2.…
mattsoulanille Mar 9, 2021
ca4f269
'remote_http_cache' -> 'remote_cache'
mattsoulanille Mar 9, 2021
5ef1bc8
Merge branch 'master' into wasm_toolchain_2.0.14
mattsoulanille Mar 9, 2021
5a4f3bc
Fix typo
mattsoulanille Mar 10, 2021
df6f4db
Merge branch 'wasm_toolchain_2.0.14' of github.com:mattsoulanille/tfj…
mattsoulanille Mar 10, 2021
4c19957
Update emscripten toolchain
mattsoulanille Mar 29, 2021
dc69450
Merge branch 'master' into wasm_toolchain_2.0.14
mattsoulanille Mar 29, 2021
dde7a90
Remove unused variables
mattsoulanille Mar 29, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 28 additions & 15 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,35 @@
# editor's search path.
build --symlink_prefix=dist/

# Use our custom-configured c++ toolchain.
build:wasm --crosstool_top=//toolchain:emscripten
# These compile flags are active no matter which build mode we are in
# (dbg vs opt). For flags specific to build mode, see cc_toolchain_config.bzl.
build --cxxopt="-std=c++11"
build --cxxopt="-fno-rtti"
build --cxxopt="-fno-exceptions"
build --cxxopt="-fomit-frame-pointer"

# Use --cpu as a differentiator.
build:wasm --cpu=wasm
# Remote cache config. Users can add credentials in their .bazelrc.user files.
build:remote --remote_cache=https://storage.googleapis.com/bazel-remote-cache-tfjs

# Use the default C++ toolchain to build the tools used during the build.
build:wasm --host_crosstool_top=@bazel_tools//tools/cpp:toolchain
# Config for Google Cloud continuous integration that uses default credentials.
build:ci --config=remote
build:ci --google_default_credentials

# These compile flags are active no matter which build mode we are in
# (dbg vs opt). For flags specific to build mode, see cc_toolchain_config.bzl.
build:wasm --cxxopt="-std=c++11"
build:wasm --cxxopt="-fno-rtti"
build:wasm --cxxopt="-fno-exceptions"
build:wasm --cxxopt="-fomit-frame-pointer"
# This flag is needed to so that the bazel cache is not invalidated
# when running bazel via `yarn bazel`.
# See https://github.com/angular/angular/issues/27514.
build --incompatible_strict_action_env
run --incompatible_strict_action_env

# Don't use a sandboxed build since it hurts performance.
# When sandboxed, the wasm backend takes hours to build on a 32 core machine.
build --spawn_strategy=worker,local

# Disable sandbox environment because emsdk caches files by writing to
# home directory.
build:wasm --spawn_strategy=local
# Load any settings specific to the current user.
# .bazelrc.user should appear in .gitignore so that settings are not shared with
# team members. This needs to be last statement in this config, as the user
# configuration should be able to overwrite flags from this file.
# See https://docs.bazel.build/versions/master/best-practices.html#bazelrc
# (Note that we use .bazelrc.user so the file appears next to .bazelrc in
# directory listing, rather than user.bazelrc as suggested in the Bazel docs).
try-import %workspace%/.bazelrc.user
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,7 @@ tfjs-backend-wasm/wasm-out/*.js
tfjs-backend-wasm/wasm-out/*.wasm
yalc.lock
yarn-error.log
cloudbuild_generated.yml
cloudbuild_generated.yml

# User-specific .bazelrc
.bazelrc.user
27 changes: 21 additions & 6 deletions WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,33 @@ yarn_install(
yarn_lock = "//:yarn.lock",
)

# Make all files under $HOME/emsdk/* visible to the toolchain. The files are
# available as external/emsdk/emsdk/*
load("//toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")
# npm_install used for emscripten dependencies
load("@build_bazel_rules_nodejs//:index.bzl", "npm_install")

# emscripten 2.0.14
http_archive(
name = "emscripten",
sha256 = "e466cd47ddd4bf0acd645412fdf08eda6d232484e48e5a2643e08062a7a4cf56",
strip_prefix = "install",
url = "https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/fc5562126762ab26c4757147a3b4c24e85a7289e/wasm-binaries.tbz2",
build_file = "//emscripten_toolchain:emscripten.BUILD",
type = "tar.bz2",
)

# Install emscripten dependencies
npm_install(
name = "emsdk_npm",
package_json = "@emscripten//:emscripten/package.json",
package_lock_json = "@emscripten//:emscripten/package-lock.json",
)

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
# xnnpack used for fast vectorized wasm operations
git_repository(
name = "xnnpack",
commit = "55d53a4e7079d38e90acd75dd9e4f9e781d2da35",
commit = "f69ac5d218a4f2a68bc2981d61603d43966e8068",
remote = "https://github.com/google/XNNPACK.git",
shallow_since = "1614036677 -0800",
shallow_since = "1614818900 -0800",
)

# The libraries below are transitive dependencies of XNNPACK that we need to
Expand Down
80 changes: 80 additions & 0 deletions emscripten_toolchain/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
load(":crosstool.bzl", "emscripten_cc_toolchain_config_rule")

package(default_visibility = ["//visibility:public"])

filegroup(
name = "common-script-includes",
srcs = [
"emar.sh",
"emcc.sh",
"emscripten_config",
"env.sh",
"@emscripten//:all",
"@nodejs//:node_files",
"@emsdk_npm//:node_modules",
],
)

filegroup(
name = "compile-emscripten",
srcs = [":common-script-includes"],
)

filegroup(
name = "link-emscripten",
srcs = [
"emcc_link.sh",
"link_wrapper.py",
":common-script-includes",
"@emscripten//:all",
"@nodejs//:node_files",
],
)

filegroup(
name = "every-file",
srcs = [
":compile-emscripten",
":link-emscripten",
"@emscripten//:all",
"@nodejs//:node_files",
],
)

filegroup(name = "empty")

# dlmalloc.bc is implictly added by the emscripten toolchain
cc_library(name = "malloc")

emscripten_cc_toolchain_config_rule(
name = "wasm",
cpu = "wasm",
emscripten_version = "emscripten",
)

cc_toolchain(
name = "cc-compiler-wasm",
all_files = ":every-file",
ar_files = ":common-script-includes",
as_files = ":empty",
compiler_files = ":compile-emscripten",
dwp_files = ":empty",
linker_files = ":link-emscripten",
objcopy_files = ":empty",
strip_files = ":empty",
toolchain_config = "wasm",
toolchain_identifier = "emscripten-wasm",
)

cc_toolchain_suite(
name = "everything",
toolchains = {
"wasm": ":cc-compiler-wasm",
"wasm|emscripten": ":cc-compiler-wasm",
},
)

py_binary(
name = "wasm_binary",
srcs = ["wasm_binary.py"],
)
Loading