Out of memory building didkit-asm-loader

[clehner]

wasm2js completes, but webpack takes several minutes to run and then exits due to out of memory.

$ make -C lib ../target/test/asmjs.stamp
make: Entering directory '/home/cel/src/didkit/lib'
cd wasm && wasm-pack build --target bundler
[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
    Updating git repository `https://github.com/timothee-haudebourg/json-rust`
    Finished release [optimized] target(s) in 5.67s
[INFO]: Installing wasm-bindgen...
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: :-) Done in 6.04s
[INFO]: :-) Your wasm pkg is ready to publish at /home/cel/src/didkit/lib/wasm/pkg.
cd wasm/pkg && PATH="$PATH:"/home/cel/binaryen"/bin" wasm2js --pedantic -o didkit_wasm_bg1.js didkit_wasm_bg.wasm
cd wasm/asm && ./repack.sh
npm --prefix wasm/asm install
npm WARN didkit-asm-loader@0.1.0 No description
npm WARN didkit-asm-loader@0.1.0 No repository field.
npm WARN didkit-asm-loader@0.1.0 No license field.

added 124 packages from 158 contributors in 32.962s

14 packages are looking for funding
  run `npm fund` for details

npm --prefix wasm/asm run build

> didkit-asm-loader@0.1.0 build /home/cel/src/didkit/lib/wasm/asm
> webpack


events.js:291
      throw er; // Unhandled 'error' event
      ^

Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
    at Worker.[kOnExit] (internal/worker.js:229:26)
    at Worker.<computed>.onexit (internal/worker.js:165:20)
Emitted 'error' event on Worker instance at:
    at Worker.[kOnExit] (internal/worker.js:229:12)
    at Worker.<computed>.onexit (internal/worker.js:165:20) {
  code: 'ERR_WORKER_OUT_OF_MEMORY'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! didkit-asm-loader@0.1.0 build: `webpack`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the didkit-asm-loader@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/cel/.npm/_logs/2021-03-02T03_20_38_588Z-debug.log
make: *** [Makefile:174: wasm/asm/didkit-asm.min.js] Error 1
make: Leaving directory '/home/cel/src/didkit/lib'

[w4ll3]

I think we can replace the webpack for something else, currently it is just being used to minify the generated script, but previously it was to remove an circular dependency.

[bumblefudge]

Hitting on the same error on my WSL2 test on a machine with 8MB (although it might just be the VM settings). Is there a way to triage this by somehow requesting extra memory for the process?

[maelswarm]

Hitting on the same error on my WSL2 test on a machine with 8MB (although it might just be the VM settings). Is there a way to triage this by somehow requesting extra memory for the process?

Unfortunately, downgrading to WSL1 or using an actual UNIX env tends to solve these sort of headaches.
I've found WSL2 to be unreliable all around. Just my two cents.

[sbihel]

I suggest to close this issue as we now use the output of wasm-pack directly.

[clehner]

I still get out of memory when building asmjs.stamp on 1c5b6af:

make -C lib ../target/test/asmjs.stamp

$ make -C lib ../target/test/asmjs.stamp
make: Entering directory '/home/cel/src/didkit/lib'
cd web && wasm-pack build --target bundler --out-dir pkg/asmjs
[INFO]: Checking for the Wasm target...
[INFO]: Compiling to Wasm...
warning: /home/cel/src/didkit/lib/Cargo.toml: only one of `license` or `license-file` is necessary
warning: /home/cel/src/didkit/http/Cargo.toml: only one of `license` or `license-file` is necessary
warning: /home/cel/src/didkit/cli/Cargo.toml: only one of `license` or `license-file` is necessary
   Compiling proc-macro2 v1.0.27
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.72
   Compiling version_check v0.9.3
   Compiling serde v1.0.126
   Compiling serde_derive v1.0.126
   Compiling ryu v1.0.5
   Compiling cfg-if v1.0.0
   Compiling log v0.4.14
   Compiling serde_json v1.0.64
   Compiling autocfg v1.0.1
   Compiling itoa v0.4.7
   Compiling wasm-bindgen-shared v0.2.74
   Compiling spin v0.5.2
   Compiling bumpalo v3.6.1
   Compiling wasm-bindgen v0.2.74
   Compiling typenum v1.13.0
   Compiling libc v0.2.94
   Compiling subtle v2.4.0
   Compiling byteorder v1.4.3
   Compiling getrandom v0.1.16
   Compiling radium v0.6.2
   Compiling futures-core v0.3.15
   Compiling block-padding v0.2.1
   Compiling ppv-lite86 v0.2.10
   Compiling memchr v2.4.0
   Compiling opaque-debug v0.3.0
   Compiling futures-channel v0.3.15
   Compiling crunchy v0.2.2
   Compiling funty v1.1.0
   Compiling wyz v0.2.0
   Compiling tap v1.0.1
   Compiling const-oid v0.5.2
   Compiling proc-macro-hack v0.5.19
   Compiling matches v0.1.8
   Compiling futures-sink v0.3.15
   Compiling ident_case v1.0.1
   Compiling strsim v0.9.3
   Compiling proc-macro-nested v0.1.7
   Compiling fnv v1.0.7
   Compiling futures-task v0.3.15
   Compiling byte-tools v0.3.1
   Compiling futures-io v0.3.15
   Compiling pin-project-lite v0.2.6
   Compiling pin-utils v0.1.0
   Compiling static_assertions v1.1.0
   Compiling libm v0.2.1
   Compiling slab v0.4.3
   Compiling ucd-trie v0.1.3
   Compiling autocfg v0.1.7
   Compiling percent-encoding v2.1.0
   Compiling fake-simd v0.1.2
   Compiling maplit v1.0.2
   Compiling cache-padded v1.1.1
   Compiling opaque-debug v0.2.3
   Compiling tiny-keccak v2.0.2
   Compiling event-listener v2.5.1
   Compiling data-encoding v2.3.2
   Compiling smallvec v1.6.1
   Compiling ryu-js v0.2.1
   Compiling once_cell v1.7.2
   Compiling regex-syntax v0.6.25
   Compiling hex v0.4.3
   Compiling arrayref v0.3.6
   Compiling async-trait v0.1.50
   Compiling derive_builder v0.9.0
   Compiling keccak v0.1.0
   Compiling base64 v0.13.0
   Compiling base64 v0.12.3
   Compiling constant_time_eq v0.1.5
   Compiling arrayvec v0.5.2
   Compiling pct-str v1.1.0
   Compiling base-x v0.2.8
   Compiling mown v0.2.1
   Compiling langtag v0.2.0
   Compiling combination v0.1.5
   Compiling json v0.12.4
   Compiling tinyvec_macros v0.1.0
   Compiling ssi-contexts v0.1.0 (/home/cel/src/ssi/contexts)
   Compiling bytes v1.0.1
   Compiling bitflags v1.2.1
   Compiling unicode-width v0.1.8
   Compiling unicode-segmentation v1.7.1
   Compiling vec_map v0.8.2
   Compiling strsim v0.8.0
   Compiling subtle v1.0.0
   Compiling cc v1.0.67
   Compiling ansi_term v0.11.0
   Compiling remove_dir_all v0.5.3
   Compiling cbindgen v0.14.3
   Compiling same-file v1.0.6
   Compiling adler v1.0.2
   Compiling gimli v0.24.0
   Compiling anyhow v1.0.40
   Compiling object v0.24.0
   Compiling rustc-demangle v0.1.19
   Compiling jni-sys v0.3.0
   Compiling cesu8 v1.1.0
   Compiling value-bag v1.0.0-alpha.7
   Compiling generic-array v0.14.4
   Compiling error-chain v0.12.4
   Compiling lazy_static v1.4.0
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling futures-macro v0.3.15
   Compiling num-iter v0.1.42
   Compiling num-bigint v0.2.6
   Compiling futures-util v0.3.15
   Compiling crossbeam-utils v0.8.4
   Compiling num-bigint v0.4.0
   Compiling miniz_oxide v0.4.4
   Compiling unicode-bidi v0.3.5
   Compiling block-padding v0.1.5
   Compiling fixed-hash v0.7.0
   Compiling pest v2.1.3
   Compiling form_urlencoded v1.0.1
   Compiling concurrent-queue v1.2.2
   Compiling num-bigint-dig v0.6.1
   Compiling async-lock v2.4.0
   Compiling iref v1.4.3
   Compiling regex v1.5.4
   Compiling blake2b_simd v0.5.11
   Compiling tinyvec v1.2.0
   Compiling http v0.2.4
   Compiling textwrap v0.11.0
   Compiling heck v0.3.2
   Compiling walkdir v2.3.2
   Compiling backtrace v0.3.59
   Compiling addr2line v0.15.1
   Compiling pest_meta v2.1.3
   Compiling pem v0.8.3
   Compiling unicode-normalization v0.1.17
   Compiling jni v0.17.0
   Compiling quote v1.0.9
   Compiling time v0.1.43
   Compiling num_cpus v1.13.0
   Compiling async-channel v1.6.1
   Compiling bitvec v0.20.4
   Compiling uint v0.9.0
   Compiling combine v4.5.2
   Compiling getrandom v0.2.3
   Compiling atty v0.2.14
   Compiling generic-array v0.12.4
   Compiling der v0.3.5
   Compiling idna v0.2.3
   Compiling primitive-types v0.9.0
   Compiling clap v2.33.3
   Compiling rand_core v0.6.2
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling crypto-mac v0.11.0
   Compiling digest v0.8.1
   Compiling block-buffer v0.7.3
   Compiling crypto-mac v0.7.0
   Compiling spki v0.3.0
   Compiling url v2.2.2
   Compiling keccak-hash v0.7.0
   Compiling rand_chacha v0.3.0
   Compiling sha2 v0.9.5
   Compiling sha3 v0.9.1
   Compiling ripemd160 v0.9.1
   Compiling hmac v0.11.0
   Compiling synstructure v0.12.4
   Compiling darling_core v0.10.2
   Compiling pest_generator v2.1.3
   Compiling sha2 v0.8.2
   Compiling hmac v0.7.1
   Compiling pkcs8 v0.6.1
   Compiling rand v0.8.3
   Compiling bs58 v0.4.0
   Compiling ctor v0.1.20
   Compiling thiserror-impl v1.0.25
   Compiling data-encoding-macro-internal v0.1.9
   Compiling async-attributes v1.1.2
   Compiling sshkeys v0.3.1
   Compiling zeroize_derive v1.1.0
   Compiling hmac-drbg v0.2.0
   Compiling pest_derive v2.1.0
   Compiling tempfile v3.2.0
   Compiling darling_macro v0.10.2
   Compiling data-encoding-macro v0.1.10
   Compiling thiserror v1.0.25
   Compiling zeroize v1.3.0
   Compiling multibase v0.8.0
   Compiling kv-log-macro v1.0.7
   Compiling wasm-bindgen-backend v0.2.74
   Compiling darling v0.10.2
   Compiling derive_builder_core v0.9.0
   Compiling wasm-bindgen-macro-support v0.2.74
   Compiling futures-executor v0.3.15
   Compiling futures v0.3.15
   Compiling wasm-bindgen-macro v0.2.74
   Compiling serde_urlencoded v0.7.0
   Compiling toml v0.5.8
   Compiling serde_jcs v0.1.0
   Compiling treediff v3.0.2
   Compiling json-patch v0.2.6
   Compiling js-sys v0.3.51
   Compiling web-sys v0.3.51
   Compiling chrono v0.4.19
   Compiling wasm-bindgen-futures v0.4.24
   Compiling rand_core v0.5.1
   Compiling ff v0.9.0
   Compiling signature v1.3.0
   Compiling simple_asn1 v0.4.1
   Compiling simple_asn1 v0.5.3
   Compiling rand_chacha v0.2.2
   Compiling curve25519-dalek v3.1.0
   Compiling group v0.9.0
   Compiling ed25519 v1.1.1
   Compiling gloo-timers v0.2.1
   Compiling reqwest v0.11.3
   Compiling rand v0.7.3
   Compiling elliptic-curve v0.9.12
   Compiling async-std v1.9.0
   Compiling ed25519-dalek v1.0.1
   Compiling libsecp256k1 v0.3.5
   Compiling didkit-cbindings v0.1.0 (/home/cel/src/didkit/lib/cbindings)
   Compiling ecdsa v0.11.1
   Compiling rsa v0.3.0
   Compiling p256 v0.8.1
   Compiling k256 v0.8.1
   Compiling ssi v0.2.2 (/home/cel/src/ssi)
warning: value assigned to `common_lang_dir` is never read
  --> /home/cel/src/ssi/src/../json-ld/src/compaction/iri.rs:96:9
   |
96 | ...                   common_lang_dir = Some(Nullable::Some((active_context.default_language(), active_context.default_base_direction())))
   |                       ^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_assignments)]` on by default
   = help: maybe it is overwritten before being read?

warning: constant is never used: `EDPK_PREFIX`
 --> /home/cel/src/ssi/src/ldp.rs:9:1
  |
9 | const EDPK_PREFIX: [u8; 4] = [13, 15, 37, 217];
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

warning: associated function is never used: `is_null`
  --> /home/cel/src/ssi/src/../json-ld/src/null.rs:22:9
   |
22 |     pub fn is_null(&self) -> bool {
   |            ^^^^^^^

warning: associated function is never used: `is_some`
  --> /home/cel/src/ssi/src/../json-ld/src/null.rs:31:9
   |
31 |     pub fn is_some(&self) -> bool {
   |            ^^^^^^^

warning: associated function is never used: `unwrap`
  --> /home/cel/src/ssi/src/../json-ld/src/null.rs:42:9
   |
42 |     pub fn unwrap(self) -> T {
   |            ^^^^^^

warning: associated function is never used: `new`
  --> /home/cel/src/ssi/src/../json-ld/src/loader.rs:82:9
   |
82 |     pub fn new() -> FsLoader {
   |            ^^^

warning: associated function is never used: `mount`
  --> /home/cel/src/ssi/src/../json-ld/src/loader.rs:89:9
   |
89 |     pub fn mount<P: AsRef<Path>>(&mut self, url: Iri, path: P) {
   |            ^^^^^

warning: associated function is never used: `from`
   --> /home/cel/src/ssi/src/../json-ld/src/syntax/container.rs:111:9
    |
111 |     pub fn from<'a, I: IntoIterator<Item = &'a ContainerType>>(iter: I) -> Result<Container, ContainerType> {
    |            ^^^^

warning: associated function is never used: `iter`
   --> /home/cel/src/ssi/src/../json-ld/src/syntax/container.rs:145:9
    |
145 |     pub fn iter(&self) -> impl Iterator<Item = &ContainerType> {
    |            ^^^^

warning: associated function is never used: `len`
   --> /home/cel/src/ssi/src/../json-ld/src/syntax/container.rs:149:9
    |
149 |     pub fn len(&self) -> usize {
    |            ^^^

warning: associated function is never used: `is_empty`
   --> /home/cel/src/ssi/src/../json-ld/src/syntax/container.rs:153:9
    |
153 |     pub fn is_empty(&self) -> bool {
    |            ^^^^^^^^

warning: associated function is never used: `new`
  --> /home/cel/src/ssi/src/../json-ld/src/context/loader.rs:11:9
   |
11 |     pub fn new(url: Iri, context: C) -> RemoteContext<C> {
   |            ^^^

warning: associated function is never used: `get_mut`
   --> /home/cel/src/ssi/src/../json-ld/src/context/inverse.rs:322:9
    |
322 |     pub fn get_mut(&mut self, term: &Term<T>) -> Option<&mut InverseDefinition<T>> {
    |            ^^^^^^^

warning: associated function is never used: `select`
   --> /home/cel/src/ssi/src/../json-ld/src/context/inverse.rs:333:9
    |
333 |     pub fn select(&self, var: &Term<T>, containers: &[Container], selection: &Selection<T>) -> Option<&str> {
    |            ^^^^^^

warning: associated function is never used: `owned`
   --> /home/cel/src/ssi/src/../json-ld/src/context/mod.rs:225:9
    |
225 |     pub fn owned(self) -> Processed<L, C> {
    |            ^^^^^

warning: function is never used: `json_ld_eq`
  --> /home/cel/src/ssi/src/../json-ld/src/util/json.rs:76:8
   |
76 | pub fn json_ld_eq(a: &JsonValue, b: &JsonValue) -> bool {
   |        ^^^^^^^^^^

   Compiling did-tz v0.1.0 (/home/cel/src/ssi/did-tezos)
   Compiling did-pkh v0.0.1 (/home/cel/src/ssi/did-pkh)
   Compiling did-web v0.1.0 (/home/cel/src/ssi/did-web)
   Compiling did-webkey v0.1.0 (/home/cel/src/ssi/did-webkey)
warning: unused import: `chrono::prelude::*`
 --> /home/cel/src/ssi/did-pkh/src/lib.rs:2:5
  |
2 | use chrono::prelude::*;
  |     ^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `chrono::prelude::*`
  --> /home/cel/src/ssi/did-tezos/src/lib.rs:20:5
   |
20 | use chrono::prelude::*;
   |     ^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

   Compiling did-sol v0.0.1 (/home/cel/src/ssi/did-sol)
warning: unused import: `chrono::prelude::*`
 --> /home/cel/src/ssi/did-sol/src/lib.rs:2:5
  |
2 | use chrono::prelude::*;
  |     ^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

   Compiling did-method-key v0.1.1 (/home/cel/src/ssi/did-key)
warning: 1 warning emitted

   Compiling did-onion v0.1.0 (/home/cel/src/ssi/did-onion)
warning: variable does not need to be mutable
   --> /home/cel/src/ssi/did-onion/src/lib.rs:109:13
    |
109 |         let mut client_builder = reqwest::Client::builder().default_headers(headers);
    |             ----^^^^^^^^^^^^^^
    |             |
    |             help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

   Compiling did-ethr v0.0.1 (/home/cel/src/ssi/did-ethr)
warning: unused import: `chrono::prelude::*`
 --> /home/cel/src/ssi/did-ethr/src/lib.rs:2:5
  |
2 | use chrono::prelude::*;
  |     ^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

warning: 1 warning emitted

warning: 1 warning emitted

warning: 16 warnings emitted

   Compiling didkit v0.2.1 (/home/cel/src/didkit/lib)
warning: unused variable: `sock_path`
   --> lib/src/lib.rs:126:14
    |
126 |         Some(sock_path) => {
    |              ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_sock_path`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: 1 warning emitted

   Compiling didkit-wasm v0.1.0 (/home/cel/src/didkit/lib/web)
    Finished release [optimized] target(s) in 1m 22s
[INFO]: Installing wasm-bindgen...
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: :-) Done in 1m 22s
[INFO]: :-) Your wasm pkg is ready to publish at /home/cel/src/didkit/lib/web/pkg/asmjs.
:-) [WARN]: There's a newer version of wasm-pack available, the new version is: 0.10.0, you are using: 0.9.1. To update, navigate to: https://rustwasm.github.io/wasm-pack/installer/
cd web/pkg/asmjs && "/home/cel/binaryen"/bin/wasm2js --pedantic -o didkit_wasm_bg1.js didkit_wasm_bg.wasm
cd web/loader/asmjs && ./repack.sh
npm --prefix web/loader/asmjs install

up to date, audited 125 packages in 2s

1 moderate severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
npm --prefix web/loader/asmjs run build

> didkit-asm-loader@0.1.0 build
> webpack


<--- Last few GCs --->

[115548:0x59c70c0]    11342 ms: Mark-sweep (reduce) 2047.3 (2051.6) -> 2046.8 (2054.6) MB, 810.1 / 0.0 ms  (+ 1.7 ms in 341 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 932 ms) (average mu = 0.309, current mu = 0.236)[115548:0x59c70c0]    12927 ms: Mark-sweep (reduce) 2048.0 (2055.1) -> 2047.7 (2055.9) MB, 1583.1 / 0.0 ms  (+ 0.1 ms in 17 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1585 ms) (average mu = 0.129, current mu = 0.001

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa04200 node::Abort() [webpack]
 2: 0x94e4e9 node::FatalError(char const*, char const*) [webpack]
 3: 0xb7978e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [webpack]
 4: 0xb79b07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [webpack]
 5: 0xd34395  [webpack]
 6: 0xd34f1f  [webpack]
 7: 0xd42fab v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [webpack]
 8: 0xd45665 v8::internal::Heap::HandleGCRequest() [webpack]
 9: 0xcebca7 v8::internal::StackGuard::HandleInterrupts() [webpack]
10: 0x105ae9a v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [webpack]
11: 0x1401219  [webpack]
Aborted
make: *** [Makefile:193: ../target/test/asmjs.stamp] Error 134
make: Leaving directory '/home/cel/src/didkit/lib'

$ node --version
v14.16.1
$ npm --version
7.8.0

[sbihel]

Can I suggest we remove it from the Makefile? I think maybe handling old browsers (which is the only reason why we would need ASM?) shouldn't be in DIDKit, but instead should be an option when compiling the frontend (which could be fully in WASM, meaning not just the DIDKit dependency).