Update ethereumjs-util to v7.0.1

[holgerd77]

@alcuadrado having problems running the test locally (on Node v12.15.0) getting

"Memory limit exceeded for Node's built-in crypto.scrypt, falling back to scryptsy (times: 1), if this happens frequently you can improve the performance of scrypt when running Node.js versions older than 12.0.0 by installing the (deprecated) scrypt package in your project"

This occurs after the

.toV3()
    ✓ should work with PBKDF2 (2463ms)

test execution.

Any idea what this might hint to? I made a clean install with package-lock.json deleted.

This does occur along the comparison with Ethers wallet results, so might also be some problem on the integrated Ethers version? //cc @ricmoo

[cgewecke]

@holgerd77

The warning is related to the change in #108 - it's logged when encountering the memory issue described in this PR comment.

[alcuadrado]

My understanding is that the Ethereum ecosystem can't use the native PBKDF2 implementation because it was previously using a non-standard implementation. Is this still the case, @cgewecke ?

[holgerd77]

@cgewecke Ah, thanks, first thought that CI failing here would also be due to the repeated warning, but CI runs through and the one test failing seems unrelated.

Not sure if CI got slower after the merge of #108, there might be some tendency but also hard to read from the CI run times. Wonder if we should test this more thoroughly before we release.

[holgerd77]

Would be proceeding with axic/scrypt.js#8 a more robust respectively better solution? Then I would try to knock at @axic 's door again on this.

[alcuadrado]

I think that suffers from the problem I described. When I created that PR I wasn't aware of it.

[cgewecke]

My understanding is that the Ethereum ecosystem can't use the native PBKDF2 implementation because it was previously using a non-standard implementation. Is this still the case?

@alcuadrado I am not certain...read through this stuff again and last week someone in the Node issue reported they've filed errata in the scrypt RFC, implying that the native impl. needs a fix.

This looks like it's up in the air a little? There's a fix for the memory issue @michaelsbradleyjr was ambivalent about because of the RFC problem which may be viable again.

(Will ping him on Gitter asking for advice)

[coveralls]

Coverage decreased (-0.3%) to 86.513% when pulling 073319c on update-ethereumjs-util-to-v700 into b7574b1 on master.

[holgerd77]

Some updates:

Node Tests
Have now fixed the one failing test in node (by updating an error message which changed in the new v4 secp2561 library) and tests for node now pass.

Coveralls Failure
Please ignore, I don't have access to the coveralls settings and add a failure decrease threshold, see also this thread that settings on coveralls can only be accessed by people who added the repo. This might be updated eventually by @axic if he reads this (then an update on the repo root directory so that we can see source comparisons - see e.g. for hdkey.ts file on coveralls - would also be nice).

So for now we just need to look over this.

Browser Tests
Browser tests did at least fail on the first run (will re-trigger, since these seems to work locally for me) with the following error:

Firefox 74.0 (Ubuntu 0.0.0) .fromExtendedKey() should work with public FAILED
	r._strip is not a function
	ireduce@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:12471:9
	imod@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:12643:39
	mul@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:12709:17
	sqr@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:12717:17
	redSqr@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:4632:21
	loadCompressedPublicKey@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:14939:13
	loadPublicKey@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:14971:14
	global.wrappers["/home/runner/work/ethereumjs-wallet/ethereumjs-wallet/node_modules/hdkey/node_modules/secp256k1/lib/elliptic/index.js"]</exports.publicKeyVerify@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:15042:10
	publicKeyVerify@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:30494:24
	set@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:30702:22
	global.wrappers["/home/runner/work/ethereumjs-wallet/ethereumjs-wallet/node_modules/hdkey/lib/hdkey.js"]</HDKey.fromExtendedKey@/tmp/karma-typescript-bundle-3248fRnuQwxF00sj.js:30861:5
	global.wrappers["/home/runner/work/ethereumjs-wallet/ethereumjs-wallet/src/hdkey.ts"]</EthereumHDKey</EthereumHDKey.fromExtendedKey@src/hdkey.ts:20:36 <- src/hdkey.js:41:52
	global.wrappers["/home/runner/work/ethereumjs-wallet/ethereumjs-wallet/test/hdkey.spec.ts"]</</<@test/hdkey.spec.ts:39:34 <- test/hdkey.spec.js:27:38

Not sure yet if this is a caching problem or something more substantial. This likely has something to do with updated secp2561 version, please also be aware (if error persists) that there are different stack traces occuring, but all with the r._strip is not a function error.

[holgerd77]

Ok, tests are still failing here.

Some observations / further context:
I could now also reproduce the tests to fail locally (yeah, progress! 😜) by doing a clean install of the dependencies.

Before (when tests were passing) I just updated by npm install ethereumjs-util which updated util to v7 but kept the secp2561 version in node_modules/secp2561 on 3.8.

After the complete reinstall with then failing browser tests node_modules/secp2561 version now is on 4.0.

🤔

[holgerd77]

The r._strip method is from this code part in the generated Karma bundle file:

  MPrime.prototype.ireduce = function ireduce (num) {
    // Assumes that `num` is less than `P^2`
    // num = HI * (2 ^ N - K) + HI * K + LO = HI * K + LO (mod P)
    var r = num;
    var rlen;

    do {
      this.split(r, this.tmp);
      r = this.imulK(r);
      r = r.iadd(this.tmp);
      rlen = r.bitLength();
    } while (rlen > this.n);

    var cmp = rlen < this.n ? -1 : r.ucmp(this.p);
    if (cmp === 0) {
      r.words[0] = 0;
      r.length = 1;
    } else if (cmp > 0) {
      r.isub(this.p);
    } else {
      r._strip();
    }

    return r;
  };

[holgerd77]

Running a test run to see if downgrading the BN.js version to a standalong v4.11.8 version will solve the issue at #122

[holgerd77]

Ah, this (downgrading BN.js) would actually solve our problem here. I would suggest that we consider merging this in and doing the release with this fix, eventually also depending how quickly it might be possible to integrate a fix on the BN.js side.

But we generally can remove the dependency again along some subsequent patch release.

[holgerd77]

@ryanio thanks for the thumbs up, also still think this is a viable way to go (merge #122 and continue here) but let's give this 2-3 days more and see how things/discussion evolve along ethereumjs/ethereumjs-util#250, would generally not so beautiful if we let the Util v7 library out in the wild and then eventually directly deprecate the version ourselves.

[holgerd77]

Ok, this is now failing again sigh (non-deterministically, on random runs) but instead of giving the r._strip is not a function error (BN._strip() being a method from v5) it now throws with r.strip is not a function (BN.strip() being a method from v5).

This is a strong indicator that this wasn't actually an interoperability issue but triggered by some other condition. Will try to give this some further investigation.

[ricmoo]

Do you have any information in the non-standard PBKDF2 thing? I use node’s in node and my own adapted version in the browser, and have never had any problems.

Are you thinking of the axic testcase? (If you test against the ethers testcases, it’s present there too)?

That should’ve only been a problem for libraries using bitpay’s HD implementation...

Or is there another issue?

[holgerd77]

Ok, I got at least a bit further. The above is the moment before this errors. r is a BN v5 object (passed in as a parameter as num). All this takes locally place in a v4 context - so MPrime.prototype.ireduce in an extract from the BN v4 version code. This code assumes that there is a BN.strip() function, which is not (it's BN_strip() in the v5 code, therefore this fails. This can actually fail in Both directions (with r being BN v4 executed in a v5 context), which makes fixing on the BN.js side even more complicated.

What was confusing at first was that there are actually only BN v4 using libraries interacting with each other, one was also wondering where the BN v5 code was coming from at all.

Following is the npm ls bn.js output:

ethereumjs-wallet@0.6.3 [PATH]/ethereumjs-wallet
├─┬ ethereumjs-util@7.0.1
│ ├── bn.js@4.11.8
│ └─┬ rlp@2.2.4
│   └── bn.js@4.11.8  deduped
├─┬ ethers@4.0.47
│ ├── bn.js@4.11.8  deduped
│ └─┬ elliptic@6.5.2
│   └── bn.js@4.11.8  deduped
├─┬ hdkey@1.1.2
│ └─┬ secp256k1@3.8.0
│   └── bn.js@4.11.8  deduped
└─┬ karma-typescript@5.0.2
  └─┬ crypto-browserify@3.12.0
    ├─┬ browserify-sign@4.1.0
    │ ├── bn.js@5.1.1
    │ ├─┬ browserify-rsa@4.0.1
    │ │ └── bn.js@4.11.8  deduped
    │ └─┬ parse-asn1@5.1.5
    │   └─┬ asn1.js@4.10.1
    │     └── bn.js@4.11.8  deduped
    ├─┬ create-ecdh@4.0.3
    │ └── bn.js@4.11.8  deduped
    ├─┬ diffie-hellman@5.0.3
    │ ├── bn.js@4.11.8  deduped
    │ └─┬ miller-rabin@4.0.1
    │   └── bn.js@4.11.8  deduped
    └─┬ public-encrypt@4.0.3
      └── bn.js@4.11.8  deduped

So turns out the v5 code is coming from the browserify-sign@4.1.0 karma-typescript sub dependency. I think all this points very much to some weird thing happening in the Karma compilation/bundling process not cleanly getting the library versions apart.

So good so far.

[holgerd77]

Ah, here this actually happens (respectively one line before in var x = new BN(xBuffer)) in the ripemd160 dependency which is used by crypto-browserify:

ethereumjs-wallet@0.6.3 [PATH]/ethereumjs-wallet
├─┬ bs58check@2.1.2
│ └─┬ create-hash@1.2.0
│   └── ripemd160@2.0.2
└─┬ karma-typescript@5.0.2
  └─┬ crypto-browserify@3.12.0
    ├─┬ create-hmac@1.1.7
    │ └── ripemd160@2.0.2  deduped
    └─┬ pbkdf2@3.0.17
      └── ripemd160@2.0.2  deduped

So this instantiates a BN v5 instance.

[holgerd77]

This problem will likely only get bigger of time. So it might be worth to suggest two - in it's scope both pretty small - bug fix releases over on the BN side for both the v5 and v4 branch.

so a simple:

if (r.strip !== undefined) {
   r.strip();
} else {
   r._strip();
}

would already do it for both versions.

[ryanio]

@holgerd77 great investigation, I agree that this problem may only get worse as some libs start upgrading to v5 and others stay on v4.

Instead of checking for r.strip or r._strip, what do you think if instead we proposed (on both v4 and v5 branches) wrapping the input param num in a new BN() to force it to use the lib's own code?

i.e. at https://github.com/indutny/bn.js/blob/master/lib/bn.js#L3059:

- var r = num
+ var r = new BN(num)

---

edit: regarding the karma failure, how weird is it that this exact branch running on my fork passes fine? ryanio#1

[holgerd77]

@ricmoo Sorry, read over your comment above at first: this is another issue with problems on the interoperability of BN.js v4 and v5 instances (see thread).

[holgerd77]

@ryanio I couldn't imagine how this goes so I made a test:

$ node
Welcome to Node.js v12.15.0.
Type ".help" for more information.
> const BN4 = require('BN.js')
undefined
> const BN5 = require('ethereumjs-util').BN
undefined
> let num4 = new BN4(3)
undefined
> num4.strip
[Function: strip]
> num4._strip
undefined
> let num5 = new BN5(4)
undefined
> num5.strip
undefined
> num5._strip
[Function: strip]
> let w5num4 = new BN5(num4)
undefined
> w5num4.strip
[Function: strip]
> w5num4._strip
undefined

This doesn't seem to work, the wrapped w5num4 instance is still not exposing a _strip function. I would then rather stick with/suggest the solution I suggested above? Doesn't has major downsides, has it?

[holgerd77]

@ryanio I've now provided a summary of our findings here over on the BN.js issue and suggested to apply two fixes (now took the conditional versions) for BN.js v4 and v5.

I would suggest that me move on here independently and continue on the release. For now I have removed the three failing browser test runs together with a comment with the Karma check you already applied on the VM.

So this is now ready for review. Phew. 😄

[holgerd77]

Update: there were releases with fixes today on the BN.js side for both v4 and v5 (thanks @fanatid), so I re-added the browser tests again.

[ryanio]

Phew, that was fast! LGTM!

[holgerd77]

@ryanio What a journey. 😄