remove buffer-dep, replace with bufferFrom

[dghelm]

PULL REQUEST

Update

Changed dependencies:

• dropped sjcl for pbkdf2
• dropped buffer for buffer-from (already used in a dep)
• updated @skynetlabs/tus-js-client@2.3.1 to update its buffer-from dep
• dropped randomBytes and instead used tweetnacl's method
• (and sjcl's hex conversion method was no longer necessary since pbkdf2 used a buffer that is easily cast as the necessary Uint8Array)

Overview

I'm looking into other dependencies and found this...

In a recent PR @peterjan made to ethers-js, the lead dev mentioned buffer adding lots of overhead in browser builds. This made me wonder if we needed it, and I found bufferFrom, which removes the need for the much larger Buffer library.

We already are using bufferFrom in tus-js-client dependency. So all we should be doing is dropping an unused dependency without actually adding an additional one.

Figured I'd submit the PR.

https://www.npmjs.com/package/buffer-from

[kwypchlo]

Looks GREAT!

[mrcnski]

As weird as it looks, having this line in this file, though unused, was preventing a bug in the browser. This should be tested before we merge it in - I can do that by running the test-skapp tests when I have time.

[dghelm]

With latest commits, I think this is fixed.

[mrcnski]

Don't merge yet.

[mrcnski]

Another thing, can you please remove package-lock.json?

[mrcnski]

When I try to webpack a skapp now I get this:

ERROR in ./node_modules/skynet-js/node_modules/safe-buffer/index.js 3:13-30
Module not found: Error: Can't resolve 'buffer' in '/Users/marcin/Sync/Repos/github.com/m-cat/identity-test-skapp/node_modules/skynet-js/node_modules/safe-buffer'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
	- add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
	- install 'buffer'
If you don't want to include a polyfill, you can use an empty module like this:
	resolve.fallback: { "buffer": false }

So it seems like it would require some more configuration on the part of the skapp author now...

[dghelm]

Ha, that's the exact error I was trying to run down, but for crypto. Not sure what safe-buffer is, but we may need to continue assessing.

[mrcnski]

Ha, that's the exact error I was trying to run down, but for crypto. Not sure what safe-buffer is, but we may need to continue assessing.

safe-buffer is a dependency of some of our dependencies, so not something that is totally trivial to remove.

[dghelm]

Ha, that's the exact error I was trying to run down, but for crypto. Not sure what safe-buffer is, but we may need to continue assessing.

safe-buffer is a dependency of some of our dependencies, so not something that is totally trivial to remove.

Ah okay -- yeah, I couldn't find an easy way to drop crypto without changing genKeyPairFromSeed to an async method, so I think the library is stuck displaying this warning for webpack 5 either way. The tests still pass, but maybe this is more of a consideration for libkernel/next SDK.

[kwypchlo]

@dghelm I did some digging and the only dependency that uses safe-buffer is randombytes and we could replace it with https://github.com/consento-org/get-random-values-polypony (npm https://www.npmjs.com/package/get-random-values) - this should allow us to drop the crypto package too (I think)

[dghelm]

@dghelm I did some digging and the only dependency that uses safe-buffer is randombytes

Good catch. tweetnacl has a randomBytes method, so the latest commit drops this dependency as well.

[dghelm]

Also, buffer-from is not de-deduped from the tus-js-client fork, since it uses 0.1.2 -- we'd maybe want to upgrade the dep in our fork if the breaking changes are minimal.

Also, crypto is still used in sjcl afaik, and alternatives to the key-gen method it uses all seem to be async methods. Digging into that is what started this PR, and I made a post on Monday in #skynet-js channel digging a bit more into it.

Current output of npm ls --prod true:

skynet-js@4.1.0 /home/dgh/dev/temp/skynet-js
├─┬ @skynetlabs/tus-js-client@2.3.0
│ ├── buffer-from@0.1.2
│ ├─┬ combine-errors@3.0.3
│ │ ├── custom-error-instance@2.1.1
│ │ └─┬ lodash.uniqby@4.5.0
│ │   ├─┬ lodash._baseiteratee@4.7.0
│ │   │ └─┬ lodash._stringtopath@4.8.0
│ │   │   └── lodash._basetostring@4.12.0
│ │   └─┬ lodash._baseuniq@4.6.0
│ │     ├── lodash._createset@4.0.3
│ │     └── lodash._root@3.0.1
│ ├── is-stream@2.0.0
│ ├── js-base64@2.6.4
│ ├── lodash.throttle@4.1.1
│ ├─┬ proper-lockfile@2.0.1
│ │ ├── graceful-fs@4.2.6
│ │ └── retry@0.10.1
│ └── url-parse@1.5.1 deduped
├─┬ async-mutex@0.3.2
│ └── tslib@2.3.1
├─┬ axios@0.26.1
│ └── follow-redirects@1.14.8
├── base32-decode@1.0.0
├─┬ base32-encode@1.2.0
│ └── to-data-view@1.1.0
├── base64-js@1.5.1
├── blakejs@1.2.1
├── buffer-from@1.1.2
├── mime@3.0.0
├── path-browserify@1.0.1
├── post-me@0.4.5
├── sjcl@1.0.8
├─┬ skynet-mysky-utils@0.3.0
│ └── post-me@0.4.5 deduped
├── tweetnacl@1.0.3
├── url-join@4.0.1
└─┬ url-parse@1.5.1
  ├── querystringify@2.2.0
  └── requires-port@1.0.0

[kwypchlo]

Also, buffer-from is not de-deduped from the tus-js-client fork, since it uses 0.1.2 -- we'd maybe want to upgrade the dep in our fork if the breaking changes are minimal.

@m-cat updated upstream on our fork and seems like we're using buffer-from 1.1.2 now too so that should use the same version now https://github.com/SkynetLabs/tus-js-client/blob/master/package.json#L79

[kwypchlo]

Also, crypto is still used in sjcl afaik, and alternatives to the key-gen method it uses all seem to be async methods. Digging into that is what started this PR, and I made a post on Monday in #skynet-js channel digging a bit more into it.

So afaik scjs is used only here:
https://github.com/SkynetLabs/skynet-js/blob/master/src/crypto.ts#L1

One method is pbkdf2 and we could use https://www.npmjs.com/package/pbkdf2 instead - it has sync alternative:

pbkdf2.pbkdf2Sync('password', 'salt', 1, 32, 'sha512')

Second method is codec.hex which looks like it's quite simple function that does not rely on crypto and maybe it could be copied over or replaced by something different:
https://github.com/bitwiseshiftleft/sjcl/blob/master/core/codecHex.js#L14-L20

[dghelm]

One method is pbkdf2 and we could use https://www.npmjs.com/package/pbkdf2 instead - it has sync alternative

I reviewed that library last week, but afaik it still depends on crypto but maybe because it's part of the crypto-browserify it handles it in a way that doesn't throw errors. See:

This library provides the functionality of PBKDF2 with the ability to use any supported hashing algorithm returned from crypto.getHashes()

I'll give it a go and see what happens though.

[kwypchlo]

One method is pbkdf2 and we could use https://www.npmjs.com/package/pbkdf2 instead - it has sync alternative

I reviewed that library last week, but afaik it still depends on crypto but maybe because it's part of the crypto-browserify it handles it in a way that doesn't throw errors. See:

This library provides the functionality of PBKDF2 with the ability to use any supported hashing algorithm returned from crypto.getHashes()

I'll give it a go and see what happens though.

as far as I can tell, the browser build (browser.js entrypoint) does not require "crypto" package - the nodejs build (index.js entrypoint) does require it but that's fine, skynet-js should correctly pull the browser version for the browser build and nodejs version for the nodejs build (I guess it should)

[dghelm]

Now getting no errors or warnings on Create React App 5.0.1, which uses webpack 5.

[mrcnski]

Hmm I'm still getting this when I try to build a plain skapp (non-React) with Webpack:

ERROR in ./node_modules/skynet-js/node_modules/safe-buffer/index.js 3:13-30
Module not found: Error: Can't resolve 'buffer' in '/Users/marcin/Sync/Repos/github.com/m-cat/identity-test-skapp/node_modules/skynet-js/node_modules/safe-buffer'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
	- add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
	- install 'buffer'
If you don't want to include a polyfill, you can use an empty module like this:
	resolve.fallback: { "buffer": false }

[dghelm]

Hmm I'm still getting this when I try to build a plain skapp (non-React) with Webpack:

And just to confirm -- you were also seeing this with webpack 5 and our current production version, right?

If so, I think this is an improvement and we should identify the webpack configuration differences as a follow-up.

[mrcnski]

@dghelm Yeah, I'm running

"webpack": "^5.16.0",

I think the situation is actually worse than before because I get errors now, whereas before I was only getting warners. Manually listing buffer as a dependency in skynet-js was preventing those errors.

[kwypchlo]

@dghelm @m-cat I added a commit that replaces pbkdf2 package with pbkdf2-hmac package here 2ba6606

Package pbkdf2-hmac uses native browser crypto function when running in browser and native node crypto package when running in node environment. Unfortunately, since browser native implementation is async, it means a breaking change in genKeyPairFromSeed and genKeyPairAndSeed - they need to be async. Let me know your thoughts. I think this is worth it, making skynet-js lighter (buffer package is 26.8kB and pbkdf2-hmac is 2.4kB), faster (native browser implementation is definitely faster than javascript one) and independent of nodejs packages (we will be able to use webpack without polyfills and warnings) is a good path forward. If this is not on the table, we can revert my commit.

Implementation details: https://github.com/juanelas/pbkdf2-hmac/blob/master/src/ts/index.ts#L55-L79

[PAlexanderFranklin]

Svelte is having trouble with randombytes when instantiating the skynet client. This is on Svelte version 3.48.0.

Example Repo https://github.com/PAlexanderFranklin/caller-kit

[dghelm]

Unfortunately, since browser native implementation is async, it means a breaking change in genKeyPairFromSeed and genKeyPairAndSeed - they need to be async.

I think this is off the table for right now -- if we planned to move forward with skynet-js this seems great (there are also browser crypto methods I think we could take advantage of by moving more towards async), but I think we should document it separately from this PR, which seems increasingly important as more devs are running into issues with React and Svelte.

[kwypchlo]

Unfortunately, since browser native implementation is async, it means a breaking change in genKeyPairFromSeed and genKeyPairAndSeed - they need to be async.

I think this is off the table for right now -- if we planned to move forward with skynet-js this seems great (there are also browser crypto methods I think we could take advantage of by moving more towards async), but I think we should document it separately from this PR, which seems increasingly important as more devs are running into issues with React and Svelte.

I think it's still pretty reasonable to bump the major to remove those deps, that's really going to help with package size and compatibility with webpack 5. I bet ether.js maintainers would be happy with that too.

[mrcnski]

@kwypchlo I would be happy with that too, but we would need @DavidVorick to sign off on it.