bcrypto fails to build on windows

[faustbrian]

Currently building bcrypto on Windows seems to fail, at least with GitHub Actions as I don't have a Windows machine at hand with a development environment to test it there.

2020-06-22T16:23:41.6430389Z ##[group]Run yarn install --frozen-lockfile
2020-06-22T16:26:34.4465741Z error D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto: Command failed.
2020-06-22T16:26:34.4466189Z Exit code: 1
2020-06-22T16:26:34.4466579Z info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
2020-06-22T16:26:34.4467098Z Command: node-gyp rebuild
2020-06-22T16:26:34.4467349Z Arguments: 
2020-06-22T16:26:34.4467833Z Directory: D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto
2020-06-22T16:26:34.4468131Z Output:
2020-06-22T16:26:34.4468447Z gyp info it worked if it ends with ok
2020-06-22T16:26:34.4468803Z gyp info using node-gyp@7.0.0
2020-06-22T16:26:34.4469334Z gyp info using node@12.18.0 | win32 | x64
2020-06-22T16:26:34.4469763Z gyp info find Python using Python version 3.7.7 found at "C:\hostedtoolcache\windows\Python\3.7.7\x64\python.exe"
2020-06-22T16:26:34.4470131Z gyp info find VS using VS2019 (16.6.30128.74) found at:
2020-06-22T16:26:34.4470512Z gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise"
2020-06-22T16:26:34.4470862Z gyp info find VS run with --verbose for detailed information
2020-06-22T16:26:34.4471224Z gyp info spawn C:\hostedtoolcache\windows\Python\3.7.7\x64\python.exe
2020-06-22T16:26:34.4471511Z gyp info spawn args [
2020-06-22T16:26:34.4472578Z gyp info spawn args   'C:\\Users\\runneradmin\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\gyp\\gyp_main.py',
2020-06-22T16:26:34.4472950Z gyp info spawn args   'binding.gyp',
2020-06-22T16:26:34.4473293Z gyp info spawn args   '-f',
2020-06-22T16:26:34.4473580Z gyp info spawn args   'msvs',
2020-06-22T16:26:34.4473863Z gyp info spawn args   '-I',
2020-06-22T16:26:34.4474419Z gyp info spawn args   'D:\\a\\desktop-wallet\\desktop-wallet\\node_modules\\@arkecosystem\\platform-sdk-crypto\\node_modules\\bcrypto\\build\\config.gypi',
2020-06-22T16:26:34.4474746Z gyp info spawn args   '-I',
2020-06-22T16:26:34.4475181Z gyp info spawn args   'C:\\Users\\runneradmin\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\addon.gypi',
2020-06-22T16:26:34.4475498Z gyp info spawn args   '-I',
2020-06-22T16:26:34.4476220Z gyp info spawn args   'C:\\Users\\runneradmin\\AppData\\Local\\node-gyp\\Cache\\12.18.0\\include\\node\\common.gypi',
2020-06-22T16:26:34.4476647Z gyp info spawn args   '-Dlibrary=shared_library',
2020-06-22T16:26:34.4476970Z gyp info spawn args   '-Dvisibility=default',
2020-06-22T16:26:34.4477443Z gyp info spawn args   '-Dnode_root_dir=C:\\Users\\runneradmin\\AppData\\Local\\node-gyp\\Cache\\12.18.0',
2020-06-22T16:26:34.4478048Z gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\runneradmin\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp',
2020-06-22T16:26:34.4478582Z gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\12.18.0\\\\<(target_arch)\\\\node.lib',
2020-06-22T16:26:34.4479112Z gyp info spawn args   '-Dmodule_root_dir=D:\\a\\desktop-wallet\\desktop-wallet\\node_modules\\@arkecosystem\\platform-sdk-crypto\\node_modules\\bcrypto',
2020-06-22T16:26:34.4479448Z gyp info spawn args   '-Dnode_engine=v8',
2020-06-22T16:26:34.4479744Z gyp info spawn args   '--depth=.',
2020-06-22T16:26:34.4480079Z gyp info spawn args   '--no-parallel',
2020-06-22T16:26:34.4480453Z gyp info spawn args   '--generator-output',
2020-06-22T16:26:34.4480938Z gyp info spawn args   'D:\\a\\desktop-wallet\\desktop-wallet\\node_modules\\@arkecosystem\\platform-sdk-crypto\\node_modules\\bcrypto\\build',
2020-06-22T16:26:34.4481271Z gyp info spawn args   '-Goutput_dir=.'
2020-06-22T16:26:34.4481552Z gyp info spawn args ]
2020-06-22T16:26:34.4481976Z gyp info spawn C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe
2020-06-22T16:26:34.4482273Z gyp info spawn args [
2020-06-22T16:26:34.4482582Z gyp info spawn args   'build/binding.sln',
2020-06-22T16:26:34.4482903Z gyp info spawn args   '/clp:Verbosity=minimal',
2020-06-22T16:26:34.4483204Z gyp info spawn args   '/nologo',
2020-06-22T16:26:34.4483586Z gyp info spawn args   '/p:Configuration=Release;Platform=x64'
2020-06-22T16:26:34.4483869Z gyp info spawn args ]
2020-06-22T16:26:34.4484288Z Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
2020-06-22T16:26:34.4484560Z   base58.c
2020-06-22T16:26:34.4484809Z   bech32.c
2020-06-22T16:26:34.4485056Z   cash32.c
2020-06-22T16:26:34.4485299Z   murmur3.c
2020-06-22T16:26:34.4485539Z   aead.c
2020-06-22T16:26:34.4485787Z   chacha20.c
2020-06-22T16:26:34.4486055Z   drbg.c
2020-06-22T16:26:34.4486296Z   dsa.c
2020-06-22T16:26:34.4486533Z   ecc.c
2020-06-22T16:26:34.4486768Z   hash.c
2020-06-22T16:26:34.4487136Z   kdf.c
2020-06-22T16:26:34.4487374Z   poly1305.c
2020-06-22T16:26:34.4487614Z   rsa.c
2020-06-22T16:26:34.4487855Z   salsa20.c
2020-06-22T16:26:34.4488097Z   siphash.c
2020-06-22T16:26:34.4488334Z   util.c
2020-06-22T16:26:34.4488621Z   bcrypto.cc
2020-06-22T16:26:34.4488867Z   mini-gmp.c
2020-06-22T16:26:34.4489120Z   secp256k1.c
2020-06-22T16:26:34.4489939Z D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto\src\secp256k1\src\secp256k1.c : fatal error C1083: Cannot open compiler generated file: '': Invalid argument [D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto\build\bcrypto.vcxproj]
2020-06-22T16:26:34.4490334Z gyp ERR! build error 
2020-06-22T16:26:34.4490818Z gyp ERR! stack Error: `C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe` failed with exit code: 1
2020-06-22T16:26:34.4491316Z gyp ERR! stack     at ChildProcess.onExit (C:\Users\runneradmin\AppData\Local\Yarn\Data\global\node_modules\node-gyp\lib\build.js:194:23)
2020-06-22T16:26:34.4491686Z gyp ERR! stack     at ChildProcess.emit (events.js:315:20)
2020-06-22T16:26:34.4492083Z gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
2020-06-22T16:26:34.4492406Z gyp ERR! System Windows_NT 10.0.17763
2020-06-22T16:26:34.4493104Z gyp ERR! command "C:\\hostedtoolcache\\windows\\node\\12.18.0\\x64\\node.exe" "C:\\Users\\runneradmin\\AppData\\Local\\Yarn\\Data\\global\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
2020-06-22T16:26:34.4493644Z gyp ERR! cwd D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto
2020-06-22T16:26:34.4493967Z gyp ERR! node -v v12.18.0
2020-06-22T16:26:34.4494263Z gyp ERR! node-gyp -v v7.0.0
2020-06-22T16:26:34.4494536Z gyp ERR! not ok
2020-06-22T16:26:34.4958706Z ##[error]Process completed with exit code 1.
2020-06-22T16:26:34.5114891Z Post job cleanup.
2020-06-22T16:26:36.1251045Z [command]"C:\Program Files\Git\bin\git.exe" version
2020-06-22T16:26:36.1424143Z git version 2.27.0.windows.1
2020-06-22T16:26:36.1483570Z [command]"C:\Program Files\Git\bin\git.exe" config --local --name-only --get-regexp core\.sshCommand
2020-06-22T16:26:36.1691417Z [command]"C:\Program Files\Git\bin\git.exe" submodule foreach --recursive "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2020-06-22T16:26:36.6081676Z [command]"C:\Program Files\Git\bin\git.exe" config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-06-22T16:26:36.6243909Z http.https://github.com/.extraheader
2020-06-22T16:26:36.6290119Z [command]"C:\Program Files\Git\bin\git.exe" config --local --unset-all http.https://github.com/.extraheader
2020-06-22T16:26:36.6505431Z [command]"C:\Program Files\Git\bin\git.exe" submodule foreach --recursive "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2020-06-22T16:26:37.0882980Z Cleaning up orphan processes
2020-06-22T16:26:37.1117923Z Terminate orphan process: pid (6952) (vctip)

[pinheadmz]

related: kyokan/bob-wallet#142

@chjj any tips?

[chjj]

This is a strange error:

fatal error C1083: Cannot open compiler generated file: ''

My best guess is that this path is too long:

D:\a\desktop-wallet\desktop-wallet\node_modules\@arkecosystem\platform-sdk-crypto\node_modules\bcrypto\src\secp256k1\src\secp256k1.c

Windows can't handle really long path names. This problem is worsened by older releases of bcrypto setting the msbuild option ObjectFileName to $(IntDir)/%(Directory)/. That essentially doubles the length of the path when creating object files. We can't use the more typical $(IntDir)/%(RelativeDir)/ because node-gyp always builds win_delay_load_hook.cc which resides in the node-gyp workspace, not the bcrypto one.

The good news is that I've removed the ObjectFileName setting because we don't need it anymore. All our object files have unique names. I'm curious whether the current master would produce the same error.

[chjj]

On a side note, I think we should consider distributing pre-built binaries for windows and apple. Building on windows and apple is an absolute nightmare. Windows requires several gigabytes of disk space to install all of the VS build tools, and it's not something windows users typically have installed. Mac OSX Catalina is having disastrous issues with builds as well. So much so that dozens of programmers couldn't figure out the true source of the issue even after months of researching it.

On the other hand, Linux and the BSDs just work. The compiler and build system is more often than not already installed. The compiler isn't some crazy Apple fork, and it doesn't require 100gb of disk space to install in the first place.

I've been hoping for a node build system that just bundles vanilla clang & make for windows and apple (or auto-downloads & caches them). I feel like this would solve all of the build issues for windows and apple, and would allows us to switch to simple Makefiles instead of using gyp files. For more complex builds, maybe cmake would be added in the future. Maybe someday a zero-headache build system for node will be created, but I guess until then, the only option is prebuilds.

[faustbrian]

The good news is that I've removed the ObjectFileName setting because we don't need it anymore. All our object files have unique names. I'm curious whether the current master would produce the same error.

I'll give it a try with master when I get a chance. Would there be anything else I could do to make it work with windows if that still fails?

[pinheadmz]

@faustbrian I'm a newb at development in Windows environment but learned last night that I could not build bcrypto unless git-bash was run as Administrator, dunno if that helps.

[xraid]

To have it build with docs install instructions on OSX with "gpk rebuild"

I had to add :

*** => bcoin/node_modules/bcrypto/src/secp256k1/src/scalar_4x64_impl.h

line 962 => #include "string.h" // *** monkey patch ***

hth
/xraid

it seems memcpy is not declared per default with the OSX toolchain and perhaps same with WIN.
either *NIX toolchain includes the declaration per default or the *NIX setup has access to the declaration of
memcpy without an explicit include ...

[bluelovers]

yarn add v1.22.4
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error T:\download\kkk\node_modules\bcrypto: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments: 
Directory: T:\download\kkk\node_modules\bcrypto
Output:
T:\download\kkk\node_modules\bcrypto>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild ) 
gyp info it worked if it ends with ok
gyp info using node-gyp@5.1.0
gyp info using node@14.8.0 | win32 | x64
gyp info find Python using Python version 3.8.5 found at "C:\Python38\python.exe"
gyp info find VS using VS2017 (15.9.28307.423) found at:
gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools"
gyp info find VS run with --verbose for detailed information
gyp info spawn C:\Python38\python.exe
gyp info spawn args [
gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-I',
gyp info spawn args   'T:\\download\\kkk\\node_modules\\bcrypto\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\User\\AppData\\Local\\node-gyp\\Cache\\14.8.0\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\User\\AppData\\Local\\node-gyp\\Cache\\14.8.0',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\User\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\14.8.0\\\\<(target_arch)\\\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=T:\\download\\kkk\\node_modules\\bcrypto',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'T:\\download\\kkk\\node_modules\\bcrypto\\build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe
gyp info spawn args [
gyp info spawn args   'build/binding.sln',
gyp info spawn args   '/clp:Verbosity=minimal',
gyp info spawn args   '/nologo',
gyp info spawn args   '/p:Configuration=Release;Platform=x64'
gyp info spawn args ]
�b���ѨM��פ��@���ظm�@�ӱM�סC�Y�n�ҥΥ���ظm�A�зs�W "/m" �ѼơC
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\VC\VCTargets\Microsoft.Cpp.WindowsSDK.targets(46,5): error MSB8036: �䤣�� Windows SDK ���� 10.0.17763.0�C�w�˩һݪ� Windows SDK �����A���ܧ�b�M���ݩʭ��� SDK �����A�Φb��פW���@�U�ƹ��k��A�ÿ�� [���w��ץؼ�]�C [T:\download\kkk\node_modules\bcrypto\build\torsion.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\VC\VCTargets\Microsoft.Cpp.WindowsSDK.targets(46,5): error MSB8036: �䤣�� Windows SDK ���� 10.0.17763.0�C�w�˩һݪ� Windows SDK �����A���ܧ�b�M���ݩʭ��� SDK �����A�Φb��פW���@�U�ƹ��k��A�ÿ�� [���w��ץؼ�]�C [T:\download\kkk\node_modules\bcrypto\build\secp256k1.vcxproj]
gyp ERR! build error 
gyp ERR! stack Error: `C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)
gyp ERR! System Windows_NT 10.0.19041
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd T:\download\kkk\node_modules\bcrypto
gyp ERR! node -v v14.8.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

[raulvictorrosa]

Find here guys, for me setting python to version 2 worked.

npm config set python /usr/bin/python2

after run yarn install

https://stackoverflow.com/questions/29673365/fails-to-build-node-package-due-to-python