node-rdkafka fails to install/build on Windows

[ManfredLange]

Environment Information

• Windows 10 Pro, 64 bit
• Node Version 8.12.0
• NPM Version 6.4.1
• C++ Toolchain: msbuild engine version 12.0.40629.0
• node-rdkafka version commit e82bb2e (latest master as of writing)
• nuget: 4.6.2.5055
• node-gyp 3.8.0

Steps to Reproduce

I followed the steps as per this comment on issue 273 which are

git clone https://github.com/Blizzard/node-rdkafka.git node-rdkafka --recursive
cd node-rdkafka
npm install nan
node-gyp rebuild --verbose

Fails with error message: "MSBUILD : error MSB1009: Project file does not exist." originating from file "build\deps\librdkafka.vcxproj"

node-rdkafka Configuration Settings

I don't know what information you need here.

Additional context

Any suggestions for how to diagnose and resolve this are welcome!

I also tried installing the package via npm. Executing npm i node-rdkafka the output also includes these lines:

Warning: Missing input files:
C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\zlib.dll
C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\librdkafka.sln
C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafkacpp.dll
C:\projects\hypr\lic\event-publisher-service\node_modules\node-rdkafka\build\deps....\deps\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafka.dll

Followed later by

Copying deps/librdkafka/win32/outdir/v120/x64/Release/librdkafkacpp.dll to ../build/Release\librdkafkacpp.dll
The system cannot find the file specified.
Copying deps/librdkafka/win32/outdir/v120/x64/Release/librdkafka.dll to ../build/Release\librdkafka.dll
The system cannot find the file specified.
nuget_restore, and also build_dependencies
Unknown command: '....\deps\restore'
NuGet.CommandLine.CommandLineException: Unknown command: '....\deps\restore'

This seems to suggest that something with the node-rdkafka installation script is not quite right.

[skyfloyd]

try npm install node-rdkafka@2.3.4

[ManfredLange]

@skyfloyd Thank you for the suggestion. I tried that. It didn't quite work and now suggests using VS 2010 (going back in time?).

The first entries that indicate something is not going according to plan are:

Warning: Missing input files:
C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafkacpp.dll
C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka\win32\outdir\v120\x64\Release\librdkafka.dll
C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka\win32\outdir\v120\x64\Release\zlib.dll

Note that I simplified my setup by creating an empty directory and then executing these commands:

1. npm init -y
2. `npm install node-rdkafka@2.3.4

This simplification didn't resolve it either.

However, after removing node.js, npm, all node_modules as well as the respective caches, I got a bit further down the road by executing the following:

1. Uninstall and reinstall VS C++ build tools manually (the full one from npm package windows-build-tools
2. Install node.js 8.12.0 including npm 6.4.1
3. Install node-gyp 3.8.0 globally
4. 'npm config set msvs_version 2015` (I tried 2017 first but ran into problems)
5. 'node-gyp configure --msvs_version=2015`
6. Added path to nuget.exe to Path (do not include nuget.exe!)
7. 'npm install node-rdkafka@2.3.4`

With that in place it still reports the missing files first but then continues to use nuget to get them.

Then the compilation fails with missing headers for OpenSSL. I believe I have seen this elsewhere, so will try to research this for a bit and report back here.

Update: I found some OpenSSL related information here. I downloaded and installed it into C:\OpenSSL-win32. However, the linker step of the node-rdkafka install fails with a several unresolved external symbols similar to the following

libeay32MT.lib(cryptlib.obj) : error LNK2019: unresolved external symbol __iob_func referenced in function OPENSSL_stderr [C:\projects\tests\kafka\node
_modules\node-rdkafka\deps\librdkafka\win32\librdkafka.vcxproj] [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka.vcxproj]
libeay32MT.lib(ui_openssl.obj) : error LNK2001: unresolved external symbol __iob_func [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafk
a\win32\librdkafka.vcxproj] [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka.vcxproj]
ssleay32MT.lib(t1_enc.obj) : error LNK2001: unresolved external symbol __iob_func [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka\wi
n32\librdkafka.vcxproj] [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka.vcxproj]
libeay32MT.lib(txt_db.obj) : error LNK2001: unresolved external symbol __iob_func [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka\wi
n32\librdkafka.vcxproj] [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka.vcxproj]
libeay32MT.lib(cryptlib.obj) : error LNK2019: unresolved external symbol sscanf referenced in function OPENSSL_cpuid_setup [C:\projects\tests\kafka\nod
e_modules\node-rdkafka\deps\librdkafka\win32\librdkafka.vcxproj] [C:\projects\tests\kafka\node_modules\node-rdkafka\deps\librdkafka.vcxproj]
...

Question: where does the node-rdkafka installer expect OpenSSL to be located during install (ie the link step)? Also, is version 1.0.2 the correct version?

[skyfloyd]

@ManfredLange github doesn't allow me to attache exe file. Please find Win64OpenSSL-1_0_2o.exe in the internet and it will does everything for you. In my case it is installed in C:\OpenSSL-Win64 and there is OPENSSL_CONF : C:\OpenSSL-Win64\bin\openssl.cfg in system variables.

[t-davies]

We had this exact same issue recently, can confirm using 2.3.4 (with VS2010 and OpenSSL set up as required) allows the install to complete successfully. I assume something about the Windows build changed between 2.3.4 and the current version?

[ManfredLange]

@skyfloyd I found version 1.0.2p but haven't been able to locate version 1.0.2o. Do you happen to have a URL for the latter? Thank you!

Alternatively, please send email (see my github profile) and we can then figure out how to get the file that you have across. That'd be awesome.

[skyfloyd]

@ManfredLange I sent it by email. Doesn't Win64 OpenSSL v1.0.2p work?

[skyfloyd]

@ManfredLange hmmm, sorry gmail doesn't allow me to send it by email This message was blocked because its content presents a potential security issue. Please try 1.0.2p, I believe it must work as well.

[ManfredLange]

@skyfloyd and @t-davies Thank you both for your suggestions. Very much appreciated!

Based on your comments and other sources I was able to make it work with the following approach in my own environment as well in other team members' environments:

A. Installing Prerequisites

1. Uninstall node.js
2. Remove manually everything that looks like remnants of nodejs. In particular check that the cache directory is gone. In my case the cache was located at C:\Users\Manfred\AppData\Roaming\npm-cache. In my case there was/is also a directory named C:\Users\Manfred\AppData\Roaming\npm. Also check C:\Program Files\nodejs. If any of these directories exist after uninstall, remove them manually. In my environment with multiple installs/uninstalls/reinstalls it was messed up at some point.
3. Install node.js, version 8.12.0 or later from here. This also installs npm version 6.4.1 or later.
4. Open a terminal windows, e.g. PowerShell
5. Install node-gyp using npm install -g node-gyp@3.8.0 (or newer version). Important: Install globally with flag -g.
6. Install VC++ / Visual Studio 2010 build tools. Note that it is not necessary to install the entire Visual Studio 2010, just the build tools. Get the installer from the URL mentioned here. As of writing the link is this. When installing accept all defaults. If you already have these tools installed, first uninstall them, then re-install.
7. Add the path for the 12.0-version VC++ build tools to your Path environment variable. In my case (all defaults) this was located at C:\Program Files (x86)\MSBuild\12.0\Bin. Important: Do not use any of the other versions that you may have. The path must contain 12.0 otherwise you will be using the wrong version. I tested version 14.0 which resulted in compiler or linker errors.
8. Install Python 2.7.15 (or 2.7.14) from python.org. Either pick the 32bit installer or the 64bit installer. Important: Versions other than 2.7.x, e.g. 3.x or later will not work.
9. Add the path for Python 2.7 to the Path environment variable. Also create a new environment variable named "Python" that contains the path including the full name of the executable. In my case this was C:\bin\python27\python.exe.
10. Download nuget.exe version 4.6.2 or later from here. Version 4.7.1 is recommended as of writing.
11. Add the path for nuget to the Path environment variable
12. Download Win32_OpenSSL_1.0.2p.exe and install using all defaults. If this link is broken, Locate the latest 1.0.2 version for Win32 that does not say "Light" on it on this web site. Note that I didn't test any of the other versions. They may work but during my research I found that everybody was referring to version 1.0.2o or version 1.0.2p.

B. Pre-Build Checks

Here are some checks you can run to determine your chances of a successful install. If any of these tests fail, the install is unlikely to succeed.

1. In your terminal window run the command node -v and confirm it is version 8.12.0 or later
2. Run command npm -v and confim it is version 6.4.1 or later
3. Run command nuget and confirm it is version 4.6.2 or later
4. Run command msbuild and confirm it is version 12.0.x.y (but nothing newer/older!!)
5. Run command python and confirm it is version 2.7.x (3.x or later will not work)

C. Build

Execute npm install node-rdkafka@2.3.4.

Assuming I haven't missed an important step this should now work. It did in our environments, i.e. more than one Windows based development machine.

[RezaRahmati]

@ManfredLange I wasn't able to install msbuild 12 from the link provided, instead downloading from here works