Install fails on windows for all versions

[benmollet]

I'm trying to install the gem on windows with:

gem install rdkafka -v '0.8.1'

(I tried all newer versions as well)

But it fails with the following:

PS C:\code> gem install rdkafka -v '0.8.1'
Temporarily enhancing PATH for MSYS/MINGW...
Building native extensions. This could take a while...
ERROR: Error installing rdkafka:
ERROR: Failed to build gem native extension.

current directory: C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rdkafka-0.8.1/ext

C:/Ruby27-x64/bin/ruby.exe -IC:/Ruby27-x64/lib/ruby/2.7.0/rubygems -rrubygems C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake RUBYARCHDIR=C:/Ruby27-x64/lib/ruby/gems/2.7.0/extensions/x64-mingw32/2.7.0/rdkafka-0.8.1 RUBYLIBDIR=C:/Ruby27-x64/lib/ruby/gems/2.7.0/extensions/x64-mingw32/2.7.0/rdkafka-0.8.1

Extracting v1.4.0 into tmp/x86_64-w64-mingw32/ports/librdkafka/1.4.0... OK
Running 'configure' for librdkafka 1.4.0... ERROR, review 'C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rdkafka-0.8.1/ext/tmp/x86_64-w64-mingw32/ports/librdkafka/1.4.0/configure.log' to see what happened. Last lines are:

mklove/modules/configure.base: line 2047: source: configure.self: file not found

rake aborted!
Failed to complete configure task

Tasks: TOP => default
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rdkafka-0.8.1 for inspection.
Results logged to C:/Ruby27-x64/lib/ruby/gems/2.7.0/extensions/x64-mingw32/2.7.0/rdkafka-0.8.1/gem_make.out

Any help would be appreciated :)

[thijsc]

I don't believe we've tested this gem on Windows so far. You're not using the linux base system right?

[benmollet]

No, I tried this on regular windows

[benmollet]

@thijsc any thoughts on what might cause this?

[thijsc]

I have no clue honestly. I don't have access to any Windows machines that I can test with. I guess we could add a windows build to our CI. Would you be willing to work on that?

[mensfeld]

Closing due to lack of activity. If anyone is willing to add Windows, happy to accept contributions. WSL works to the extend of my knowledge.

[sirzeta]

I put significant effort (and love) into making librdkafka compile on Windows, requires MSYS and these dependencies installed:

# UCRT
pacman -S mingw-w64-ucrt-x86_64-cmake
pacman -S mingw-w64-ucrt-x86_64-toolchain

# MINGW64
pacman -S mingw-w64-x86_64-cmake
pacman -S mingw-w64-x86_64-toolchain

This issue is blocking some projects/libraries that depends on this gem to run on Windows-based systems, like karafka.

As indicated in this issue in librdkafka repository, mingw build must use CMake and not mklove. Furthermore, the code doesn't compile if the paths are too long.

To override this, I had to enable LongPathsEnabled in the Windows Registry and apply a patch to CMakeLists.txt:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"LongPathsEnabled"=dword:00000001

This configuration can also be controlled via Group Policy at Computer Configuration > Administrative Templates > System > Filesystem > Enable Win32 long paths.

Even with long path support enabled, mingw32-make fails to locate directories:

  It fails with the following output:

    Change Dir: 'D:/Development/sirzeta/repos/karafka-rdkafka/ext/tmp/x86_64-w64-mingw32/ports/librdkafka/git-bef137adc7909b63f82ccbb05e1266c9fbf3721b/librdkafka-bef137adc7909b63f82ccbb05e1266c9fbf3721b/CMakeFiles/CMakeScratch/TryCompile-bjcj5v'

    Run Build Command(s): C:/msys64/ucrt64/bin/cmake.exe -E env VERBOSE=1 C:/msys64/ucrt64/bin/mingw32-make.exe -f Makefile cmTC_36943/fast
    C:/msys64/ucrt64/bin/mingw32-make.exe  -f CMakeFiles\cmTC_36943.dir\build.make CMakeFiles/cmTC_36943.dir/build
    mingw32-make[1]: Entering directory 'D:/Development/sirzeta/repos/karafka-rdkafka/ext/tmp/x86_64-w64-mingw32/ports/librdkafka/git-bef137adc7909b63f82ccbb05e1266c9fbf3721b/librdkafka-bef137adc7909b63f82ccbb05e1266c9fbf3721b/CMakeFiles/CMakeScratch/TryCompile-bjcj5v'
    mingw32-make[1]: CMakeFiles\cmTC_36943.dir\build.make: No such file or directory
    mingw32-make[1]: *** No rule to make target 'CMakeFiles\cmTC_36943.dir\build.make'.  Stop.
    mingw32-make[1]: Leaving directory 'D:/Development/sirzeta/repos/karafka-rdkafka/ext/tmp/x86_64-w64-mingw32/ports/librdkafka/git-bef137adc7909b63f82ccbb05e1266c9fbf3721b/librdkafka-bef137adc7909b63f82ccbb05e1266c9fbf3721b/CMakeFiles/CMakeScratch/TryCompile-bjcj5v'
    mingw32-make: *** [Makefile:126: cmTC_36943/fast] Error 2

So I made some whacky changes to the task Build librdkafka at the given git sha or tag to make it work , so it did, here some code:

if RbConfig::CONFIG['host_os'] =~ /mswin|mingw|cygwin/ and exced_win32_max_path(version)
  # This gives us 66 characters more before hit MAX_PATH
  version = "git-#{ref[0...7]}"
end
  
# Specific to Windows-based systems, checks if CMakeTryCompile will generate paths longer than MAX_PATH.
# The function tries to anticipate the path to be generated by deducting some characters for the final comparison.
def exced_win32_max_path(version)
  ref = version.gsub('git-', '')
  dummy_recipe = MiniPortile.new("librdkafka", version)
  # Example path CMakeTryCompile could generate during build process, wasted 67 characters less.
  try_compile_obj_file_dir_example = 'CMakeFiles/CMakeScratch/TryCompile-p9t28r/CMakeFiles/cmTC_e7b3e.dir'
  work_dir =  File.join(File.dirname(__FILE__), "tmp/#{dummy_recipe.host}/ports/#{dummy_recipe.name}/#{dummy_recipe.version}/librdkafka-#{ref}")
  File.join(work_dir, try_compile_obj_file_dir_example).length > 250
end

The reason why I didn't PR it yet, this is the first time I play with Ruby and I do not know if I am coding this well, at least it’s working in every case I’ve tried.

Any advice is welcome.

Hello dll! 👇

[mensfeld]

I myself do not feel competent enough to manage or support officially windows builds. Not sure what @thijsc stance is on that one.

The reasons are two: I do not work on non-nix machines and there has been almost no interest in the community. for this feature (here I may be wrong though).

I am not sure we should roll it as an officially supported option without someone else looking at it and adding windows builds to the CI.

[sirzeta]

Maybe we could have the same posture as librdkafka, experimental but unsupported.

https://github.com/confluentinc/librdkafka/blob/bef137adc7909b63f82ccbb05e1266c9fbf3721b/README.md?plain=1#L138-L139

**NOTE**: See [CMake instructions](packaging/cmake/README.md) for experimental
          CMake build (unsupported).

I also work almost 99% on *nix like machines, my Macbook broke temporarily, I tried to work without WSL2 as an experiment, to avoid having to use Remote Development locally with RubyMine, it was a fun experience.

Maybe people’s interest is diluted by having WSL2 on hand, it also frustrated me when I saw that it did not work natively in Windows.

[mensfeld]

I don't have a strong opinion about that. Only that I do not feel competent or technically viable to support windows. If there is anyone willing to PR and work on adding Windows specs and all that is needed to have at least basic sanity checks, I would be happy to work with that person.

[sirzeta]

I wanted to give it a last try, just for fun, I was hoping the tests would work, but they failed miserably.

There is also the problem that the windows image (windows-latest) of Github Actions can only run containers with the windows/amd64 architecture, which the Docker image confluentinc/cp-kafka:7.6.1 does not have, so for testing we need a custom runner with hypervisor capabilities to run WSL2, just for Kafka.

It was fun, but I think I’ve had my dose of Windows for a while.

fail.txt

[mensfeld]

Yeah that's why I will close this issue. If someone comes that can add support to the CI so we at least have some stability I will be happy to merge.