Building on Windows fails

[arasic]

Bug Report

Describe the bug
When building from source code for Windows, it fails with error: LINK : fatal error LNK1104: cannot open file 'libcrypto.lib'

It occurs when the command cmake --build . --config Release is executed.

Here is the last part of the error output:

[ 98%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_ra_key.c.obj
flb_ra_key.c
[ 98%] Building C object src/CMakeFiles/fluent-bit-shared.dir/__/lib/lwrb/lwrb/src/lwrb/lwrb.c.obj
lwrb.c
[ 98%] Linking C shared library ..\bin\fluent-bit.dll
LINK: command "C:\PROGRA~1\MICROS~1\2022\COMMUN~1\VC\Tools\MSVC\1440~1.338\bin\Hostx86\x86\link.exe /nologo @CMakeFiles\fluent-bit-shared.dir\objects1.rsp /out:..\bin\fluent-bit.dll /implib:..\library\fluent-bit.lib /pdb:C:\Users\<username>\github\fluent-bit\build\bin\fluent-bit.dll.pdb /dll /version:0.0 /pdb:C:/Users/<username>/github/fluent-bit/build/bin/fluent-bit.dll.pdb /machine:X86 ws2_32.lib crypt32.lib Bcrypt.lib Shlwapi.lib ..\library\flb-http-server.lib ..\library\libcfl.lib ..\library\fluent-otel-proto.lib ..\library\libcmetrics.lib ..\library\libctraces.lib ..\lib\monkey\library\mk_core.lib ..\library\jsmn.lib ..\library\msgpack-c-static.lib ..\library\mpack-static.lib ..\library\chunkio-static.lib ..\library\miniz.lib ..\libr

....
LINK : fatal error LNK1104: cannot open file 'libcrypto.lib'
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=CMakeFiles\fluent-bit-shared.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x86\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x86\mt.exe --manifests -- C:\PROGRA~1\MICROS~1\2022\COMMUN~1\VC\Tools\MSVC\1440~1.338\bin\Hostx86\x86\link.exe /nologo @CMakeFiles\fluent-bit-shared.dir\objects1.rsp @C:\Users\<username>\AppData\Local\Temp\nmF2F6.tmp' : return code '0xffffffff'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX86\x86\nmake.exe" -s -f src\CMakeFiles\fluent-bit-shared.dir\build.make /nologo -SL                 src\CMakeFiles\fluent-bit-shared.dir\build' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX86\x86\nmake.exe" -s -f CMakeFiles\Makefile2 /nologo -LS                 all' : return code '0x2'
Stop.

To Reproduce

• Steps to reproduce the problem:
• Go to https://docs.fluentbit.io/manual/installation/windows#compile-from-source
• Clone the project
• cd fluent-bit/build
• build:

% cmake .. -G "NMake Makefiles"
% cmake --build .

Expected behavior
A fluent-bit.exe is generated in build\bin

Your Environment

• Version used: master and/or tags/v3.1.5 as well.
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version: Windows 11 Home
• Filters and plugins: default settings

Additional context
My goal is actually to use latest fluent-bit version (3.1.5) which contains support for kafka on Windows with ssl/tls

[arasic]

When I tested v3.1.4, it worked and was able to generate the binary, fluent-bit.exe

[cosmo0920]

My goal is actually to use latest fluent-bit version (3.1.5) which contains support for kafka on Windows with ssl/tls

The latest fluent-bit 3.1.5 already includes input and output kafka plugins, I'm afraid.
We enabled them in #8662.

[arasic]

Hiro, thats correct, I tried to build from source code for Windows but was unable to do so.

Now with a little back-and forth, I was able to achieve it, but had to do slight changes in the CMakeLists.txt under fluent-bit/ directory starting from line 522:

if(NOT WIN32)
  target_link_libraries(${PROJECT_NAME} pthread)
endif()

if(FLB_TLS)
  FLB_DEFINITION(FLB_HAVE_TLS)

  option(ENABLE_TESTING  OFF)
  option(ENABLE_PROGRAMS OFF)
  option(INSTALL_MBEDTLS_HEADERS OFF)

  # Link OpenSSL statically on Windows
  if (WIN32)
    set(OPENSSL_USE_STATIC_LIBS ON)
    set(OPENSSL_MSVC_STATIC_RT  ON)
  endif()

  # Use environment variable for OpenSSL root directory, fallback to a default if not set
  if(NOT DEFINED ENV{OPENSSL_ROOT_DIR})
    set(ENV{OPENSSL_ROOT_DIR} "C:/Program Files/OpenSSL-Win64")
  endif()
  set(OPENSSL_ROOT_DIR "$ENV{OPENSSL_ROOT_DIR}")
  set(OPENSSL_LIB_DIR "${OPENSSL_ROOT_DIR}/lib/VC/static")
  set(OPENSSL_INCLUDE_DIR "${OPENSSL_ROOT_DIR}/include")
  # Find OpenSSL
  find_package(OpenSSL REQUIRED)
  if(OPENSSL_FOUND)
    FLB_DEFINITION(FLB_HAVE_OPENSSL)
    # Specific configuration for Windows
    if (FLB_SYSTEM_WINDOWS)
      # Override the static libraries with the non-static versions if needed
      set(OPENSSL_LIBRARIES "${OPENSSL_LIB_DIR}/libssl_static.lib;${OPENSSL_LIB_DIR}/libcrypto_static.lib;${OPENSSL_LIB_DIR}/libssl.lib;${OPENSSL_LIB_DIR}/libcrypto.lib")
      message(STATUS "Using OpenSSL libraries: ${OPENSSL_LIBRARIES}")
      # Debugging lines to check for libcrypto.lib
      list(GET OPENSSL_LIBRARIES 1 LIBCRYPTO_PATH)
      if(EXISTS "${LIBCRYPTO_PATH}")
        message(STATUS "libcrypto.lib found at ${LIBCRYPTO_PATH}")
      else()
        message(WARNING "libcrypto.lib not found at ${LIBCRYPTO_PATH}")
      endif()
    endif()

  else()
    # Fatal error if OpenSSL is not found on Windows
    if (FLB_SYSTEM_WINDOWS)
      message(FATAL_ERROR "OpenSSL required on Windows, see DEVELOPER_GUIDE.md")
    endif()
  endif()
endif()

it is essentially the line:

set(OPENSSL_LIBRARIES "${OPENSSL_LIB_DIR}/libssl_static.lib;${OPENSSL_LIB_DIR}/libcrypto_static.lib;${OPENSSL_LIB_DIR}/libssl.lib;${OPENSSL_LIB_DIR}/libcrypto.lib")

that made the difference after appending ;${OPENSSL_LIB_DIR}/libssl.lib;${OPENSSL_LIB_DIR}/libcrypto.lib

I dont know what would be a more robust and elegant way of doing this, but I at least have something that works in my environment and probably other peoples. I used VS 2022 with the latest OpenSSL Win64 OpenSSL v3.3.1 MSI from https://slproweb.com/products/Win32OpenSSL.html

[cosmo0920]

We use to install OpenSSL libraries via vcpkg with x86-windows-static or x64-windows-static or arm64-windows-static triplet in these days.
So, we didn't encountered this type of issues.

[ridwanmsharif]

We're running into this too and we don't have openssl installed though vcpkg. Shouldn't the Fluent Bit build be robust enough that it doesn't require openssl libraries installed via vcpkg

[arasic]

I am reopening this issue as @ridwanmsharif is experiencing this issue and raising a good point.

In the meantime, @ridwanmsharif did you try by applying the change I mentioned earlier?

You might need to do the following as well:

mklink /D "C:\Program Files\OpenSSL-Win64\lib\VC\static" "C:\Program Files\OpenSSL-Win64\lib\VC\x64\MT"

[arasic]

I am closing this issue because its related with how fluent-bit was built from source code. Using exe/msi Win64 OpenSSL v3.3.1 from https://slproweb.com/products/Win32OpenSSL.html does not work. You must use vcpkg to install openssl

Lets consider that VS is already installed (https://visualstudio.microsoft.com/downloads/) and make sure that Desktop development with C++ is selected in installation. After installation is complete, launch Developer Command Prompt for Visual Studio

Lets suppose you install vcpkg in C:\

cd C:\
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg && bootstrap-vcpkg.bat
.\vcpkg.exe integrate install

Now lets install openssl:
from C:\vcpkg, type:

.\vcpkg.exe --triplet x64-windows-static install openssl

Set in the env variables of Windows:
OPENSSL_ROOT_DIR=C:\vcpkg\installed\x64-windows-static

Now to to fluent-bit\build and run:

cmake .. -G "Visual Studio 17 2022" -A x64
cmake --build . --config Release

[braydonk]

I am unable to build Fluent Bit on Windows due to this issue. The instructions in the above comment did not help me. I have tried both installing OpenSSL through vcpkg and through an installer, and I have tried stopping static build of librdkafka entirely, but neither have worked and I always get the libcrypto.lib link failure.

I was going to open my own issue about this, but it would have been a dupe of this issue so I am reopening this instead.

Everything I have tried is in this Fluent Slack thread.

EDIT: I did find a combination of options that will allow me to not build librdkafka so at least I can finally build again.