Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[curl] x64-windows build failure with features brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl after curl Fixes pkgconfig configuration file (#14350) #14681

Closed
StarGate-One opened this issue Nov 20, 2020 · 17 comments · Fixed by #14715
Assignees
Labels
category:port-bug The issue is with a library, which is something the port should already support

Comments

@StarGate-One
Copy link
Contributor

StarGate-One commented Nov 20, 2020

Issue
curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl] build failure after commit 878d7ea [curl] Fixes pkgconfig configuration file (#14350)

Host Environment

  • OS: [Microsoft Windows [Version 10.0.19042.630]
  • Compiler:
    • VS2017 Visual Studio 2017 Version: 15.9.29 VS Tools Version: 14.16.27023
      Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27044 for x64
    • VS2019 Visual Studio 2019 Version: 16.8.2 VS Tools Version: 14.28.29333
      Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29334 for x64

To Reproduce
Steps to reproduce the behavior:

  1. git pull
  2. .\vcpkg.exe install --x-use-aria2 --recurse --keep-going boost[core,mpi]:x64-windows boost-iostreams[core]:x64-windows boost-mpi[core]:x64-windows boost-parameter-python[core]:x64-windows boost-python[core]:x64-windows ctbignum[core]:x64-windows curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]:x64-windows curlpp[core]:x64-windows date[core,remote-api]:x64-windows devil[core,jasper,lcms,libjpeg,libpng,openexr,tiff]:x64-windows directxtex[core,dx12,openexr]:x64-windows forge[core]:x64-windows freeimage[core]:x64-windows libarchive[bzip2,core,libxml2,lz4,lzma,lzo,openssl,zstd]:x64-windows libharu[core,notiffsymbols]:x64-windows liblzma[core]:x64-windows libpq[core,icu,nls,openssl,xml,zlib]:x64-windows libpqxx[core]:x64-windows libwebp[all,anim,core,cwebp,dwebp,extras,gif2webp,img2webp,info,mux,nearlossless,simd,swap16bitcsp,unicode,vwebp,vwebp-sdl]:x64-windows libxml2[core]:x64-windows libzip[bzip2,commoncrypto,core,default-aes,liblzma,mbedtls,openssl,wincrypto]:x64-windows libzippp[core]:x64-windows magnum-plugins[core,ddsimporter,devilimageimporter,drflacaudioimporter,freetypefont,harfbuzzfont,icoimporter,jpegimageconverter,jpegimporter,miniexrimageconverter,openddl,opengeximporter,pngimageconverter,pngimporter,stanfordimporter,stanfordsceneconverter,stbimageconverter,stbimageimporter,stlimporter]:x64-windows ogre[core]:x64-windows openexr[core]:x64-windows poco[core,pdf,postgresql]:x64-windows pybind11[core]:x64-windows python3[core]:x64-windows sdl2-image[core,libjpeg-turbo,libwebp,tiff]:x64-windows sdl2pp[core]:x64-windows shiva[core]:x64-windows tiff[core,tool]:x64-windows tmx[core]:x64-windows wxwidgets[core]:x64-windows

Failure logs

  • Cut and past the appropriate build messages from the console output.
    Starting package 9/34: curl:x64-windows
    Building package curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]:x64-windows...
    -- Using cached D:/vcpkg_x64/vs2017/prod/downloads/curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz
    -- Extracting source D:/vcpkg_x64/vs2017/prod/downloads/curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz
    -- Applying patch 0002_fix_uwp.patch
    -- Applying patch 0004_nghttp2_staticlib.patch
    -- Applying patch 0005_remove_imp_suffix.patch
    -- Applying patch 0006_fix_tool_depends.patch
    -- Applying patch 0007_disable_tool_export_curl_target.patch
    -- Applying patch 0009_fix_openssl_config.patch
    -- Applying patch 0010_fix_othertests_cmake.patch
    -- Applying patch 0011_fix_static_build.patch
    -- Using source at D:/vcpkg_x64/vs2017/prod/buildtrees/curl/src/da0230d937-b280319101.clean
    -- Configuring x64-windows
    -- Building x64-windows-dbg
    -- Building x64-windows-rel
    -- Installing: D:/vcpkg_x64/vs2017/prod/packages/curl_x64-windows/share/curl/curl-config
    -- Using msys root at D:/vcpkg_x64/vs2017/prod/downloads/tools/msys2/17e91aa90d284835
    -- Fixing pkgconfig file: D:/vcpkg_x64/vs2017/prod/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc
    -- CHECK_LIB_optimized D:/vcpkg_x64/vs2017/prod/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE:CHECK_LIB_optimized D:/vcpkg_x64/vs2017/prod/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE-NOTFOUND
    CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message):
    Library "optimized
    D:/vcpkg_x64/vs2017/prod/packages/curl_x64-windows/lib/pthreadVC3.lib" was
    not found! If it is a system library use the SYSTEM_LIBRARIES parameter for
    the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file

    Call Stack (most recent call first):
    scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files)
    ports/curl/portfile.cmake:170 (vcpkg_fixup_pkgconfig)
    scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED
Elapsed time for package curl:x64-windows: 1.169 min

  • Please attach any additional failure logs mentioned in the console output.
    No other logs produced

Additional context
Add any other context about the problem here, such as what you have already tried to resolve the issue.
@PhoebeHui @Matioupi cc: @strega-nil

  • CI Pipeline shows curl passed but did not install features [brotli,c-ares,http2,mbedtls,ssh,tool]
    2020-11-20T05:54:15.3094113Z Starting package 156/237: curl:x64-windows
    2020-11-20T05:54:15.3099330Z Building package curl[core,non-http,openssl,schannel,ssl,sspi,winssl]:x64-windows...
    2020-11-20T05:54:15.3104446Z Building package curl[core,non-http,openssl,schannel,ssl,sspi,winssl]:x64-windows... done
    2020-11-20T05:54:15.3109786Z Installing package curl[core,non-http,openssl,schannel,ssl,sspi,winssl]:x64-windows...
    2020-11-20T05:54:15.3115017Z Installing package curl[core,non-http,openssl,schannel,ssl,sspi,winssl]:x64-windows... done
    2020-11-20T05:54:15.3119841Z Elapsed time for package curl:x64-windows: 48.97 ms
@strega-nil
Copy link
Contributor

strega-nil commented Nov 20, 2020

Looking into this, thanks for reporting @StarGate-One

@strega-nil
Copy link
Contributor

@StarGate-One this doesn't seem to be a real issue; I think it is caused by not updating the libraries after a git pull.

I recommend ./vcpkg x-set-installed to remove all installed libraries, and then re-running the install.

However, when looking at this issue, I did find an issue with libpq: #14685

@StarGate-One
Copy link
Contributor Author

@StarGate-One this doesn't seem to be a real issue; I think it is caused by not updating the libraries after a git pull.

I recommend ./vcpkg x-set-installed to remove all installed libraries, and then re-running the install.

However, when looking at this issue, I did find an issue with libpq: #14685

I am trying on a fresh clone and see what happens.
Normally I rename the buildtrees[port-name] directory(ies) when a port install fails (the port and features ports) and I try again.
I failed do that this time as I was researching the libzippp issue at the same time.
Oh no, not a libpq also 😒
Never a dull moment and job security 👍😊

@StarGate-One
Copy link
Contributor Author

@strega-nil on this one, it still fails with the same issue, as fresh vanilla clone as of a few minutes ago, no previous ports installed.
The only ports installed before port curl failed was:
SYSTEM@ANDROMEDA 11/20/2020 19:16:55 D:\Git\vcpkg\vs2017-x64-v141
->vcpkg list

7zip:x64-windows 19.00#2 Library for archiving file with a high compressi...
brotli:x64-windows 1.0.9 a generic-purpose lossless compression algorithm...
bzip2:x64-windows 1.0.8#1 bzip2 is a freely available, patent free, high-q...
c-ares:x64-windows 2019-5-2-1 A C library for asynchronous DNS requests
dbg-macro:x64-windows 2019-07-11 A dbg(...) macro for C++
gzip-hpp:x64-windows 0.1.0 Gzip header-only C++ library
liblzma:x64-windows 5.2.5#2 Compression library with an API similar to that ...
libssh2:x64-windows 1.9.0 The SSH library
mbedtls:x64-windows 2.24.0#1 An open source, portable, easy to use, readable ...
mbedtls[pthreads]:x64-windows Multi-threading support
nghttp2:x64-windows 1.39.2-2 Implementation of the Hypertext Transfer Protoco...
openssl:x64-windows 1.1.1h#2 OpenSSL is an open source project that provides ...
pthreads:x64-windows 3.0.0-6 pthreads for windows
yasm-tool:x86-windows 2020-10-12 A modular assembler. This port is intended to bu...
zlib:x64-windows 1.2.11#9 A compression library

Console log about the same:
Starting package 11/35: curl:x64-windows
Building package curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]:x64-windows...
-- Downloading curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz...
-- Extracting source D:/Git/vcpkg/vs2017-x64-v141/downloads/curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz
-- Applying patch 0002_fix_uwp.patch
-- Applying patch 0004_nghttp2_staticlib.patch
-- Applying patch 0005_remove_imp_suffix.patch
-- Applying patch 0006_fix_tool_depends.patch
-- Applying patch 0007_disable_tool_export_curl_target.patch
-- Applying patch 0009_fix_openssl_config.patch
-- Applying patch 0010_fix_othertests_cmake.patch
-- Applying patch 0011_fix_static_build.patch
-- Using source at D:/Git/vcpkg/vs2017-x64-v141/buildtrees/curl/src/da0230d937-b280319101.clean
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/share/curl/curl-config
-- Using msys root at D:/Git/vcpkg/vs2017-x64-v141/downloads/tools/msys2/c809757c94447846
-- Fixing pkgconfig file: D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc
-- CHECK_LIB_optimized D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE:CHECK_LIB_optimized D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE-NOTFOUND
CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message):
Library "optimized
D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib"
was not found! If it is a system library use the SYSTEM_LIBRARIES parameter
for the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file
Call Stack (most recent call first):
scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files)
ports/curl/portfile.cmake:170 (vcpkg_fixup_pkgconfig)
scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED
Elapsed time for package curl:x64-windows: 1.199 min
Starting package 12/35: dbg-macro:x64-windows
Building package dbg-macro[core]:x64-windows...

@strega-nil
Copy link
Contributor

@StarGate-One that's really odd... it works completely fine on my machine. Can you try the clean install anyways?

Also, don't worry about renaming buildtrees, that gets cleaned, the concern is installed.

@StarGate-One
Copy link
Contributor Author

  • Well there goes my weekend that just wiped out every installed package I had been installing in a fresh clone/build 😢
  • It seems x-set-installed rebuilds the ports specified on the command line (if installed) or installs them fresh - but the major point to live with it removed all 406 of the other installed ports not specified on the command line.

@StarGate-One
Copy link
Contributor Author

@strega-nil FYI

  • using the x-set-installed did install port curl with the desired features correctly
  • there must be some processing difference how vcpkg install versus vcpkg x-set-installed works
  • using x-set-installed is not a viable option for me:
    • I would need to include 400+ ports on the command line each time there is a change/update
    • leaves the logs but removes the source directories after the installation needed by SOX automated technical code review routines

@StarGate-One
Copy link
Contributor Author

@PhoebeHui @Matioupi cc: @strega-nil

  1. I do not know how to fix, but I figured out the problem is related to mbedtls with feature pthreads
  2. If you install port mbedtls
    • with only feature core curl builds fine with all features
    • with both features core, pthreads curl fails as described above
  3. Either method of install fails with same issue:
    • .\vcpkg.exe x-set-installed c-ares curl[brotli,c-ares,http2,mbedtls,non-http,openssl,ssh,ssl,sspi,tool,winssl] libssh2 mbedtls[core,pthreads] nghttp2 openssl pthreads zlib
    • .\vcpkg.exe install --x-use-aria2 --recurse --keep-going -- c-ares curl[brotli,c-ares,http2,mbedtls,non-http,openssl,ssh,ssl,sspi,tool,winssl] libssh2 mbedtls[core,pthreads] nghttp2 openssl pthreads zlib
  4. Error for both as follows:

-- Fixing pkgconfig file: D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc
-- CHECK_LIB_optimized D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE:CHECK_LIB_optimized D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE-NOTFOUND
CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message):
Library "optimized
D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib"
was not found! If it is a system library use the SYSTEM_LIBRARIES parameter
for the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file
Call Stack (most recent call first):
scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files)
ports/curl/portfile.cmake:170 (vcpkg_fixup_pkgconfig)
scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: curl:x64-windows
Vcpkg version: 2020.11.12-nohash

@Matioupi
Copy link
Contributor

Hello, this looks like the same kind or error message I had in adding "fix" cases for the libcurl.pc file. Still I missed testinf those features and adding the same kind of search & replace fix. Will take care of it today. (and maybe refactor a little bit to make it more generic)

Matioupi added a commit to Matioupi/vcpkg that referenced this issue Nov 21, 2020
@Matioupi
Copy link
Contributor

Hello,

This commit : #14681

Fixes the issue (only handle and tested for windows case at time, more work needed for other platforms).

I decided to remove the -loptimized / -ldebug that were in the package stage libcurl.pc. Maybe a better fix would be to add those in the vcpkg_fixup_pkgconfig calls at the end of the portfile.cmake

It also seems that the release/debug libcurl.pc at the package stage contains both the debug/release versions of the pthread libraries which is weird.

@StarGate-One
Copy link
Contributor Author

OK - I got the fix from commit Matioupi@236dae5 PR First fix attempts for #14681 and got a new error😢 with zlib with brotlidec.lib and brotlicommon lib.
Unfortunately I would guess we end up fixing all the features.

vcpkg install curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]

-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/share/curl/curl-config
-- Using msys root at D:/vcpkg_x64/vs2019/prod/downloads/tools/msys2/17e91aa90d284835
-- Fixing pkgconfig file: D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc
>-- CHECK_LIB_zlib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib_RELEASE:CHECK_LIB_zlib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib_RELEASE-NOTFOUND
CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message):
Library "zlib
D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib
D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib" was
not found! If it is a system library use the SYSTEM_LIBRARIES parameter for
the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file
Call Stack (most recent call first):
scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files)
ports/curl/portfile.cmake:178 (vcpkg_fixup_pkgconfig)
scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: curl:x64-windows

Matioupi added a commit to Matioupi/vcpkg that referenced this issue Nov 21, 2020
@Matioupi
Copy link
Contributor

More fix here : Matioupi@fe90a39

@Matioupi
Copy link
Contributor

I just tested a linux build and get no errors with the specific features and build order that were used. Still the result libcurl.pc do not contain references to brotli libs even when they are present and the feature was activated. Not sure if this requires a specific fix or not.

@Matioupi
Copy link
Contributor

Also, shouldn't the CONTROL section about brotli feature be changed from

Feature: brotli
Description: brotli support (brotli)

to

Feature: brotli
Build-Depends: brotli
Description: brotli support (brotli)

in order to force brotli build when the feature is activated ? is there a rationale for not having it ?

@StarGate-One
Copy link
Contributor Author

@Matioupi

  • The latest change to portfile.cmake fixes the issue - I was able to build/compile/link/install with these options without issue:
    curl[brotli,c-ares,http2,mbedtls,non-http,openssl,ssh,ssl,sspi,tool,winssl] on both VS2017/VS2019 x64-windows
  • The other questions, I am not sure about. There has been so many changes lately I am unable to keep up with what is/is not required in the CONTROL or vcpkg.json file(s).

Thank you for the hard work👍👍👍😊😊😊

@PhoebeHui PhoebeHui added the category:port-bug The issue is with a library, which is something the port should already support label Nov 23, 2020
@PhoebeHui
Copy link
Contributor

@StarGate-One, thanks for reporting this issue and figure out the repro steps!

@Matioupi, could you help summit the fix?

@Matioupi
Copy link
Contributor

@PhoebeHui : I opened a new PR with the proposed fix : Phoe #14715

strega-nil pushed a commit that referenced this issue Nov 23, 2020
* Fix pkgconfig for windows debug build. Re-allow tool feature on non windows platforms

* Fix tool feature

* Modified debug/release prvate libs according to #14350 (comment)
Added vcpkg_fixup_pkgconfig() according to #14350 (comment)

* Delete portfile.cmake.orig

* Fix pkgconfig

* Revert "Fix pkgconfig"

This reverts commit cb4da8b.

* Bump port version to 3

* add missing system libs for windows. made case type coherent for ws2_32

* Fix build on x64-linux triplet

* Reverted ws2_32 to Ws2_32 as discudssed in #14350 (comment)

* More fixes for Linux build with features

* Fix for feature c-ares

* Formatting

* First fix attemps for #14681

* More fix for #14681 (Windows only for now)

* Add brotli dependcy when brotli feature is enabled

* Fixed indentation

Co-authored-by: Phoebe Ma <v-phma@microsoft.com>
Sungeun0318 pushed a commit to Sungeun0318/-vcpkg that referenced this issue May 15, 2024
* Fix pkgconfig for windows debug build. Re-allow tool feature on non windows platforms

* Fix tool feature

* Modified debug/release prvate libs according to microsoft/vcpkg#14350 (comment)
Added vcpkg_fixup_pkgconfig() according to microsoft/vcpkg#14350 (comment)

* Delete portfile.cmake.orig

* Fix pkgconfig

* Revert "Fix pkgconfig"

This reverts commit cb4da8b.

* Bump port version to 3

* add missing system libs for windows. made case type coherent for ws2_32

* Fix build on x64-linux triplet

* Reverted ws2_32 to Ws2_32 as discudssed in microsoft/vcpkg#14350 (comment)

* More fixes for Linux build with features

* Fix for feature c-ares

* Formatting

* First fix attemps for microsoft/vcpkg#14681

* More fix for microsoft/vcpkg#14681 (Windows only for now)

* Add brotli dependcy when brotli feature is enabled

* Fixed indentation

Co-authored-by: Phoebe Ma <v-phma@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:port-bug The issue is with a library, which is something the port should already support
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants