Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vcpkg_fixup_pkgconfig] Remove required in first find_program call for pkg-config #12569

Merged
merged 3 commits into from
Jul 31, 2020

Conversation

Neumann-A
Copy link
Contributor

@Neumann-A Neumann-A commented Jul 25, 2020

Due to cmake 3.18 actually respecting it. (Editted from mobile, currently on vacation without wlan access)
closes #12565

@Neumann-A Neumann-A changed the title Remove required in first find_program call [vcpkg_fixup_pkgconfig] Remove required in first find_program call for pkg-config Jul 25, 2020
@cenit
Copy link
Contributor

cenit commented Jul 26, 2020

@Neumann-A there is an error in the paths you are looking into, also
"Cellar" folders should not be used, any brew upgrade might move, rename, do whatever they want.
For pkg-config on Mac you should look for /usr/local/bin/pkg-config, which means a pkg-config executable inside /usr/local/bin

@PhoebeHui PhoebeHui self-assigned this Jul 27, 2020
@JackBoosY JackBoosY self-assigned this Jul 27, 2020
@JackBoosY JackBoosY added the category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly label Jul 27, 2020
@JackBoosY
Copy link
Contributor

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@JackBoosY JackBoosY linked an issue Jul 27, 2020 that may be closed by this pull request
@JackBoosY JackBoosY linked an issue Jul 27, 2020 that may be closed by this pull request
@JackBoosY JackBoosY added the info:reviewed Pull Request changes follow basic guidelines label Jul 27, 2020
@cenit
Copy link
Contributor

cenit commented Jul 27, 2020

@Neumann-A if there is not a special reason to keep it, I'd really remove the Cellar folder from the search path. It will become old in a couple of weeks at maximum (we cannot and should not chase it)

@ras0219
Copy link
Contributor

ras0219 commented Jul 29, 2020

Starting package 257/684: eigen3:x64-windows
Building package eigen3[core]:x64-windows...
Could not locate cached archive: W:\f1\f12b450bd980be77228d68337f662988eb16ef9f.zip
-- Downloading https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.tar.gz...
-- Downloading https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.tar.gz... Failed. Status: 35;"SSL connect error"

The x64_windows failure is due to a download error and should not block this PR.

@JackBoosY
Copy link
Contributor

@ras0219 I'll rerun the failed test after x64-osx test finish.

@JackBoosY
Copy link
Contributor

JackBoosY commented Jul 29, 2020

itk:x64-osx

[1621/1622] : && /Library/Developer/CommandLineTools/usr/bin/c++  -fPIC  -mtune=native -march=corei7 -Wall -Wcast-align -Wdisabled-optimization -Wextra -Wformat=2 -Winvalid-pch -Wno-format-nonliteral -Wpointer-arith -Wshadow -Wunused -Wwrite-strings -funit-at-a-time -Wno-strict-overflow -Wno-deprecated -Wno-invalid-offsetof -Wno-undefined-var-template -Woverloaded-virtual  -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fPIE -Xlinker -pie Modules/Core/TestKernel/src/CMakeFiles/itkTestDriver.dir/itkTestDriver.cxx.o  -o bin/itkTestDriver  lib/libITKIOBMP-5.0.a  lib/libITKIOGDCM-5.0.a  lib/libITKIOGIPL-5.0.a  lib/libITKIOJPEG-5.0.a  lib/libITKIOMeshBYU-5.0.a  lib/libITKIOMeshFreeSurfer-5.0.a  lib/libITKIOMeshGifti-5.0.a  lib/libITKIOMeshOBJ-5.0.a  lib/libITKIOMeshOFF-5.0.a  lib/libITKIOMeshVTK-5.0.a  lib/libITKIOMeta-5.0.a  lib/libITKIONIFTI-5.0.a  lib/libITKIONRRD-5.0.a  lib/libITKIOPNG-5.0.a  lib/libITKIOTIFF-5.0.a  lib/libITKIOVTK-5.0.a  lib/libITKCommon-5.0.a  lib/libITKIOImageBase-5.0.a  lib/libITKTestKernel-5.0.a  lib/libITKIOBMP-5.0.a  lib/libITKIOGDCM-5.0.a  lib/libitkgdcmMSFF-5.0.a  lib/libitkgdcmDICT-5.0.a  lib/libitkgdcmIOD-5.0.a  lib/libitkgdcmDSED-5.0.a  lib/libitkgdcmCommon-5.0.a  -framework  CoreFoundation  lib/libitkgdcmjpeg8-5.0.a  lib/libitkgdcmjpeg12-5.0.a  lib/libitkgdcmjpeg16-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libopenjp2.a  lib/libitkgdcmcharls-5.0.a  lib/libitkgdcmuuid-5.0.a  lib/libITKIOGIPL-5.0.a  lib/libITKIOJPEG-5.0.a  lib/libITKIOMeshBYU-5.0.a  lib/libITKIOMeshFreeSurfer-5.0.a  lib/libITKIOMeshGifti-5.0.a  lib/libITKgiftiio-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libexpat.a  lib/libITKIOMeshOBJ-5.0.a  lib/libITKIOMeshOFF-5.0.a  lib/libITKIOMeshVTK-5.0.a  lib/libITKIOMeshBase-5.0.a  lib/libITKQuadEdgeMesh-5.0.a  lib/libITKMesh-5.0.a  lib/libITKIOMeta-5.0.a  lib/libITKMetaIO-5.0.a  lib/libITKIONIFTI-5.0.a  lib/libITKniftiio-5.0.a  lib/libITKznz-5.0.a  lib/libITKTransform-5.0.a  lib/libITKIONRRD-5.0.a  lib/libITKNrrdIO-5.0.a  lib/libITKIOPNG-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libpng16d.a  lib/libITKIOTIFF-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libtiffd.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/liblzmad.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libz.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libjpeg.a  lib/libITKIOVTK-5.0.a  lib/libITKIOImageBase-5.0.a  lib/libITKCommon-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libdouble-conversion.a  lib/libitksys-5.0.a  lib/libITKVNLInstantiation-5.0.a  lib/libitkvnl_algo-5.0.a  lib/libitkvnl-5.0.a  lib/libitkv3p_netlib-5.0.a  lib/libitknetlib-5.0.a  lib/libitkvcl-5.0.a  -lm  -lm && :
FAILED: bin/itkTestDriver 
: && /Library/Developer/CommandLineTools/usr/bin/c++  -fPIC  -mtune=native -march=corei7 -Wall -Wcast-align -Wdisabled-optimization -Wextra -Wformat=2 -Winvalid-pch -Wno-format-nonliteral -Wpointer-arith -Wshadow -Wunused -Wwrite-strings -funit-at-a-time -Wno-strict-overflow -Wno-deprecated -Wno-invalid-offsetof -Wno-undefined-var-template -Woverloaded-virtual  -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fPIE -Xlinker -pie Modules/Core/TestKernel/src/CMakeFiles/itkTestDriver.dir/itkTestDriver.cxx.o  -o bin/itkTestDriver  lib/libITKIOBMP-5.0.a  lib/libITKIOGDCM-5.0.a  lib/libITKIOGIPL-5.0.a  lib/libITKIOJPEG-5.0.a  lib/libITKIOMeshBYU-5.0.a  lib/libITKIOMeshFreeSurfer-5.0.a  lib/libITKIOMeshGifti-5.0.a  lib/libITKIOMeshOBJ-5.0.a  lib/libITKIOMeshOFF-5.0.a  lib/libITKIOMeshVTK-5.0.a  lib/libITKIOMeta-5.0.a  lib/libITKIONIFTI-5.0.a  lib/libITKIONRRD-5.0.a  lib/libITKIOPNG-5.0.a  lib/libITKIOTIFF-5.0.a  lib/libITKIOVTK-5.0.a  lib/libITKCommon-5.0.a  lib/libITKIOImageBase-5.0.a  lib/libITKTestKernel-5.0.a  lib/libITKIOBMP-5.0.a  lib/libITKIOGDCM-5.0.a  lib/libitkgdcmMSFF-5.0.a  lib/libitkgdcmDICT-5.0.a  lib/libitkgdcmIOD-5.0.a  lib/libitkgdcmDSED-5.0.a  lib/libitkgdcmCommon-5.0.a  -framework  CoreFoundation  lib/libitkgdcmjpeg8-5.0.a  lib/libitkgdcmjpeg12-5.0.a  lib/libitkgdcmjpeg16-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libopenjp2.a  lib/libitkgdcmcharls-5.0.a  lib/libitkgdcmuuid-5.0.a  lib/libITKIOGIPL-5.0.a  lib/libITKIOJPEG-5.0.a  lib/libITKIOMeshBYU-5.0.a  lib/libITKIOMeshFreeSurfer-5.0.a  lib/libITKIOMeshGifti-5.0.a  lib/libITKgiftiio-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libexpat.a  lib/libITKIOMeshOBJ-5.0.a  lib/libITKIOMeshOFF-5.0.a  lib/libITKIOMeshVTK-5.0.a  lib/libITKIOMeshBase-5.0.a  lib/libITKQuadEdgeMesh-5.0.a  lib/libITKMesh-5.0.a  lib/libITKIOMeta-5.0.a  lib/libITKMetaIO-5.0.a  lib/libITKIONIFTI-5.0.a  lib/libITKniftiio-5.0.a  lib/libITKznz-5.0.a  lib/libITKTransform-5.0.a  lib/libITKIONRRD-5.0.a  lib/libITKNrrdIO-5.0.a  lib/libITKIOPNG-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libpng16d.a  lib/libITKIOTIFF-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libtiffd.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/liblzmad.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libz.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libjpeg.a  lib/libITKIOVTK-5.0.a  lib/libITKIOImageBase-5.0.a  lib/libITKCommon-5.0.a  /Users/vagrant/Data/installed/x64-osx/debug/lib/libdouble-conversion.a  lib/libitksys-5.0.a  lib/libITKVNLInstantiation-5.0.a  lib/libitkvnl_algo-5.0.a  lib/libitkvnl-5.0.a  lib/libitkv3p_netlib-5.0.a  lib/libitknetlib-5.0.a  lib/libitkvcl-5.0.a  -lm  -lm && :
Undefined symbols for architecture x86_64:
  "gdcm::Rescaler::SetMinMaxForPixelType(double, double)", referenced from:
      itk::GDCMImageIO::Write(void const*) in libITKIOGDCM-5.0.a(itkGDCMImageIO.cxx.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Your PR #11208 should fix this issue, please add Supports: !osx to port/itk/CONTROL temporary.

@PhoebeHui
Copy link
Contributor

@strega-nil, could you help merge this PR firstly?

@kcgen
Copy link

kcgen commented Jul 31, 2020

Thank you @JackBoosY, @PhoebeHui, and @strega-nil!

Looking forward to this being fixed as our CI chain has been down since this started, so test results are also down, and that's a pre-req for merging code. Talk about dominoes eh? Thank you again!

@kcgen
Copy link

kcgen commented Jul 31, 2020

The ripple effect from this has affected many projects, as cmake and zlib are some of the most commonly used/linked base packages. This is revealing how critical vspkg is to commercial and open-source projects:

Once its fixed, hopefully some test cases can be added as hard gates in your CI chain to catch this going forward.

@strega-nil strega-nil merged commit e299457 into microsoft:master Jul 31, 2020
@strega-nil
Copy link
Contributor

@kcgen thank you for being invested in vcpkg; hopefully, this fixes this issue. I'm not sure how to test this well, since it was caused by an update to CMake; we should look into that.

@kcgen
Copy link

kcgen commented Jul 31, 2020

Thank you @strega-nil ! Yes; I wonder if testing the installation of some known-stable packages that rely on cmake would catch it.

For example, vspkg install zlib: would that pull in new updates to CMake (and fail hard), similar to how the recent updates broke in this issue?

@cenit
Copy link
Contributor

cenit commented Jul 31, 2020

it was caused by an update to CMake

there is a PR in which CMake 3.18 is already tested, it should highlight any remaining problem #12612

@cenit
Copy link
Contributor

cenit commented Jul 31, 2020

would that pull in new updates to CMake (and fail hard), similar to how the recent updates broke in this issue?

no, vcpkg is still using cmake 3.17 internally if not found on the system. #12612 will upgrade it

@kcgen
Copy link

kcgen commented Jul 31, 2020

Thanks @cenit; that explains the difficulty in avoiding this exact issue from happening again in the future, like @strega-nil noted.

  1. July 31st - Vcpkg updated to internally use cmake 3.18
  2. Aug 1st - vcpkg users w/ cmake 1.18 installed rejoice that vcpkg works again.
  3. Aug 2nd - cmake 1.19 is released, and those same users upgrade.
  4. Aug 2nd - vcpkg again starts using the host cmake instead of its internal version, and again starts falling over because something's changed in 1.19.
  5. User issues pour into vxpkg, and it's another scramble for a week to sort through it.

I suggest two changes to address this:

  1. Only use the host-installed cmake if it's version is equal to or less than vxpkg's internal (proven-tested) version.
  2. Provide a command line flag to ignore that safety rule, so users can still force vcpkg to try the host-installed cmake even if it's beyond the proven-tested version.

@cenit
Copy link
Contributor

cenit commented Jul 31, 2020

Another way would be to review scripts: this REQUIRED keyword was really misplaced from the beginning for WIN32, since this line was supposed to fail there...

@snnn
Copy link
Member

snnn commented Jul 31, 2020

Only use the host-installed cmake if it's version is equal to or less than vxpkg's internal (proven-tested) version.

Sounds good. But it will also change how Azure DevOps VM image builder works. They need to install cmake and vcpkg together, and ensure cmake version matches.

@Neumann-A Neumann-A deleted the patch-7 branch July 31, 2020 21:04
@JackBoosY
Copy link
Contributor

For the update of these tools, we should add a test port pkgconfig-test to the dependencies of the ports that use them. When these tools are updated, update the version of this test port to test all affected ports.

@Neumann-A
Copy link
Contributor Author

@JackBoosY this is no longer required since script changes rebuilds all affected ports. The CI logic was changed before the PR was merged and it was tested with all affected ports! The problem here is that vcpkg is not testing with the newest cmake version and it thus "slipped" through CI.

@cenit
Copy link
Contributor

cenit commented Aug 3, 2020

Also, now there is no failure in case pkg-config is not found, on Linux/macOS...

@Neumann-A
Copy link
Contributor Author

@cenit is that a problem for you? For me it is theoretically enough if the check is run in the VCPKG CI (or if you do something similar)

@cenit
Copy link
Contributor

cenit commented Aug 3, 2020

It's just for user experience

hellozee pushed a commit to hellozee/vcpkg that referenced this pull request Sep 11, 2020
…r pkg-config (microsoft#12569)

* Remove required in first find_program call

* add usr/local/bin for mac

* lets try without cellar and see if osx ci agrees
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:reviewed Pull Request changes follow basic guidelines
Projects
None yet
8 participants