CI: Fix MacOS vcpkg libgeotiff build #69

brendan-ward · 2022-04-08T19:53:39Z

No description provided.

brendan-ward · 2022-04-08T22:55:02Z

Something about the update of libgeotiff from 1.7.0#1 to 1.7.1 is breaking on MacOS on CI, even though it builds fine for me on local MacOS 10.15:

Contents of libgeotiff/config-x64-osx-dynamic-rel-err.log:

CMake Error at /usr/local/share/vcpkg/installed/x64-osx-dynamic/share/unofficial-sqlite3/unofficial-sqlite3-targets.cmake:55 (set_target_properties):
  The link interface of target "unofficial::sqlite3::sqlite3" contains:

    Threads::Threads

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  /usr/local/share/vcpkg/installed/x64-osx-dynamic/share/unofficial-sqlite3/unofficial-sqlite3-config.cmake:7 (include)
  /usr/local/share/vcpkg/scripts/buildsystems/vcpkg.cmake:816 (_find_package)
  /usr/local/Cellar/cmake/3.23.0/share/cmake/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /usr/local/share/vcpkg/installed/x64-osx-dynamic/share/proj/proj-config.cmake:13 (find_dependency)
  /usr/local/share/vcpkg/scripts/buildsystems/vcpkg.cmake:816 (_find_package)
  CMakeLists.txt:118 (FIND_PACKAGE)

jorisvandenbossche · 2022-04-09T07:54:01Z

I can't directly help with the build failure on Mac (I would maybe report the build failure upstream at https://github.com/microsoft/vcpkg, since it worked before; it's a bit a pity that the dynamic osx build it not one of the built-in triplets, which would make it less likely to get broken).

But, maybe we should try to pin the build to an exact "baseline" for which we know it works (https://devblogs.microsoft.com/cppblog/take-control-of-your-vcpkg-dependencies-with-versioning-support/#baselines), and then only from time to time manually update the version.

brendan-ward · 2022-04-13T01:06:40Z

This should now be using the vcpkg manifest approach for all builds, which includes a baseline commit in the vcpkg repository (set to just before libgeotiff 1.7.1 was merged), and a pinned version of libgeotiff that works. We'll occasionally need to bump this version / baseline commit.

The instructions weren't particularly clear; given the basline that I set, it shouldn't have tried to install libgeotiff 1.7.1 at all, but it did so until I added an override to pin that to a prior version.

I couldn't reproduce the libgeotiff error locally, so hopefully that will be fixed at a later point. I'm not seeing anything obvious to call attention to in the vcpkg setup for it, so I'm not sure what we would log out there as an issue other than it doesn't build in our specific environment.

brendan-ward · 2022-04-20T18:29:34Z

Since this worked on last test of building the wheels, I'm going to go ahead and merge this so we can get closer to cutting an 0.4.0 release that builds wheels. We can add future PRs to fix things here if needed.

jorisvandenbossche · 2022-04-20T19:47:26Z

Sorry for the late review here, and thanks a lot for fixing this! Adding a few questions inline.

jorisvandenbossche · 2022-04-20T19:51:58Z

In any case it is a nice solution!

brendan-ward · 2022-04-20T19:55:26Z

Until it broke on CI, at time of merge. Because CI is... fun... (investigating now, further fixes will be a in a different PR)

jorisvandenbossche · 2022-04-20T19:49:17Z

.github/workflows/release.yml

        run: |
-          vcpkg install gdal[core] --overlay-triplets=./ci/custom-triplets
+          echo "VCPKG_INSTALLATION_ROOT: ${VCPKG_INSTALLATION_ROOT}"


Is this a debugging left-over? (the variable is documented by github actions about what it is)

jorisvandenbossche · 2022-04-20T19:50:47Z

ci/manylinux2014_x86_64-vcpkg-gdal.Dockerfile

+    --overlay-ports=opt/vcpkg/custom-ports \
+    --feature-flags="versions,manifests" \
+    --x-manifest-root=opt/vcpkg \
+    --x-install-root=opt/vcpkg/installed && \


Question: the reason that it is now needed to mention the install-root, is that because using manifest mode changes that? (it would now try to install inside the project, instead of in the vcpkg root?)

Yes, using manifest mode changes this, and would have put it in ci/vcpkg_installed/ which then would have required other updates here to use that path.

jorisvandenbossche · 2022-04-20T19:51:37Z

ci/vcpkg.json

+    "overrides": [
+        {
+            "name": "libgeotiff",
+            "version": "1.7.0"


I would have expected that only the baseline would be sufficient? (assuming the baseline is from a commit where libgeotiff is still at 1.7.0?)

Provided I did it correctly (I picked commit immediately prior to 1.7.0 being merged), I would have expected the same. However, I found it went ahead and tried to install 1.7.1 anyway, without setting this override.

I don't know that it is necessarily a problem for us to explicitly list it as a pinned version though - if only for documentation purposes, since this was the root of the problem.

I don't know that it is necessarily a problem for us to explicitly list it as a pinned version though - if only for documentation purposes, since this was the root of the problem.

Indeed

@jorisvandenbossche @brendan-ward I am finding vcpkg manifests difficult to use. Do you know if there is a vcpkg mode that will fail if no vcpkg.json is found? I suspect it's continuing without one in my case, but can't verify that.

@sgillies I think you are missing a "manifests" feature flag. Assuming this is about rasterio/rasterio-wheels#90, will comment over there

jorisvandenbossche · 2022-04-20T19:58:31Z

Until it broke on CI, at time of merge. Because CI is... fun... (investigating now, further fixes will be a in a different PR)

But the Mac build passed :)

jorisvandenbossche · 2022-04-20T19:59:04Z

The GDAL data files probably changed again location on windows ..

brendan-ward added 4 commits April 8, 2022 12:52

Itemize log files to include

6f1bb6b

Try again with actions/upload-artifact@v3

bdc77f1

temp: comment out other steps

ffd8ce4

Try again...

248ce2f

brendan-ward added 7 commits April 11, 2022 10:23

Try using pinned version of vcpkg for build

3ca4a12

try with pinned libgeotiff and env vars

42ad040

only use CLI vars

d9a9a65

try again with --x-install-root

c069805

Re-enable windows build and wheel tests, try pinned version for Linux

a1e1e6d

Add missing updates to release.yml from last commit

080889f

Remove version pinning from Linux build

1b024d1

brendan-ward marked this pull request as ready for review April 12, 2022 15:30

brendan-ward added 2 commits April 12, 2022 08:37

Try again for Linux with all parameters this time

47b3fbf

Try again with triplet on command line

5571ac9

brendan-ward marked this pull request as draft April 12, 2022 16:23

brendan-ward added 3 commits April 12, 2022 09:50

Only use triplet flag on Linux

cc5ae8e

Delete commented line in vcpkg dockerfile

59f6c8d

List outputs after install, formatting

e84bbb4

brendan-ward marked this pull request as ready for review April 13, 2022 01:01

brendan-ward requested a review from jorisvandenbossche April 13, 2022 01:12

Remove unneeded gitignore item

e41e9dd

brendan-ward merged commit 9ed3e5a into main Apr 20, 2022

brendan-ward changed the title ~~CI: Try to fix MacOS vcpkg libgeotiff build~~ CI: Fix MacOS vcpkg libgeotiff build Apr 20, 2022

brendan-ward deleted the fix_macos_vckpkg_libgeotiff branch April 20, 2022 18:32

jorisvandenbossche reviewed Apr 20, 2022

View reviewed changes

jorisvandenbossche mentioned this pull request Apr 21, 2022

CI: Fix vcpkg Windows and Linux builds #77

Merged

jorisvandenbossche mentioned this pull request Sep 29, 2022

Test build of 34c0dd5 rasterio/rasterio-wheels#90

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Fix MacOS vcpkg libgeotiff build #69

CI: Fix MacOS vcpkg libgeotiff build #69

brendan-ward commented Apr 8, 2022

brendan-ward commented Apr 8, 2022

jorisvandenbossche commented Apr 9, 2022

brendan-ward commented Apr 13, 2022

brendan-ward commented Apr 20, 2022

jorisvandenbossche commented Apr 20, 2022 •

edited

jorisvandenbossche commented Apr 20, 2022

brendan-ward commented Apr 20, 2022

jorisvandenbossche Apr 20, 2022

jorisvandenbossche Apr 20, 2022

brendan-ward Apr 20, 2022

jorisvandenbossche Apr 20, 2022

brendan-ward Apr 20, 2022

jorisvandenbossche Apr 20, 2022

sgillies Sep 28, 2022

jorisvandenbossche Sep 29, 2022

jorisvandenbossche commented Apr 20, 2022

jorisvandenbossche commented Apr 20, 2022 •

edited

CI: Fix MacOS vcpkg libgeotiff build #69

CI: Fix MacOS vcpkg libgeotiff build #69

Conversation

brendan-ward commented Apr 8, 2022

brendan-ward commented Apr 8, 2022

jorisvandenbossche commented Apr 9, 2022

brendan-ward commented Apr 13, 2022

brendan-ward commented Apr 20, 2022

jorisvandenbossche commented Apr 20, 2022 • edited

jorisvandenbossche commented Apr 20, 2022

brendan-ward commented Apr 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Apr 20, 2022

jorisvandenbossche commented Apr 20, 2022 • edited

jorisvandenbossche commented Apr 20, 2022 •

edited

jorisvandenbossche commented Apr 20, 2022 •

edited