Support acquiring libmagic from vcpkg. #4119
Conversation
…nor modifications.
This would increase consistency across platforms.
They are only being used on the pcre2grep tool which is not built by the port.
|
This pull request has been linked to Shortcut Story #29902: Support acquiring libmagic from vcpkg.. |
|
Other than the random error on Azure's macOS runner this looks pretty good as far as I can tell. Theoretically I'll have a bit of time this afternoon to take a look and see if I can figure out the issue. Given that the two GHA macOS runners succeed I'm hopefully it'll be an easy fix. |
|
I think I found what causes the macOS failures. Vcpkg normally (unless we tell it otherwise which we don't on macOS) builds the ports in both debug and release configurations. The configure step for both happens concurrently (unless we pass the However, libmagic auto-generates a file in the source directory, and writes to it gradually. When both debug and release configurations write to it at the same time, the file gets corrupted and the |
|
@teo-tsirpanis Awesome find! I duplicated this locally by backing up to before the concurrent build fix, observing the build failure, then including the concurrent build fix and it compiles cleanly for me. |
|
Can we merge it? |
|
@ihnorton I'm gonna leave this decision to you. I'm happy with the PR, but I don't know enough about our downstream packaging/release processes to be certain there won't be an issue with any of that. |
|
Maybe it could not get into 2.16, if we are about to branch it off soon, but this is blocking further work on the build system and vcpkg. |
[SC-38521](https://app.shortcut.com/tiledb-inc/story/38521/update-libmagic-and-use-the-upstream-vcpkg-port) Split from #4553. This PR updates libmagic to version 5.45 and switches to using a vcpkg port closer to the upstream one, which we can easily consume with find_package(unofficial-libmagic) since microsoft/vcpkg#35274. One complication is that the upstream port builds libmagic with its official autotools-based build system, which is significantly slower on Windows (on Linux it builds pretty fast). I tried to upstream the CMake-based port I had added in #4119, but the PR was rejected. Apart from binary caching, there is unfortunately nothing we can do about the build performance regression. We could maintain the CMake-based port for our own use, but it will split what we build and what a future user of TileDB from vcpkg will build, and that port is not without its problems (it failed for example when I tried cross-compiling to arm64-windows, because it tried to execute the arm64 file.exe on my x64 machine). --- TYPE: BUILD DESC: Update libmagic to version 5.45
SC-29902
This PR ports libmagic, the last of our external dependencies to have been exclusively available via
ExternalProject, to vcpkg.The upstream libmagic vcpkg port was unsuitable for our use because it uses
autoconfigandmake, and therefore does not expose CMake targets. For this reason I created a custom CMake-powered port, inspired from both the upstream port and our own fork of libmagic that we have been using.In case
regex.his not available (namely on Windows), the upstream vcpkg port used the tre library. This vcpkg port always uses pcre2 for consistency, matching our fork's behavior. Pcre2's CMake integration had some problems so I added a custom port for it as well.I have also opened PCRE2Project/pcre2#260 to fix pcre2's CMake integration upstream, and after that gets merged I plan to contribute these two custom ports to upstream vcpkg.
It builds locally on Linux; on Windows it has some issues that might or might not occur due to my local configuration. Let's see what CI thinks.
TYPE: IMPROVEMENT
DESC: Support acquiring libmagic from vcpkg.