Strategies for combining multiple Vcpkg roots

[mbeutel]

I would like to use Vcpkg in a multi-user environment. My problem is that there may be a large number of users (students), and storage space is somewhat constrained because all home directories lie on a shared network drive. Just installing Vcpkg itself and Boost on Linux consumes ~1.1 GB of disk space, and that is already after cleaning up the "buildtrees", "packages", and "downloads" directories.

Ideally I would provide a global Vcpkg package root where tools and some common packages are installed, and refer to these from any user-built Vcpkg instances.

There are some approaches I could think of to address this problem:

• It would be very helpful if "--clean-after-build" could be enabled by default by some (global or user-local) configuration option.
• I could use the VCPKG_NO_INSTALL variable introduced by PR Create VCPKG_NO_INSTALL variable #4480 to supply users with a custom triplet file that skips a fixed set of packages known to be available globally. I would also need users to chainload a toolchain file which adjusts CMAKE_PREFIX_PATH so CMake can find those global packages.
• Vcpkg could be made aware of a "global root", or even of a chain of "parent roots":
  • Child Vcpkg would simply skip the installation of packages available in parent Vcpkg. The Vcpkg toolchain would also consider the search directories of parent roots when finding packages.
  • Also, this would permit tools to be reused if available in a parent root directory. A child Vcpkg instance could check whether a tool is available in a parent instance, and symlink the tool directory if it is. Ideally, this would accomodate the common situation where child Vcpkg needs a newer tool version than available in parent Vcpkg (=> child Vcpkg installs the tool), but parent Vcpkg later catches up; if child Vcpkg notices that the tool has become available in parent Vcpkg, it would discard its own copy and symlink the parent tool.

Of course combining different Vcpkg roots is necessarily fragile because Vcpkg doesn't track dependency version information, but I think this is not worse than the common practice of pinning a portfile to a particular package version. Nevertheless, there should be an easy way out if a package in parent Vcpkg is too old.

Any thoughts and comments?

[BillyONeal]

Does the binary caching feature resolve this concern for you? It doesn't reduce overall on-disk footprint if you need multiple roots but it does make it such that common bits need not be rebuilt, and the result is effectively as well as can be done without accepting "extreme footguns" as you enumerated above.

[mbeutel]

We were mostly concerned with storage space, not with build time, so binary caching doesn't really help. However, we since decided that increasing the available storage space is the best solution to this problem, so I'll close this issue.

---

Side note:

We use Spack to manage software and libraries on a small HPC cluster. Spack supports coexistence of multiple versions in one install root, can resolve version-specific dependencies and also supports loading packages from multiple unrelated package roots. However, this proved problematic in practice. Installing a non-trivial package rarely works without some extra effort due to unforeseen incompatibilities between packages. Also, Spack is notorious for almost never reusing already-installed packages; just by occasionally upgrading tools and libraries to the latest version, you tend to end up with a gazillion of different versions (and even identical versions with slightly different flags) of autotools, CMake, zlib, and other basic stuff nobody asked for, and with the root directory approaching a few hundred GB. And Spack won't save you from accidentally linking multiple versions of one library into the same project.

I came to the conclusion that Vcpkg's design is more appropriate for library management. The axiom "there can only be one version of everything" makes continuous testing feasible in the first place. Not allowing to combine multiple roots avoids "footguns" and extra complexity in resolving packages at the expense of some redundant storage (which still compares favorably to the bloat Spack accumulates over time).

For me, the user experience with Vcpkg is mostly "it just works", vs. "broken by default" with Spack.