Remove the superbuild and the external projects.

[teo-tsirpanis]

SC-36913

Historically, the Core's build system has been using CMake external projects to download and build external dependencies, and a "superbuild" architecture to ensure a build order. With the advent of vcpkg, we have stopped building the dependencies ourselves and instead rely on vcpkg (or the "system" in general) to provide them for us. The superbuild has thus became a relic of the past, consisting of only the inner tiledb project when the new system is enabled (formerly by specifying -DTILEDB_VCPKG=ON, now always).

This PR removes the superbuild. TileDB became a regular CMake project, whose targets can be built directly without first building the outer project, and then building the build/tiledb subdirectory. The CI scripts were updated accordingly to not use the subdirectory.

This is inevitably a breaking change in the build system. For starters, local development environment will certainly need to make a clean build after this change. Furthermore, there will need to be changes in build scripts to not build again on the tiledb subdirectory.

For examples of downstream migrations, TileDB-Inc/TileDB-Go#316 uses a make invocation that has a similar effect both with and without the superbuild, and conda-forge/tiledb-feedstock#290 uses a semi-documented CMake option to disable the superbuild (which will have no effect after the superbuild gets removed).

The majority of first-party downstreams (VCF, SOMA, MariaDB, Vector Search, the Python and Java APIs) use the install-tiledb target, which currently is defiend on the superbuild and builds the install target in the inner tiledb project. With this PR the install-tiledb target will be kept for compatibility, but alias to install.

I tried to build VCF with a TileDB external project from this branch, but it fails with an error that will be fixed with #4989. I will try again once that PR gets merged. Never mind, VCF builds with the latest changes.

---

TYPE: BUILD
DESC: The superbuild architecture of the build system has been removed and TileDB is a regular CMake project. Build commands of the form make && make -C tiledb <targets> will have to be replaced by make <targets>.

[KiterLuc]

Was the breaking change announced in any way?

[teo-tsirpanis]

The change will be announced in the release notes. There is unfortunately no way to gradually introduce it (proposals to first make the superbuild off by default and then remove it were rejected), but at the same time the change is quite simple to migrate.

[KiterLuc]

The change will be announced in the release notes. There is unfortunately no way to gradually introduce it (proposals to first make the superbuild off by default and then remove it were rejected), but at the same time the change is quite simple to migrate.

We probably need to announce removal and then actually do the removal in a couple releases. We cannot ship a release that will break dependencies and then expect them to fix it.

[teo-tsirpanis]

Undrafting, regardless of when we will take it, it is finished and ready for review.

---

Also just built successfully¹ libtiledbsoma with just the following change:

diff --git a/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmake b/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmake
index 910fc4e..62bd32a 100644
--- a/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmake
+++ b/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmake
@@ -113,8 +113,8 @@ else()
     else() # Build from source
         ExternalProject_Add(ep_tiledb
           PREFIX "externals"
-          URL "https://github.com/TileDB-Inc/TileDB/archive/2.23.0.zip"
-          URL_HASH SHA1=2acca37618f0a6192ca648930138231476422b96
+          GIT_REPOSITORY "https://github.com/TileDB-Inc/TileDB.git"
+          GIT_TAG teo/remove-superbuild
           DOWNLOAD_NAME "tiledb.zip"
           CMAKE_ARGS
             -DCMAKE_INSTALL_PREFIX=${EP_INSTALL_PREFIX}

Footnotes

1. With some compile errors because Windows, but configuring has been successful. ↩