Setup new tiledb_experimental c-api headers

[Shelnutt2]

Array Schema evolution is the first use of this new c-api header. The goal is to allow introducing new features into TileDB before we stabilize the c-api/cpp-api. Experimental APIs allow for modifications between TileDB versions. This allows for real world use to determine if the API achieves what users are requesting.

On disk format is ALWAYS guaranteed.

---

TYPE: C_API
DESC: Introduce new `tiledb_experimental.h` c-api header for new feature that don't have a stabilized api yet

[shortcut-integration]

This pull request has been linked to Clubhouse Story #9806: Create tiledb_experimental.h and move schema evolution api there.

[eddelbuettel]

I like the idea quite a bit, but what is a little different right now is that it will require build changes in clients. Until now, wiith published APIs, we can go via the installed headers (from the standard make install step). No C headers are installed that way.

With this, at least for now, we can only work from source. That is a stark difference, but this "bug" could be "feature" as a more explicit buy-in is required from the user or developer.

[Shelnutt2]

I like the idea quite a bit, but what is a little different right now is that it will require build changes in clients. Until now, wiith published APIs, we can go via the installed headers (from the standard make install step). No C headers are installed that way.

With this, at least for now, we can only work from source. That is a stark difference, but this "bug" could be "feature" as a more explicit buy-in is required from the user or developer.

C headers are installed with a standard TileDB installation, this doesn't change anything with the installation pattern. It only introduces a new header which can be optionally included by end users/api/applications to access the new features which we might change the API of because they are new. If the c-api (and soon cpp) user only wants "stable" APIs they don't change anything

[eddelbuettel]

C headers are installed with a standard TileDB installation

Can you remind me where they go after we do make -C tiledb install? When I look in the location I choose, typically /usr/local, I do not have them.

dev as of right now:

[... earlier build lines omitted for brevity ...]
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libtiledb.so.2.4
-- Set runtime path of "/usr/local/lib/libtiledb.so.2.4" to "/usr/local/lib"
-- Up-to-date: /usr/local/lib/libtiledb.so
-- Installing: /usr/local/include/tiledb/tiledb.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_enum.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_version.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_serialization.h
-- Up-to-date: /usr/local/include/tiledb/tiledb
-- Up-to-date: /usr/local/include/tiledb/array.h
-- Up-to-date: /usr/local/include/tiledb/array_schema.h
-- Up-to-date: /usr/local/include/tiledb/attribute.h
-- Up-to-date: /usr/local/include/tiledb/config.h
-- Up-to-date: /usr/local/include/tiledb/context.h
-- Up-to-date: /usr/local/include/tiledb/core_interface.h
-- Up-to-date: /usr/local/include/tiledb/deleter.h
-- Up-to-date: /usr/local/include/tiledb/dimension.h
-- Up-to-date: /usr/local/include/tiledb/domain.h
-- Up-to-date: /usr/local/include/tiledb/exception.h
-- Up-to-date: /usr/local/include/tiledb/filter.h
-- Up-to-date: /usr/local/include/tiledb/filter_list.h
-- Up-to-date: /usr/local/include/tiledb/fragment_info.h
-- Up-to-date: /usr/local/include/tiledb/group.h
-- Up-to-date: /usr/local/include/tiledb/object.h
-- Up-to-date: /usr/local/include/tiledb/object_iter.h
-- Up-to-date: /usr/local/include/tiledb/query.h
-- Up-to-date: /usr/local/include/tiledb/query_condition.h
-- Up-to-date: /usr/local/include/tiledb/schema_base.h
-- Up-to-date: /usr/local/include/tiledb/stats.h
-- Up-to-date: /usr/local/include/tiledb/type.h
-- Up-to-date: /usr/local/include/tiledb/utils.h
-- Up-to-date: /usr/local/include/tiledb/version.h
-- Up-to-date: /usr/local/include/tiledb/vfs.h
-- Up-to-date: /usr/local/include/tiledb/arrowio
-- Up-to-date: /usr/local/include/tiledb/arrow_io_impl.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_export.h
-- Up-to-date: /usr/local/lib/cmake/TileDB/TileDBConfig.cmake
-- Up-to-date: /usr/local/lib/cmake/TileDB/TileDBTargets.cmake
-- Installing: /usr/local/lib/cmake/TileDB/TileDBTargets-release.cmake
-- Up-to-date: /usr/local/lib/pkgconfig/tiledb.pc
make: Leaving directory '/home/ubuntu/git/tiledb/build/tiledb'
ubuntu@aws:~/git/tiledb/build(dev)$ 

[Shelnutt2]

C headers are installed with a standard TileDB installation

Can you remind me where they go after we do make -C tiledb install? When I look in the location I choose, typically /usr/local, I do not have them.

dev as of right now:

Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libtiledb.so.2.4
-- Set runtime path of "/usr/local/lib/libtiledb.so.2.4" to "/usr/local/lib"
-- Up-to-date: /usr/local/lib/libtiledb.so
-- Installing: /usr/local/include/tiledb/tiledb.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_enum.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_version.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_serialization.h
-- Up-to-date: /usr/local/include/tiledb/tiledb
-- Up-to-date: /usr/local/include/tiledb/array.h
-- Up-to-date: /usr/local/include/tiledb/array_schema.h
-- Up-to-date: /usr/local/include/tiledb/attribute.h
-- Up-to-date: /usr/local/include/tiledb/config.h
-- Up-to-date: /usr/local/include/tiledb/context.h
-- Up-to-date: /usr/local/include/tiledb/core_interface.h
-- Up-to-date: /usr/local/include/tiledb/deleter.h
-- Up-to-date: /usr/local/include/tiledb/dimension.h
-- Up-to-date: /usr/local/include/tiledb/domain.h
-- Up-to-date: /usr/local/include/tiledb/exception.h
-- Up-to-date: /usr/local/include/tiledb/filter.h
-- Up-to-date: /usr/local/include/tiledb/filter_list.h
-- Up-to-date: /usr/local/include/tiledb/fragment_info.h
-- Up-to-date: /usr/local/include/tiledb/group.h
-- Up-to-date: /usr/local/include/tiledb/object.h
-- Up-to-date: /usr/local/include/tiledb/object_iter.h
-- Up-to-date: /usr/local/include/tiledb/query.h
-- Up-to-date: /usr/local/include/tiledb/query_condition.h
-- Up-to-date: /usr/local/include/tiledb/schema_base.h
-- Up-to-date: /usr/local/include/tiledb/stats.h
-- Up-to-date: /usr/local/include/tiledb/type.h
-- Up-to-date: /usr/local/include/tiledb/utils.h
-- Up-to-date: /usr/local/include/tiledb/version.h
-- Up-to-date: /usr/local/include/tiledb/vfs.h
-- Up-to-date: /usr/local/include/tiledb/arrowio
-- Up-to-date: /usr/local/include/tiledb/arrow_io_impl.h
-- Up-to-date: /usr/local/include/tiledb/tiledb_export.h
-- Up-to-date: /usr/local/lib/cmake/TileDB/TileDBConfig.cmake
-- Up-to-date: /usr/local/lib/cmake/TileDB/TileDBTargets.cmake
-- Installing: /usr/local/lib/cmake/TileDB/TileDBTargets-release.cmake
-- Up-to-date: /usr/local/lib/pkgconfig/tiledb.pc
make: Leaving directory '/home/ubuntu/git/tiledb/build/tiledb'
ubuntu@aws:~/git/tiledb/build(dev)$ 

The following are the c headers:

 /usr/local/include/tiledb/tiledb.h
/usr/local/include/tiledb/tiledb_enum.h
/usr/local/include/tiledb/tiledb_version.h
/usr/local/include/tiledb/tiledb_serialization.h

[eddelbuettel]

Penny very sloooowly dropping here. So post-merge, the experimental headers gets added, standard -I.... switches or default find them and all is well. I was being mislead by the current line in the test file, i.e.

#include "tiledb/sm/c_api/tiledb_experimental.h"

which has the tiledb/sm/c_api part from the source. But I am on the same page now, I think.

[Shelnutt2]

Penny very sloooowly dropping here. So post-merge, the experimental headers gets added, standard -I.... switches or default find them and all is well. I was being mislead by the current line in the test file, i.e.

#include "tiledb/sm/c_api/tiledb_experimental.h"

which has the tiledb/sm/c_api part from the source. But I am on the same page now, I think.

The testing weirdness is an artifact of how the tests are setup to use the c/cpp apis but we don't do a full installation/linking but instead link the tiledb_unit directly with the object files from the build. We are slowly and but surely working on the testing infrastructure (see the interval pr #2435 and #2417 where we add a "proper" unit test). Hopefully we'll eventually switch the testing around to be more standard and any integration/compliance tests will use a test-driven local "install (in build env)" instead of direct object linking.