GH-37537: [Integration][C++] Add C Data Interface integration testing

[pitrou]

Rationale for this change

Currently there are no systematic integration tests between implementations of the C Data Interface, only a couple ad-hoc tests.

What changes are included in this PR?

1. Add Archery infrastructure for integration testing of the C Data Interface
2. Add implementation of this interface for Arrow C++

Are these changes tested?

Yes, by construction.

Are there any user-facing changes?

No.

• Closes: [Integration] Add C Data Interface integration testing #37537

[pitrou]

@wjones127 @tustvold, FYI, following this PR, you'll need to add the --run-ipc option in these two places:

• https://github.com/apache/arrow-rs/blob/master/arrow-integration-testing/README.md
• https://github.com/apache/arrow-rs/blob/master/.github/workflows/integration.yml#L107

[tustvold]

Thank you for the heads up - created a tracking issue so this doesn't get missed - apache/arrow-rs#4828

[bkietz]

Looks solid

[bkietz]

I find this parameter's type quite confusing. IIUC it's a cancel token to cut off long-running gc? I'm not sure why this control is given to an exporter or importer. Instead, I would think it would make more sense for the runner check_memory_released to construct this token. It can be Callable[[], bool] and returns true if GC is taking too long and the runner is aborting the current case

[pitrou]

No, really, it's as the doc states. Some runtimes may need several GC calls to properly release memory, hence the name.

[bkietz]

I'm probably missing something, but it seems like this could be done more simply with something like:

@contextmanager
def check_memory_released(exporter: CDataExporter, importer: CDataImporter,
                          gc_timeout: Callable[[], bool] = _default_timeout):
    do_check = (exporter.supports_releasing_memory and
                importer.supports_releasing_memory)
    if not do_check:
        yield; return
    before = exporter.record_allocation_state()
    yield
    while exporter.record_allocation_state() != before:
        if gc_timeout():
            raise ...
        importer.gc_once()

[pitrou]

It probably could, though I'm not sure it's simpler :-). Both solutions (yours and mine) are IMHO not very elegant, and we may have to revisit if making two GCs coexist ends up more complicated...

[bkietz]

I prefer mine for less inversion of control since the exporter doesn't need to manage the importer's garbage collection.

[pitrou]

I'll go with the current version. This can evolve quite easily as this is internal tooling, so no API guarantees.

[wjones127]

Overall this looks good. Have a few minor comments.

[pitrou]

I'll merge this PR now, thanks for the reviews!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 3b646ad.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.