CI: Add integration tests to cover `cuda.bindings` ABI stability

[leofang]

We should add a CI job that tests running newer cuda.bindings + projects built against older cuda.bindings (from the same major release) to catch this in the future, either on a per-PR or nightly (#294) basis.

Originally posted by @leofang in #274

[mdboom]

A simpler approach might be to use abigail to detect ABI changes. I looked at this a few weeks ago, and I think I was seeing false positives from it (changes in C ABI that don't matter in practice for Cython). I plan to have another go at it and run over a chain of existing commits to see if it can detect legitimate ABI breakage. If that works, that would be easier than integrating across multiple projects.

What are the intended expectations of ABI stability for CUDA bindings? No breakage across a major version?

[leofang]

changes in C ABI that don't matter in practice for Cython

RIght. This is the unusual part of this task. We're not dealing with C ABIs (for which we could use various tools, including those from the ABI Lab), but Cython ABIs. However it is unclear what it entails for symbols that are cimport'd. It is never properly documented and the upstream does not want to document it either (related: cython/cython#3077, cython/cython#6484).

In the past we've had Cython ABI-breaking changes at least twice, in both 11.x (conda-forge/cuda-python-feedstock#15, @kkraus14 found this one) and 12.x (#274), and every time it happened it caused a lot of churns. We'd really prefer to avoid another break if possible.

No breakage across a major version?

Yes. Projects built against cuda-bindings 12.x (via cimport) should work with all 12.y, for arbitrary x <= y (x > y should also be allowed if no new symbol from 12.y is used).

[mdboom]

I ran an experiment using abigail tools (the same tool CPython uses to check ABI compatibility). It's definitely doable, but I've pretty much convinced myself it wouldn't add much value in our case to be worth it. Cython modules really only have 2 exported symbols at the C ABI level, and they basically never change, and changing that it really under Cython's control anyway, not ours. Anyway here's the experiment for reference.

  <elf-function-symbols>
    <elf-symbol name='PyInit_driver' type='func-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
  </elf-function-symbols>
  <elf-variable-symbols>
    <elf-symbol name='__pyx_module_is_main_cuda__bindings__driver' size='4' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
  </elf-variable-symbols>

IIUC, what matters is ultimately what's in a module's __pyx_capi__ dictionary. We could write a check that compares that against an expected value. Like you say, though, it's sort of a semi-documented thing and there's a lot of unknowns about running a check like that. It still may be worth including it and seeing what happens to learn from it.

Then there is a separate set of issues to detect API changes at the Python level. For that, mypy's stubgen does a decent job of extracting the public API of a Python extension. We could then use its output as a baseline to detect changes.