Convert server plugin API to C

[Spacechild1]

Motivation

Currently, the server plugin API is actually a C++ API. This has the significant downside that you cannot write server plugins in other languages without at least some C++ glue code.

Since most languages can interface with C, a pure C API would make it easier to develop server plugins in other languages (C, Rust, Zig, Fortran, etc.)

There are two distinct, but related aspects:

1. languages that understand C headers (C, C++, Obj-C, D, Zig, etc.) can work directly with our header files

2. all other languages (e.g. Rust) can reimplement our plugin API by following the C ABI

Plan for Implementation

The required changes necessarily break the ABI, so we need to bump the plugin API version and recompile existing plugins. However, we should try to maintain source compatibility as much as possible, i.e. existing C++ plugins should still compile.

In general, these are the required changes:

1. replace bool data members and function arguments with int
   Only requires recompilation.

2. replace reference functions parameters with pointers, e.g. struct FifoMsg& --> struct FifoMsg *
   This would obviously break C++ source compatibility.

   If we can guarantee that reference and pointer arguments are interchangable on the ABI level, we might do this conditionally for non-C++ code.

   Otherwise we would have to break source compatibility for all API functions with reference parameters.
   AFAICT, only the following functions would be affected:

   bool (*fSendMsgFromRT)(World* inWorld, struct FifoMsg& inMsg);
   bool (*fSendMsgToRT)(World* inWorld, struct FifoMsg& inMsg);
   struct scfft* (*fSCfftCreate)(size_t fullsize, size_t winsize, SCFFT_WindowFunction wintype, float* indata,
                                 float* outdata, SCFFT_Direction forward, SCFFT_Allocator& alloc);
   void (*fSCfftDestroy)(scfft* f, SCFFT_Allocator& alloc);
   bool (*fGetScopeBuffer)(World* inWorld, int index, int channels, int maxFrames, ScopeBufferHnd&);
   void (*fPushScopeBuffer)(World* inWorld, ScopeBufferHnd&, int frames);
   void (*fReleaseScopeBuffer)(World* inWorld, ScopeBufferHnd&);
   

   Assuming that most people actually use the corresponding macros, we may get away with changing their definition.
   For example:
   #define SendMsgFromRT (*ft->fSendMsgFromRT)
   would become
   #define SendMsgFromRT(inWorld, inMsg) (*ft->fSendMsgFromRT(inWorld, &(inMsg))
   The scope buffer functions don't have corresponding macros, but I guess they are also very rarely used.

3. If not compiled as C++, remove inline member functions from structs (sc_msg_iter, Complex, FifoMsg, RGen, etc.).
   Each language wrapper can then provide their own helper functions for dealing with these structs.
   This does not affect C++ plugins.

4. replace "real" C++ classes with C equivalent.
   AFAICT, this only concerns SCFFT_Allocator (see SCFFT_Allocator not ABI compatible #4438) and BelaScope (see Bela: convert BelaScope to a C API #5411)

5. replace nova spinlocks in World and SndBuf for Supernova plugins
   Since the spinlocks come from an external library, we cannot just remove the C++ parts (as in 3.)
   For non-C++ plugins, we could replace the C++ classes with C structs of the same data layout, so other languages would just need to add appropriate helper functions.
   However, these helper functions would have to match the implementation of the corresponding C++ member functions in the nova library! This is particularly important for the reader-writer spinlocks!

[telephon]

In Rethinking the Computer Music Language: SuperCollider (2002) James McCartney defined it this way: "The unit generator applications programming interface (API) is a simple C interface." (p. 64)

… and IIRC there was no discussion about whether to change it. It probably just happened?

[Spacechild1]

The unit generator applications programming interface (API) is a simple C interface."

I doubt that this was ever really true... But people definitely added more and more C++ bits over time.

[shimpe]

Just a wild idea (not really thought out): would it be possible to implement a new API as pure C, then provide the existing C++ API in terms of the new C API (a kind of wrapper) so ultimately no-one is affected?

[Spacechild1]

would it be possible to implement a new API as pure C, then provide the existing C++ API in terms of the new C API

The Server exposes the interface table and various structs. These need to be the same for all languages. A C++ wrapper cannot magically change the data layout or function parameters.

Note that the changes outlined above mostly affect the ABI and only require compilation. There are just a few cases in 2. and 4. where maintaining source compatibility is a bit more tricky.

As a side note: we already have a C++ wrapper over the (pseudo-)C API: SC_Plugin.hpp. However, this is just inline C++ code on the plugin side.

[scztt]

Hmm - could we avoid breaking compatibility with existing plugins by:

1. Keeping the existing C++ API
2. Defining a C API that wraps the C++ API (mostly this wrapper would be trivial - and where it's not trivial, e.g. SCFFT_Allocator new code would only affect C plugins, and in ways that would be unavoidable regardless how our C API is created)
3. Building and exporting BOTH from scsynth.

There should be no danger of collisions between the extern "C" symbols and the current C++ symbols. I would consider this approach less than ideal in other cases, but our plugin API changes almost never, so the maintenance burden here is extremely low once it's implemented. It would save headaches for anyone using unmaintained plugins who aren't set up to recompile themselves (or plugins where recompiling is non-trivial), and it would make it possible to continue to use one set of plugin binaries on your system across multiple scsynth versions (otherwise, we probably need to figure out some kind of workflow for a versioned Extensions folder to keep C-ABI and C++-ABI copies of plugins separate).

[scztt]

There may be cases where struct layout differs between C and C++, but considering that this was at SOME point nominally a C ABI - it seems likely that everything would be equivalent? But I haven't thought the through the implications of this, maybe there are some other lurking problems...

[Spacechild1]

There should be no danger of collisions between the extern "C" symbols and the current C++ symbols.

scsynth does not export any symbols, it passes the interface table to the plugin entry point function. The whole API is defined by structs containing data and function pointers. My point is that these structs - and all referenced functions - should strictly follow the C ABI. I don't see how a "C++ wrapper" could work in this case.

It would save headaches for anyone using unmaintained plugins who aren't set up to recompile themselves

We need to bump the plugin API anyway in the near future, particulary if we want to implement #5347. I would just try to avoid bumping it several times in a row. For example, we could have a dedicated branch as a PR target for all plugin API changes and only merge it into main once all necessary PRs are done.

There may be cases where struct layout differs between C and C++

One problem are bool members, as they have no well-defined size. They typically occupy 1 byte, but there is no guarantee. So I would rather replace all bool members with char or int32_t. Again, we need to bump the plugin API anyway, so we don't have to worry about ABI breakage.

The general idea is that anyone can look at our structs and correctly deduce the data layout, so people can reimplement our plugin API in other languages (that can interop with C).