Type propagation proposal

What

This is a proposal to simplify the CFFI access API and make type declarations optional for most places.

Why

Because my knowledge of macroexpansion and CFFI is not deep enough to get to work on this, so i'm inviting readers to help sketch up an implementation plan here that is feasible and detailed enough to get to work on it.

How

The discussion on the mailing list: http://thread.gmane.org/gmane.lisp.cffi.devel/2882

Features that would be nice to have

• Optional wrapping of foreign SAP's with lisp objects to enable runtime type checks, method dispatch, finalizers
• Type derivation so that the FFI type only needs to be specified at a few places
• Clear semantics of pointer dereference; e.g. I think this should signal an error:

CFFI-TESTS> (defcstruct foo (slot :int))
CFFI-TESTS> (defcfun zork :void (x (:pointer (:struct foo))))
CFFI-TESTS> (with-foreign-object (x '(:struct foo))
              (zork x))

And this, or its equivalent, should work without the mem-ref's:

(with-foreign-object (i :char)
  (setf (mem-ref i :char) (char-code #\a))
  (toupper (cffi:mem-ref i :char)))

• Slime syntax highlight for FFI calls. Through a special syntax (in cl-autowrap c-ref, c-fun, etc)? Or something similar to the coloring of readtime conditions?

• Proper handling of arrays; e.g. consider this:

(defcenum (numeros :int)
  (:one 1)
  :two)

(defcstruct numeros-in-struct
  (numeros-slot numeros :count 2))

What should be the return value of (cffi:foreign-slot-value x '(:struct numeros-in-struct) 'numeros-slot)?

And through what API can the second element be reached?

Idea 1: use macrolet's (and/or symbol-macrolet's)

The general idea is that macros like with-foreign-object, that introduce a new binding and thus require a type to be declared, could expand into code that contains a macrolet along the lines of:

(macrolet
  ((,var ()
      ',type))
    ,@body)

This way whichever other API element needs the type of a variable could use the (var) macro call to get to it.

It's an option to keep two parallel API's, the current one with little magic and thus probably few constraints, and the new one doing magic at macroexpand time and thus with probably more constraints on how it can be used.

Or just one API that accommodates for optionally providing type info by hand where necessary, e.g. as an &optional or &key arg.

Access macros could know about each other and do some lookahead to extract the type (e.g. mem-ref could look at the form in its pointer arg position and recognize a foreign-slot-pointer and deal with it specially to extract the type). A compile-time error could be signaled for failed type derivation.

Idea 2: use some form of code walking

This opens up a big can of worms that may make the whole thing not worthwhile. Or maybe just in a limited form where the toplevel API calls start walking downwards.

Idea 3: use custom declare forms

Use something like (declare (cffi-type var :int)) and macros could access the environment to look up these type annotations.

Idea 4: sb-alien

Port SBCL's sb-alien to use CFFI-SYS?

Rejected ideas

A common 'var-type-of' accessor macrolet

The body of a macrolet sees itself, so there's no way to "(call-next-method)" from a macro.

A symbol-macrolet on e.g. a 'foo-type' kind of generated symbol

This would pollute the package with symbols and i'm not sure it's easy to force macroexpansion of a symbol-macro from the running body of another macro.

Precedents

The c-ref macro in https://github.com/rpav/cl-autowrap/blob/master/plus-c/plus-c.lisp#L54 does something like this, and the cl-autowrap package in general has some of this implemented.