*foreign-string-mappings* and reuse of CFFI string encoders/decoders

[Lovesan]

Is there a reason for *foreign-string-mappings* being private? It would be handy to reuse CFFI instances of babel encoders and decoders to allow, for example, in-place encoding and decoding of lisp strings, and to not look up mappings each time some string operation is performed.

[luismbo]

It'd be nice if the features you mention were supported by the CFFI API without exposing Babel details. The last one might be a bit tricky. Would a compiler macro to get the mapping function at compile time when the encoding is constant be enough for your use case?

[Lovesan]

Compiler macro(+ load-time-value?) would be sufficient for simple cases and would be good to have. But sometimes an encoding is not known beforehand.

Consider an example of a (large) buffered data stream coming from some foreign source like a pipe, a socket, or something like that.

Suppose that we have obtained a pointer to the buffer which is filled by some foreign function.

Now, it would be nice to have some macro (say, named with-foreign-string-decoder) which would allow the following:

(let* ((enc (babel-encodings:get-character-encoding enc-designator))
       (bytes-per-unit (ceiling (babel-encodings:enc-code-unit-size enc) 8))
       (max-bytes-per-char (* bytes-per-unit
                              (babel-encodings:enc-max-units-per-char enc))))
  ;; ...
  (with-foreign-string-decoder (enc decoder-fn char-counter-fn)
    ;; ...
    (loop :for max-char-count =
            (floor
              (max 0 (- buffer-end
                        buffer-offset
                         ;; To avoid an error in code point counter,
                         ;;   in the case of a partial char being read: 
                        (if end-of-data-stream 0 max-bytes-per-char)))
              bytes-per-unit)
          :do (whatever) ;; Read next data portion
                         ;;   and set `end-of-data-stream' var, etc.
              (multiple-value-bind (char-count new-buffer-offset)
                  (funcall char-counter-fn buffer-ptr
                                           buffer-offset
                                           buffer-end
                                           max-char-count)
                (when (plusp char-count)
                 (ensure-string-buffer-size (+ string-buffer-offset char-count))
                 (funcall decoder-fn buffer-ptr
                                     buffer-offset
                                     new-buffer-offset
                                     string-buffer
                                     string-buffer-offset)
                 (do-string-buffer-processing)
                 (incf string-buffer-offset char-count)
                 (setf buffer-offset new-buffer-offset)
                 ;; etc.
                 (do-something-else))))))

A corresponding with-foreign-string-encoder would also be nice to have, of course.