Saving core images with variables bound to the result of require-namespace does not work.

[kat-co]

I was wondering if anyone could shed some light on why this is occurring. I am using SBCL v2.0.0.

If I load a REPL, and do this (defparameter *ns* (gir:require-namespace "Arrow")), everything works as intended.

If I do (asdf:make :my-system) (i.e. save-lis-and-die) on a system which utilizes cl-apache-arrow, all operations involving that *ns* parameter fail with something like:

** (process:1840): CRITICAL **: 02:02:15.405: g_irepository_find_by_name: assertion 'typelib != NULL' failed
WARNING: No such FFI name ArrowFileWriter

If I replace all instances of *ns* with (gir:require-namespace "Arrow"), everything works as expected again. Something is not surviving the persistence of the core, and I was hoping someone more familiar with the internals of this library would know what.

cc kat-co/cl-apache-arrow#5

[vindarel]

I'll link to the related, great project of kat-co, used for cl-apache-arrow: https://github.com/kat-co/gir2cl

This is a library for generating a Common-Lisp foreign function interface (FFI) from GObject Introspection XML files (GIR files).

[andy128k]

gir:require-namespace implicitly loads dynamic library (.so, .dll, .dylib). Loaded image has memory dump only. All dynamic libraries need to be reloaded again.

In case of SBCL a hook sb-ext:*init-hooks* can be used for re-loading needed libraries.

[kat-co]

This makes a lot of sense, and I was hoping it would solve my issue, but unfortunately despite some tinkering, I can't seem to get it to work. Here's a small reproducible test-case:

/tmp/experiment.lisp:

(asdf:load-system :cffi)
(asdf:load-system :cl-gobject-introspection)

(defparameter *ns* (gir:require-namespace "Arrow"))

(defun bug-p ()
  (format t "Foreign libraries loaded before call:~%~a~%" (cffi:list-foreign-libraries))
  (gir:invoke (*ns* "BooleanDataType" "new"))
  (format t "Foreign libraries loaded after call:~%~a~%" (cffi:list-foreign-libraries)))

(pushnew
 (lambda ()
   (format t "Loading libraries...~%")
   (mapcar #'cffi:load-foreign-library
           (list "libgirepository-1.0.so"
                 "libgobject-2.0.so")))
 sb-ext:*init-hooks*)

(pushnew
 (lambda ()
   (let ((foreign-libs (cffi:list-foreign-libraries)))
     (format t "Libraries loaded when dumped:~%~a~%" foreign-libs)
     (loop for lib in foreign-libs
           do (cffi:close-foreign-library lib))))
 sb-ext:*save-hooks*)

(sb-ext:save-lisp-and-die "bug-p" :toplevel #'bug-p :executable t)

/tmp $ sbcl --load experiment.lisp
This is SBCL 2.0.0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
Libraries loaded when dumped:
(#<FOREIGN-LIBRARY GIREPOSITORY "libgirepository-1.0.so.1.0.0">
 #<FOREIGN-LIBRARY GOBJECT "libgobject-2.0.so.0.6000.6">)
[undoing binding stack and other enclosing state... done]
[performing final GC... done]
[defragmenting immobile space... (fin,inst,fdefn,code,sym)=1269+1054+20487+20900+26486... done]
[saving current Lisp image into bug-p:
writing 0 bytes from the read-only space at 0x50000000
writing 544 bytes from the static space at 0x50100000
writing 36470784 bytes from the dynamic space at 0x1000000000
writing 2150400 bytes from the immobile space at 0x50300000
writing 13746176 bytes from the immobile space at 0x52100000
done]

/tmp $ ./bug-p 
Loading libraries...
Foreign libraries loaded before call:
(#<FOREIGN-LIBRARY LIBGOBJECT-2.0.SO-439 "libgobject-2.0.so.0.6000.6">
 #<FOREIGN-LIBRARY LIBGIREPOSITORY-1.0.SO-438 "libgirepository-1.0.so.1.0.0">)

** (process:13175): CRITICAL **: 13:03:16.171: g_irepository_find_by_name: assertion 'typelib != NULL' failed
WARNING: No such FFI name BooleanDataType

debugger invoked on a SB-PCL::NO-APPLICABLE-METHOD-ERROR in thread
#<THREAD "main thread" RUNNING {10005884C3}>:
  There is no applicable method for the generic function
    #<STANDARD-GENERIC-FUNCTION GIR::NSGET (6)>
  when called with arguments
    (NIL "new").
See also:
  The ANSI Standard, Section 7.6.6

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY] Retry calling the generic function.
  1: [ABORT] Exit from the current thread.

((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION GIR::NSGET (6)> NIL "new") [fast-method]
0]

In my production code, I'm using uiop:register-image-restore-hook and uiop:register-image-dump-hook.

I'm still not sure if this is an issue with cl-gobject-introspection or not, so thanks for any feedback if it ends up not being your issue.

[kat-co]

If I place this within a pre-image-dump hook:

(setf gir::*namespace-cache* (make-hash-table :test 'equal))

and then this in an image restore hook:

(setf *ns* (gir:require-namespace "Arrow"))

things work as expected. So I think it has something to do with the namespace cache not working across image persists?