Changes for connections from the web browser

[dkochmanski]

This set of changes introduces a new connection type :ASYNC that relies on Gray Stream interfaces. Moreover SLIME-NET-SEND anticipates that the connection may be associated with a websocket, and if that's the case we use a specialized function to send data.

With these changes I've managed to successfully connect from an image running inside the web browser to a shim server bridging SLIME and websockets.

[stassats]

And for some reason the CI breaks on CCL.

[dkochmanski]

I'll look into CCL tests not passing tomorrow. Thanks for reviewing these changes.

EDIT: CCL failure was caused by not having all gray stream protocol operators implemented and exported. I've withdrawn the part that pulls symbols to swank's gray package because of that and left only essential parts.

[dkochmanski]

I've submitted a minimal set of changes to slime.el and for ecl backend (as requested) and they were merged some time ago. The only remaining change is to introduce :async communication style.

This style is meant for swank communicating through a stream, where the user implements the gray protocol that handles the communication without assumptions that there is an underlying file handler (many backends make that assumption).

It is called :async, because the control loop that is usually started by server-requests is not invoked from swank, but rather the client serves requests at its own discretion (most notably, from a callback that may signal that there is a new message sent from slime).

Why singlethreaded-connection or multithreaded-connection are not enough:

• multithreaded-connection assumes threads, while in case of :async, it is at the implementor discretion whether the connection runs in a single or in multiple threads

• singlethreaded-connection style implies slime-use-sigint-for-interrupt, but most notably wait-for-event/event-loop is structured in a way, that it never returns when in the debugger context (because wait-for-input is called even when timeout is set to nil)

(defun wait-for-event/event-loop (connection pattern timeout)
  (assert (or (not timeout) (eq timeout t)))
  (loop 
   (check-slime-interrupts)
   (let ((event (poll-for-event connection pattern)))
     (when event (return (car event))))
   (let ((events-enqueued (sconn.events-enqueued connection))
         (ready (wait-for-input (list (current-socket-io)) timeout)))
     (cond ((and timeout (not ready))
            (return (values nil t)))
           ((or (/= events-enqueued (sconn.events-enqueued connection))
                (eq ready :interrupt))
            ;; rescan event queue, interrupts may enqueue new events 
            )
           (t
            (assert (equal ready (list (current-socket-io))))
            (dispatch-event connection
                            (decode-message (current-socket-io))))))))

Contrast this with the variant for async connection, where listen is called only when timeout is true (after checking whether there are already available events):

(defun wait-for-event/async-read (connection pattern timeout)
  (assert (or (not timeout) (eq timeout t)))
  (loop
    (check-slime-interrupts)
    (let ((event (poll-for-event connection pattern)))
      (when event (return (car event))))
    (let ((events-enqueued (aconn.events-enqueued connection)))
      (cond
        ;; New events pushed or executed an interrupt. Loop over.
        ((or (/= events-enqueued (aconn.events-enqueued connection))
             (check-slime-interrupts))
         nil)
        ;; No input available and we are not about to be waiting.
        ((and timeout (not (listen (current-socket-io))))
         (return (values nil t)))
        ;; Process the next message as soon as it is available.
        ((dispatch-event connection
                         (decode-message (current-socket-io))))))))

This second version ensures, that when there is no timeout, we will proceed straight to decode-message that will block until there is new input. Why it matters in WASM port:

When a condition is signaled, we enter sldb-loop to allow debugging the condition from emacs. Understandably, to preserve context, it doesn't return until we are done with the debugger. So wait-for-event is called with timeout set to nil.

In my port, stream-listen is implemented by checking whether there is data in the buffer, while reading from the stream (a blocking operation), will yield control to javascript, that will invoke the callback when event occurs, effectively populating the buffer, and returning control to stream-read (called by decode-message).

So if wait-for-input is called when timeout is nil, it will wait forever checking whether the stream has data (and it won't, because we didn't return control to the main process by yielding).

This set of changes does not impact existing code paths, so it should be safe to merge. I also think that it is a useful general-purpose solution for communication that levereges gray stream protocol in a backend-independent way.

[stassats]

Is there code anywhere that uses this new functionality that I can look at?

[dkochmanski]

Yes; there are even two clients, so you can try it from SBCL without building WECL in the browser.

• slime adapter to websockets: https://fossil.turtleware.eu/wecl/file?name=App/lime.el

load this file in emacs and evaluate the form (lime-net-listen "localhost" 8889) to start listening for connections

• swank adapter to websockets (portable):
  https://fossil.turtleware.eu/wecl/file?name=App/slug.lisp
  https://fossil.turtleware.eu/wecl/file?name=App/frig.lisp

sbcl --load slug.lisp --load frig.lisp
CL-USER> (EU.TURTLEWARE.WECL/FRIG::test-frig)

This should open a new repl in the emacs session. Note that frig.lisp assumes that quicklisp is present and loads "websocket-driver-client" and "bordeaux-threads" system

• swank adapter to websockets (browser):
  https://fossil.turtleware.eu/wecl/file?name=App/slug.lisp
  https://fossil.turtleware.eu/wecl/file?name=App/wank.lisp

This client runs from the browser, i.e using a page like this:

<!doctype html>
<html>
  <head>
    <title>Web Embeddable Common Lisp</title>
    <link rel="stylesheet" href="easy.css" />
    <script type="text/javascript" src="boot.js"></script>
    <script type="text/javascript" src="wecl.js"></script>
    <script type="text/common-lisp" src="slug.lisp"></script>
    <script type="text/common-lisp" src="wank.lisp"></script>
    <script type="text/common-lisp" src="easy.lisp">
      (defvar *connect-button* (make-element "button" :text "Connect"))
      (define-js-callback (connect-to-slug :type :null) ((event :js-ref))
       (wank-connect "localhost" 8889)
       (setf (inner-text *connect-button*) "Crash!"))
      (add-event-listener *connect-button* "click" (js-callback connect-to-slug))
      (append-child [body] *connect-button*)
    </script>
  </head>
  <body>
  </body>
</html>

you may try it without cross compiling ecl here: https://turtleware.eu/static/misc/wecl-20250821/slug.html, when you click "Connect", it will try to connect to the server created by lime.el at port 8889 at localhost.

Note that loading the page first time may take some time depending on your internet connection, because wecl.wasm is around 10MB.

[dkochmanski]

rebased against current master.

[stassats]

I've been meaning to review it, but I'm always busy with other stuff, and I don't want to hastily introduce new APIs to be stuck with it forever after.

[dkochmanski]

Any update on this? technically I'm also slime dev, so if anything breaks due to this new API I will address the issue when @mentioned.