[WIP] first preview of shadow.remote

UI needs a lot of work but the basic stuff is working

doc/remote.md

@@ -0,0 +1,116 @@
+# shadow.remote
+
+New architecture intended for internal uses for now but designed to be accessible by all eventually. Maybe as an alternative to nREPL addressing the CLJS related shortcomings. It is also meant to "solve" some other REPL related issues (eg. the print problem).
+
+Good summary of what this is about is the REBL talk given by Stuard Halloway.
+
+- https://www.youtube.com/watch?v=c52QhiXsmyI
+
+## Print Problem
+
+The P in REPL is a problem if E returned a very large value or something that just isn't printable. Yet most REPL tools presume that they are going to be able to get a good enough representation. I frequently blow up my CLJ REPL in Cursive when I accidentally print the shadow-cljs build-state, which is easily several hundred MB when printed.
+
+## Why not REBL?
+
+REBL solves this neatly **BUT** as far as I can tell can only run inside the Clojure VM itself. It is not designed to be used remotely. Maybe that is still coming in the future but given that it is closed source we can't really tell anything about the internal architecture. The lack of CLJS supports makes REBL unusable for the use-cases I'm building all this for.
+
+It isn't practical to assume that something like the REBL UI is going to be able to run in all possible runtimes (eg. react-native, browser, node-script, something cloud-hosted, etc).
+
+Tooling in Browser-targeted builds is also problematic since they can add really huge amounts of code (eg. re-frame-10x) and the UI might not actually "fit" when using responsive layouts. The tools also don't work then using `react-native`.
+
+The goal is to build something generic that can be used in all possible runtimes. The tools should be providing the UI and run separately or embedded like REBL if wanted. They can talk to the relay remotely or in process.
+
+# Architecture
+
+## Relay
+
+A relay just handles receiving messages from and to endpoints based on "routing" numbers. A relay is required since not all runtimes can allow direct connections. It is not possible to connect to a Browser remotely, it must first connect to something itself. Therefore by design all "runtimes" and "tools" have to connect to one "relay" and it will forward messages and notfiy of lifecycle events (eg. runtime or tool disconnecting).
+
+## Runtime
+
+Anything that is willing and capable to execute commands from "tools". This can be a generic CLJ or CLJS runtime but can also be something way more specific. Each runtime has different capabilities and the protocol should allow negotiating what these are.
+
+## Tool
+
+Anything that wants to talk to "runtimes". Most commonly this will be editors or some kind of other tool UI. Could be command line tools. For the most part it is assumed that these tools will connect remotely to a given Relay.
+
+# The Protocol
+
+The protocol exchanges messages which are simple EDN maps.
+
+The relay itself is implemented in CLJ. The only network protocol currently used is using websockets with transit encoding. Others could be added though. A tool may use EDN over TCP to talk to the relay which the runtimes still uses websockets/transit to commicate with the relay.
+
+Each message MUST contain an `:op` keyword describing it's purpose. Some reserved keywords are used for protocol purposes but each `:op` is free to define any additional keywords.
+
+Reserved keywords include:
+
+- optional `:msg-id` unique identifier for each message sent. Each party is supposed to send this as part of all responses a given msg may trigger.
+- `:runtime-id id` set by a tool will tell the relay which runtime to forward the message to
+- `:runtime-broadcast true|false` set by a tool to send message to all connected runtimes
+- `:tool-id id` set by a runtime to send a message to a specific tool
+- `:tool-broadcast true|false` set by a runtime to send message to all connected tools
+- if none of these are set the message will be handled by the relay itself
+
+## Relay Message Flow
+
+On Connect the relay will send `{:op :welcome :tool-id id}` or `{:op :welcome :runtime-id id}`. The client may store its assigned id for later use but does not need to send it since the relay will automatically add the `:tool-id` or `:runtime-id` to all received messages before forwarding.
+
+After that the client may start sending messages and will start receiving other messages. This is not RPC, messages may arrive at any time in any order.
+
+These are triggered whenever a runtime connects or leaves and sent to all connected tools. Might make this optional later so that tools have to subscribe to this info first.
+- `{:op :runtime-connect :runtime-id 123 :runtime-info {...}}`
+- `{:op :runtime-disconnect :runtime-id 123}`
+
+These are triggered when a tool connects or leaves
+- `{:op :tool-connect :tool-id 123}`
+- `{:op :tool-disconnect :tool-id 123}`
+
+## Tap Message Flow
+
+`tap>` support is first since it is much simpler than a REPL but still makes used of the "P" related features.
+
+A tool can subscribe to a give runtime if it has `tap>` support.
+
+- `{:op :tap-subscribe :runtime-id id}`
+- `{:op :tap-unsubscribe :runtime-id id}`
+
+Once subscribed a given runtime may send `:tap` notifications to the subscribed tools via the relay.
+
+- `{:op :tap :obj-id id}`
+
+`id` by default is a random UUID. It must be unique and should be globally unique since a given tool may be talking to different runtimes and having overlapping `:obj-id` may complicate things. The actual tap value is not included in any way.
+
+## Object Flow
+
+A tool may use any `:obj-id` it received to query the runtime for additional info or "views" of that object. The intent is to allow the tool to incrementally "query" the object and maybe "navigate" from it.
+
+- `{:op :obj-request-view :obj-id id :view-type view-type}`
+- `{:op :obj-view :obj-id id :view ...}`
+
+`:view-type` should be a keyword, with additional entries in the message for configure that view type if needed.
+
+The defaults should include
+
+- `:edn` resulting in `{:obj :obj-view :view "string repr of the given object"}`
+- `:edn-limit` `{:op :request-view :obj-id id :view-type :edn-limit :limit 20}` returning `{... :view [true|false "string limited at :limit chars"]}`. The first boolean inditicates whether a limit was reached or not.
+
+These are still a WIP. Mostly structured this way for UI purposes currently.
+- `:summary` `{:data-type :map|:set|:vec|... :obj-type "cljs.core/PersistentArrayMap" :count num}`
+
+Maps are sorted by key and idx refers to their index in the sorted result. Sorting may fail so the summary will include `:sorted true|false`. The tool may request fragments via `:start num` and `:num num`. It should not exceed the previous `:count`. Additionally a `:key-limit num` and `:val-limit num` can be configured to control how much of each value should be attempted to be printed.
+
+- `:fragment` `{idx {:key edn-limit :val edn-limit} ...}`
+
+"nav" can only be done by index, typically received in a `:fragment` previously. It is structured this way to avoid actually having to be able to serialize the key. Most of the time that would be fine but sometimes it won't.
+
+- `{:op :obj-nav :obj-id id :idx num}` might at support for actual `:key` at some point
+- `{:op :obj-nav-success :obj-id id :nav-obj-id id}` the resulting `:nav-obj-id` can then be queried again like everything else.
+
+The tool can decide whether it wants the "complicated" `:summary|:fragment` logic. It could send a `:edn-limit` request with `1000` or so default and only use the more complicated logic for larger values.
+
+Tools may just request the full `:edn` at any time in which case this would match what regular REPLs do. Other `:view` formats can be added easily (by extending the multi-method in the runtime).
+
+
+## REPL
+
+TBD, P will just send out an `:obj-id` to be queried as above.

project.clj

@@ -129,11 +129,12 @@
    :cljs
    {:java-opts ^:replace ["-XX:-OmitStackTraceInFastThrow"]
     :dependencies
-    [[fulcrologic/fulcro "2.8.3"
+    [[com.fulcrologic/fulcro "3.0.5"
       :exclusions
       [clojure-future-spec
        com.stuartsierra/component
        garden]]
+
      [fulcrologic/fulcro-inspect "2.2.5"]
      [funcool/bide "1.6.0"]
      [com.andrewmcveigh/cljs-time "0.5.2"]

shadow-cljs.edn

@@ -65,7 +65,7 @@
 
    :compiler-options
    {:infer-externs :auto
-    :warnings {:invalid-arithmetic true}}
+    :warnings {:invalid-arithmetic false}}
 
    :js-options
    {:js-package-dirs ["packages/ui/node_modules"]}
@@ -79,10 +79,14 @@
      :depends-on #{:app}}
     :build
     {:init-fn shadow.cljs.ui.pages.build/init
+     :depends-on #{:app}}
+    :inspect
+    {:init-fn shadow.cljs.ui.pages.inspect/init
      :depends-on #{:app}}}
 
    :devtools
-   {:preloads [fulcro.inspect.preload]
+   {:preloads [com.fulcrologic.fulcro.inspect.preload
+               shadow.remote.runtime.cljs]
     ;; :loader-mode :eval
     }}
 
@@ -168,6 +172,7 @@
    :devtools
    {;; :loader-mode :script
     :repl-init-ns demo.browser
+    :preloads [shadow.remote.runtime.cljs]
     :before-load-async demo.browser/stop-from-config
     :after-load demo.browser/start-from-config}}
 

src/main/shadow/cljs/devtools/server.clj

@@ -28,6 +28,8 @@
     [shadow.cljs.devtools.server.reload-npm :as reload-npm]
     [shadow.cljs.devtools.server.reload-macros :as reload-macros]
     [shadow.cljs.devtools.server.build-history :as build-history]
+    [shadow.remote.relay :as relay]
+    [shadow.remote.runtime.clojure :as clj-runtime]
     [shadow.cljs.devtools.server.system-bus :as system-bus]
     [shadow.cljs.devtools.server.system-bus :as sys-bus])
   (:import (java.net BindException Socket SocketException InetSocketAddress)
@@ -477,6 +479,16 @@
                    :start repl-system/start
                    :stop repl-system/stop}
 
+                  :relay
+                  {:depends-on []
+                   :start relay/start
+                   :stop relay/stop}
+
+                  :clj-runtime
+                  {:depends-on [:relay]
+                   :start clj-runtime/start
+                   :stop clj-runtime/stop}
+
                   :out
                   {:depends-on [:config]
                    :start (fn [config]

src/main/shadow/cljs/devtools/server/web.clj

@@ -23,6 +23,11 @@
     [clojure.data.json :as json]
     [shadow.cljs.devtools.server.dev-http :as dev-http]))
 
+(defn create-index-handler [{:keys [db] :as env}]
+  (fn index-handler [request]
+    {:status 200
+     :body "hello world"}))
+
 (defn index-page [{:keys [dev-http] :as req}]
   (common/page-boilerplate req
     {:modules [:app]
@@ -253,6 +258,8 @@
       (update :ring-request ring-params/params-request {})
       (http/route
         ;; temp fix for middleware problem
+        (:ANY "/api/runtime" web-api/api-runtime)
+        (:ANY "/api/tool" web-api/api-tool)
         (:ANY "/api/ws" web-api/api-ws)
         (:ANY "^/api" web-api/root)
         (:ANY "^/ws" ws/process-ws)

src/main/shadow/cljs/devtools/server/web/api.clj

@@ -16,7 +16,8 @@
     [clojure.core.async :as async :refer (go >! <! alt!! >!! <!!)]
     [shadow.core-ext :as core-ext]
     [shadow.cljs.devtools.server.system-bus :as sys-bus]
-    [shadow.cljs.devtools.server.repl-system :as repl-system])
+    [shadow.cljs.devtools.server.repl-system :as repl-system]
+    [shadow.remote.relay :as relay])
   (:import [java.util UUID]))
 
 (defn index-page [req]
@@ -116,20 +117,20 @@
           (alt!!
             ws-in
             ([msg]
-              (if-not (some? msg)
-                ws-state
-                (-> ws-state
-                    (process-api-msg msg)
-                    (recur))))
+             (if-not (some? msg)
+               ws-state
+               (-> ws-state
+                   (process-api-msg msg)
+                   (recur))))
 
             tool-out
             ([msg]
-              (if-not (some? msg)
-                ws-state
-                (do (>!! ws-out {::m/op ::m/tool-msg
-                                 ::m/tool-msg msg})
+             (if-not (some? msg)
+               ws-state
+               (do (>!! ws-out {::m/op ::m/tool-msg
+                                ::m/tool-msg msg})
 
-                    (recur ws-state))))))]
+                   (recur ws-state))))))]
 
     (async/close! tool-in)
 
@@ -194,3 +195,49 @@
        :body ""}
       )))
 
+(defn api-runtime-loop! [{:keys [relay ws-out ws-in runtime-info] :as ws-state}]
+  (let [from-relay (relay/runtime-connect relay ws-in runtime-info)]
+    (loop []
+      (when-some [msg (<!! from-relay)]
+        (>!! ws-out msg)
+        (recur))))
+  ::done)
+
+(defn api-runtime [{:keys [relay transit-read transit-str] :as req}]
+  ;; FIXME: negotiate encoding somehow? could just as well use edn
+  (let [ws-in (async/chan 10 (map transit-read))
+        ws-out (async/chan 10 (map transit-str))]
+    {:ws-in ws-in
+     :ws-out ws-out
+     :ws-loop
+     (async/thread
+       (api-runtime-loop!
+         {:relay relay
+          :runtime-info
+          {:lang :cljs
+           :remote-addr (get-in req [:ring-request :remote-addr])
+           :user-agent (get-in req [:ring-request :headers "user-agent"])}
+          :ws-in ws-in
+          :ws-out ws-out}))}))
+
+(defn api-tool-loop! [{:keys [relay ws-out ws-in] :as ws-state}]
+  ;; FIXME: should take tool-info, just like runtime-connect and runtime-info
+  (let [from-relay (relay/tool-connect relay ws-in)]
+    (loop []
+      (when-some [msg (<!! from-relay)]
+        (>!! ws-out msg)
+        (recur))))
+  ::done)
+
+(defn api-tool [{:keys [relay transit-read transit-str] :as req}]
+  ;; FIXME: negotiate encoding somehow? could just as well use edn
+  (let [ws-in (async/chan 10 (map transit-read))
+        ws-out (async/chan 10 (map transit-str))]
+    {:ws-in ws-in
+     :ws-out ws-out
+     :ws-loop
+     (async/thread
+       (api-tool-loop!
+         {:relay relay
+          :ws-in ws-in
+          :ws-out ws-out}))}))

src/main/shadow/cljs/devtools/server/web/common.clj

@@ -40,6 +40,7 @@
                   (.getCanonicalFile)
                   (.getName))]
       [:link {:rel "stylesheet" :href "/css/main.css"}]
+      [:link {:rel "stylesheet" :href "/css/tailwind.min.css"}]
       [:meta {:name "viewport" :content "width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"}]]
      [:body {:class body-class}
       content