RFC: Alternative server-side rendering (SSR) architecture

[raquo]

This is quite long and windy, perhaps the code example at the end is the best TLDR.

The problem

It would be nice to "use Laminar on the backend", i.e. share Laminar code between frontend and backend. This would let the server return static HTML with all the web page content, instead of returning an empty HTML document that is only populated with the content by the Javascript (Scala.js / Laminar) on the client side.

Inconveniently to this goal, Laminar components are heavily reliant on the single-threaded JS runtime environment and native JS types, and can not run on the JVM. Eliminating these dependencies would require a lot of work and would complicate our design with nontrivial cross-platform abstractions. It would not be a good tradeoff.

Some other libraries like React.js can offer server side rendering because their DOM virtualization layer makes them less tied to real JS types, and because Javascript is dynamically typed, and because server side rendering with React.js is still done in Javascript (on node.js), whereas for Scala.js libraries like Laminar, "server side" means the JVM – a completely different environment that our library is not suited for.

Status quo: Rendering with Javascript on the backend

It is theoretically possible to run Laminar in JS on the backend, e.g. in node.js with jsdom, or in a headless browser (example). Such an approach can provide all Laminar features while requiring few if any code changes to your frontend code.

For example, if you have a Laminar frontend to book hotels, and you want to make the content of hotel pages available to search engines that can't crawl content generated by client-side Javascript, you could pre-render these Laminar pages in a headless browser on the backend, and output the resulting HTML to the users (and search engine bots). Then when the static page loads, you could initialize a live Laminar app which would overwrite the server-returned HTML to enable the interactive web app functionality that the users need, but the search engine bots don't care about.

So far I've been recommending this approach precisely because you can apply it to any Laminar web app, and get (almost) all Laminar features working out of the box. I still think that this could be a viable approach, but for now I personally don't have the time to investigate it further.

See also: #60

Proposal: Rendering on the JVM

So, I can not re-implement Airstream and Laminar to run on the JVM – while that is technically possible, the required complexity is not worth it, and extremely so. Also, the amount of work this would require, both for the initial implementation and subsequent maintenance is not something I can afford to offer to the community. Laminar can be very simple precisely because it only runs in JS, and I'd like to keep it that way.

With that out of the way, here is what I am thinking: we create a new project that lets you build HTML on the backend using the new, generator-based Scala DOM Types (raquo/scala-dom-types#87). So, you would say stuff like div(attr := "key", b("bold text")), get an instance of some TemplateElement class, from which you can later read all the key-value pairs and all the children that you've added to it. You can call an amend method to add more modifiers, or call getHtmlString.

On its own, this would essentially be a templating language with a familiar syntax that looks like Laminar, but without the reactive parts – no onClick, no children, no observables.

So, let's see how we can integrate these new templates with Laminar. First, let's outline some of their properties:

1. Every template describes the a single element subtree, e.g. div(attr := "key", b("bold text"))
2. Such templates are cross platform – their code can be shared between Scala.js and JVM
3. Every template can be encoded as a string for network transport – as pure HTML (or SVG, I guess?)
4. Templates are fully declarative. Airstream observables, onMount* hooks, and custom modifiers are not allowed.
5. Every template can be diffed against another template, or against a real JS DOM element

With these features and constraints, we could easily use the same template both on the server to output its HTML, and on the client to create a real DOM element. We could even achieve efficient hydration, that is, the frontend code could instantiate a template / Laminar element from the HTML returned by the server without re-creating elements from scratch. And of course we would be able to similarly build a new Laminar element from any template on the frontend, even if it wasn't returned by the server as HTML.

This is all well and good, but the question remains – how can we integrate Laminar's reactive functionality – inserting dynamic children, adding event listeners, etc. – into these cross platform templates? Laminar already has good support for integrating with third party DOM libraries (at least in 15.0.0), so this is actually very doable:

1. For adding event listeners, onMount hooks, and other arbitrary binders / modifiers, all we need is a reference to the target element, so we need to somehow "mark" it in the template's HTML code, so that we can find the target element later on the frontend.
2. For inserting children, we need to mark not an element, but a certain location in the DOM produced by the template as the injection point for the children. Laminar already does this using the so called sentinel nodes: whenever you add e.g. children <-- stream to an element, Laminar actually adds an empty comment node in that location, so that it knows where to insert the children once the stream starts emitting.

The common thread here is that we somehow need to mark places in the DOM / HTML of the template, and then some way for Laminar to query the resulting DOM to find those elements. For referencing elements, we could associate the target element with some string identifier via data-* attributes, HTML class names, or id attributes, and similarly for children insertion reference points we could use some variation of sentinel nodes.

This approach feels rather similar to returning static HTML from the server, then initializing islands of interactivity on it with jquery. Conceptually, yes, it's quite similar, but I'm hoping that the details are different enough to make our approach safer and more ergonomic.

Example

// Cross-platform code:
// Note that T.div(...) is not a laminar ReactiveElement, it's a completely separate, cross-platform type as described above.
// That type is merely descriptive / declarative, instantiating it does NOT instantiate a real HTML element.

val template: TemplateElement = T.div(
  T.button(ref("btn")),
  slot("children-x")(startWith = T.i("Loading..."),
  "Hello world"
)
teamplate.htmlString == """
  <div>
    <button id="btn" />
    <span id="children-x" style="display:none">
      <i>Loading...</i>
    </span>
    Hello world
  </div>
"""

// Frontend-only code:

// create a DOM element from the template, and wrap it into a reactive Laminar element type.
val laminarElement: ReactiveElement = createElementFromTemplate(template)

// alternative: if the server provided the HTML of this template, and you know where to find it in the existing DOM:
// val elementInDom: dom.Element = ??? // find by id, by data attr, etc.
// val laminarElement: ReactiveElement = hydrateFromDom(template, elementInDom)

// add event handler to the div itself
laminarElement.amend(onClick --> ...)

// find the button by its ref id, and add event handler to it (not the div)
laminarElement.findChildByRef("btn").amend(onClick --> ...)

// alternatively if the server has already provided the HTML of the template, and you don't otherwise
// need the `template` variable on the frontend, you can just find the button by its ref in the DOM,
// specifically within the `elementInDom` element in the DOM:
val myButton: ReactiveElement = findRef("btn", elementInDom)
// and then you can do Laminar things with it
myButton.amend(onClick --> ...)

// add children to the end of the div (after "hello world")
laminarElement.amend(children <-- ...)

// find the slot by its id, and add children to it, replacing the default "Loading..." element.
laminarElement.findSlot("children-x")(children <-- ...)

Of course, using strings for referencing elements is not very safe – you might make a typo, or forget to add a necessary event listener altogether. I still need to come up with some design patterns for minimizing the unsafety, which could be as simple as shared scoped constants.

There's also the annoyance of needing the identifier strings to be unique, either globally or within some subtree of the DOM (which the frontend then needs to first find the root of). This will be especially painful if you use multiple usages of templates for reusable components like buttons with more than one reference inside (e.g. referencing the button text, and separately the button's icon). Again, need to find design patterns that would reduce the chance of name collisions, and ideally throw errors or warnings if duplicate ref ids are found. Since the DOM is hierarchical, perhaps some kind of namespacing could work.

With all this, you still wouldn't want to write your whole web app UI using these cross platform templates because they require some boilerplate to integrate with Laminar, and do not offer any reactive features on their own. However, you probably don't need to use the new templates for most of your components. If rendering pages statically, you probably only need the server to render the static HTML of the main layout, header/footer/navigation, and the main content – you don't care about any interactive elements, calendar widgets, etc. – you can just initialize those parts on the client side only, without making them cross-platform. So basically the only cross platform templates you'll have will be the ones with a lot of content and minimal interactivity.

Requesting comments

This new approach isn't fully fleshed out yet, but I welcome comments and concerns.

I am very interested to know whether this level of client-server integration / code sharing will meaningfully address the server side rendering demand for your application, or if the other approach of rendering Laminar apps inside a headless browser (or node.js+jsdom) is more appealing to you despite the requirement for more complex backend infrastructure. Note that I am not yet committing to implement either of these approaches, I need your feedback to inform the decision and prioritize the implementation.

[deterdw]

I definitely think this approach has possibilities. We already do something similar (just for static rendering) in our app, where we have templates shared between Laminar on the front end and Scalatags on the backend - a few platform specific import objects on each side mean I can use the same code in both cases. But that does nothing for hydration, etc. which of course is the big problem.

One possibility that might help ease some of the pain for the library user is to have another library/extension on the JVM side that accepts all the Laminar syntax, but then does nothing for stuff like onClick or onMount, and just generates the sentinel nodes/markers needed for other cases. You could even have a fake library that implements the AirStream syntax for the JVM, but basically does nothing, so that the client code compiles without changes. That's quite ugly architecturally, but I think it definitely beats having to manually look up and wire things from the client side.

[raquo]

@deterdw I'm not sure how this would work on a technical level. I assume the goal is to be able to write a Laminar component once, put it into shared, and cross-compile it for both the JVM and JS targets. On the JVM it would be used to render static HTML, and on JS it would be used as usual.

If I understand your idea correctly, you're saying I could define all Laminar types in a parallel JVM library – all with the same package names and class names etc. – but on the JVM side the types would be slightly different, e.g. our ReactiveElement would have no .ref (because it's a JS type), and no type param (again, JS type), so in your shared code you won't be able to use .ref, and you would need to use type aliases like HtmlElement that don't reference JS types explicitly, and on the JVM they would in fact alias to a different, JVM-compatible type, whereas in JS they would stay what they are (ReactiveHtmlElement[dom.html.Element]).

For that, I would need to reimplement the entirety of Laminar and Airstream API in JVM, but the entirety of Airstream would essentially be noop, and all the parts that don't directly affect the produced HTML (such as event handlers) would also be noop.

Is that approximately what you meant?

On its own this would let us generate the HTML code for Laminar components on the backend, but would not immediately provide efficient hydration functionality.

Regarding hydration in this model... well, I guess it would be possible if we can make certain assumptions like "the backend has already created all the exact same static elements that this Laminar component creates, so all we need to do is hang event listeners and insert dynamic children that we know the backend didn't do". It could work with sentinel nodes etc. as you said, but I imagine it would be rather fragile when the backend unexpectedly generates different HTML (e.g. because it passed different input arguments to the component on the backend, compared to the arguments we pass to the same component on the frontend). Well, If we ever get to this point I would need to research other libraries' hydration algorithms.

---

I wonder if any other libraries do this kind of thing, where the user writes code in shared, but that code ends up compiling to use different types and implementations on the frontend and on the backend, with no shared trait common to those types. OTOH I suspect that Scala might be ok with that, but IDEs might throw a fit.

I feel like maybe code-generating macros would be a better technical solution to this. We don't really care that the frontend and backend copies of the component use the same source code, right? We just want them to produce the same HTML to make server side HTML generation and hydration possible without source code redundancies.

So, perhaps we could write our Laminar components as usual, none of that JVM stuff, and have them annotated with some macro that would parse the component source code and create a JVM-compatible version of the component, and perhaps also modify the frontend version of the component to add the necessary hydration helpers like sentinel nodes (for example, if it sees a child <-- foo, it could wrap it into hydrationSlot("1")(...) on the frontend, and also create the same hydrationSlot("1") in the JVM version. Maybe the hydration will be less fragile that way.

Another bonus of this macro approach is that we can start with the simple approach outlined in the ticket, and then add this on top later as time allows. The downside is that I know nothing about Scala macros, so I can't even be sure that this approach is workable, but overall it does look more promising to me than reimplementing Laminar APIs with JVM types. Although it's still a lot of work. I would really need to be on a different level in terms of how much time I can devote to Laminar development if I am ever to attempt this. But yet again on the upside, this does not require any changes to Laminar APIs, so other developers with more time and experience with macros can try to take a shot at this as well.

So anyway, thanks for your comment, even if I misunderstood it, it gave me more promising ideas.

[deterdw]

Thanks for the friendly reply. I think you got the gist of the idea. I agree it's a lot of work in any way that one approaches it.

The proposal at the top remains the necessary first step. Just generate a text-outputting library with the same structural API as Laminar has. Then one can trivially output the static HTML, which serves some important use cases. As I outlined, this can already be done today with a Scalatags-based hack. Based on that hack, I can confirm that IDEs really don't like this approach, but Scala is OK with it :-)

And one can follow a pattern of defining the component statically in the shared sources and then using .amend on the front end to add the dynamic functionality.

Regarding hydration, it's not necessary to implement all of Airstream. You can have a minimal set of types, e.g. Source[A] and Sink[A] and stipulate that shared code is limited to <-- Source and --> Sink or something like that. The downside is that one has to do some of the wiring independently of the rendering, but that's not worse than React, is it?

The point is that there is no need to support all of Laminar and all of Airstream. The library author gets to stipulate what is supported in shared code and users who want more are free to contribute it :-)

A macro-based approach or even transforming source files with an SBT plugin is another way to do it for sure. I think even in that case there will be some limits as to what is supported and what not, because otherwise parsing might become too complicated.

Regarding other libraries for the domain, the only non-virtual dom one that I know that supports shared templates is WebSharper (C#/F#). But that uses a bunch of custom attribs in the HTML, so it's a very different beast.