Document the architecture behind Svelte

[StarpTech]

Hi,
I'm really interested in how svelte internally works and which problems and challenges you had faced. I could read the whole source code but the author can provide a much better picture. Svelte is able to compile itself away this is amazing because when the API is settled you can share performant, runtime-independent web components.

Interesting points

• Code generation
• Compile-time static analysis
• Role of Typescript
• Your motivation
• Roadmap
• .....

[Rich-Harris]

Yes! This could make for a good blog post. I have no idea when I'd get round to it, but I like the idea.

Let me try and do the short version for the bullet points you raised:

Code generation and static analysis

As with any compiler, the first step is to generate an abstract syntax tree (AST). Svelte actually uses three parsers — its own internal one, for HTML and {{tags}}, Acorn for JS, and css-tree for CSS. Put those together, and you get a tree-like data structure that you can 'traverse' to find out things about the component (such as which data properties it expects, what methods it declares, what lifecycle hooks it has and so on.)

Once we've figured out the shape of the component, we can move on to the code generation step. Each component has a main fragment, which is what all the markup represents — this gets created by the create_main_fragment block. As we traverse the AST and find things inside the component (like elements and text nodes), we add code to the block's methods using an internal utility called CodeBuilder which makes it slightly easier to combine different statements.

Sometimes, we'll encounter if blocks and each blocks, etc — these contain their own fragments, so we create a new block for each of those. Each block has its own methods for creating, updating and destroying the DOM, which allows Svelte to work with the DOM directly rather than incurring the overhead of a virtual DOM.

At the end of that process, we basically just concatenate all those blocks and add some boilerplate. There's a little sleight-of-hand involved, for the sake of debugging — when we're using code that the component author actually wrote (as opposed to stuff the compiler creates from scratch) we don't just copy the code, we insert a marker like this: [✂42-46✂]. That means 'insert the characters between 42 and 46', which allows us to record the source location to generate a sourcemap from the finished code.

Role of TypeScript

TypeScript is a purely internal choice, and not something that users of Svelte need to know about at all (though we may provide some form of TypeScript-in-components support in the future). I was sceptical for a long time, but I've become a huge fan. It makes for a much nicer development experience. For example, you start typing the first few characters of a method or property, and it will offer to autocomplete the rest. Or if you're not sure what the arguments to a function or method should be, just hover over the name and it'll look them up for you. I tend to be a fairly careless programmer, and TypeScript means I write way fewer bugs (and when I do write bugs, they're much easier to find).

Downsides: there's quite a lot of work updating an existing project to use TypeScript, and it complains if your dependencies aren't also written in TS (or at least have type definitions available). But overall I'm a happy customer, and I don't think it's deterred any would-be contributors (though that's obviously unknowable).

Motivation

Partly it's just the same itch that I think most programmers share — solving tricky problems and whittling something down to the most efficient possible implementation is a satisfying thing to do. And there are less savory motivations, like the ego-inflating buzz you get from seeing people using your work or borrowing your ideas.

But it's also because of my job. I work as a graphics editor for the New York Times, and before that the Guardian, and as everyone knows news websites tend to be full of stuff that slows your browser down — analytics, ads, social media crap, comment widgets, etc (the NYT and the Graun are among the lesser offenders, but the entire industry needs to improve on this front). Because of that, loading a framework like React or Vue for a simple interactive embedded in an article just isn't a good option. We need tools that allow us to build extremely quickly, and not worry about performance, app size, CSS leakage, and all that stuff. And there's a constant tension between the desire to reuse code and the fact that every news graphic is unique, with unique requirements. A declarative component-driven approach is helpful in that context.

All devs have these issues to some degree, but I believe they're particularly acute in the news biz, where deadlines are often measured in hours (no time for optimization!), the browser environment is unusually hostile, and programming might only be about 30% of the job. So we're probably even more motivated than most devs to find good solutions. (I don't think it's a coincidence that D3, Underscore, Backbone and CoffeeScript all came out of the news business, just to list projects by former members of the NYT graphics desk.)

Roadmap

The big priority for me at the moment is Sapper. The idea is to have a first-rate way to build Node apps using Svelte. It's inspired by Next.js, but departs from it in a few important ways. It's a very early work-in-progress but I'm very excited about — the early results are very encouraging.

Beyond that, there are a few things I want to work on:

• Transitions are pretty good in Svelte already, but they could get a lot better — more control over timing, methods of transitioning between nodes, and so on
• As custom element support improves, we'll need to spend more time making sure that Svelte-generated custom elements are rock solid
• There are some syntax improvements we could make. For example I'm not sure you need both curly braces either side of a {{tag}} — JSX seems to do fine with just one.
• Devtools and editor plugins!
• The next frontier in app performance, I believe, is whole-app optimisation. Right now, if you create a component that has a {{foo}} tag, then Svelte has to generate code that updates the tag if the value of foo changes. But if it could see that the only time that property is set is in when a parent component does <Thing foo='bar'/>, then it could skip that — less code, and even less to do when it's checking for state changes. There are lots of similar situations where Svelte could generate better code if only it knew how components were being consumed (similar to how tree-shaking allows a module to skip some of its exports if it knows they're unused). That's a tricky problem that the current generation of bundlers aren't really equipped to deal with.
• Glimmer are doing some interesting work around generating binary bytecode representations of templates. I'm waiting to see the numbers, but there are definitely some theoretical wins from doing that instead of generating JavaScript. Svelte could have a 'bytecode mode' that switched to bytecode if your app was large enough to warrant the overhead of the interpreter that you need to ship in order to use it. (This is the sort of thing that Sapper could even do automatically, based on certain heuristics.)

In short, lots to do!

Sorry this answer got a bit long, I didn't have time to write a shorter one 😀

[StarpTech]

Hi @Rich-Harris thank you for the great insights. I really appreciate your work and I like your attitude. I'm interested and trying to be a part of this movement ;)

[marvinhagemeister]

@Rich-Harris That was a really great dive into the internals. Thank you so much for taking the time to write down your thoughts and perspective 👍

[hrj]

Disclaimer: I am mostly familiar with backend dev, and beginning to investigate the framework land in frontend. I hope this is a good architectural level question:

Does svelte do anything along the lines of virtual DOM and DOM diffing? I didn't see any mention of it in the svelte docs, but I suppose this is an important feature of other contemporary frameworks, which allows rendering to be stateless, in that, the whole app can be rendered fresh from a given state, without causing a UI jank.

[StarpTech]

Hi @hrj read the statement by Harris.

... Each block has its own methods for creating, updating and destroying the DOM, which allows Svelte to work with the DOM directly rather than incurring the overhead of a virtual DOM.

[StarpTech]

@hrj svelte doesn't need a virtual DOM because it has no runtime.

[hrj]

Thanks. I think I have understood it (please correct if wrong):

From the templates, the compiler knows at compile time which DOM nodes need to be mutated for a given state change. The generated code has this knowledge encoded within itself, and hence it doesn't need to diff against a Virtual DOM. It just makes the precise change based on the change that happens to the state.

[StarpTech]

@hrj exactly it's quite easy because you have to use setter and getter methods to update your state.

[dummdidumm]

Closing due to no activity and no intermediate plans to make this happen. But a maintainer basically did what was requested and did a small series of blog posts on Svelte internals. That together with Rich's comment above should be enough for now.

• https://lihautan.com/the-svelte-compiler-handbook/
• https://lihautan.com/compile-svelte-in-your-head-part-1/
• https://lihautan.com/compile-svelte-in-your-head-part-2/
• https://lihautan.com/compile-svelte-in-your-head-part-3/

[pngwn]

I still think this is important for onboarding. I love the community stuff that is going on but we need something here as well.