Typescript support

[jimafisk]

Svelte now supports Typescript: https://youtu.be/djRkyyBoIKs

We'll need to update the build process and potentially add a CLI flag and new plenti.json key to enable this.

[Himujjal]

How do I help you with this? I can't work without TypeScript. Plenti is impressive!

[jimafisk]

Thanks @Himujjal! I would need better understanding on how to implement this without Rollup. Do you know if there's a way to run TypeScript in the Svelte compiler directly?

The key to TS support seems to be svelte-preprocess, which we have a discussion going here: #134

Here are some details about using svelte-preprocess with esbuild/snowpack: sveltejs/svelte-preprocess#177.

[Himujjal]

I looked into svelte-preprocess and stuff. BTW you say plenti works without using node. Can you link me to how plenti works internally?

Maybe I can brainstorm a solution. And we can further discuss this in #177

[jimafisk]

We don't have great docs from the perspective of someone who wants to write Go code to help build out the Plenti "engine" unfortunately. It's something I'd like to improve on, but we're not there yet.

At a high level, the way it works is we use the Svelte compiler directly by running the code in V8go. We load each Svelte component to a specific namespace in a large virtual context, and then pull out the dom js and css and write it to the filesystem for our clientside app, then recompile the components to get the ssr js and run the .render() method on them to generate static HTML for every route. The challenge is Svelte is written to be run in nodejs, but since we're using v8 directly we have to preprocess the code to remove unsupported features like commonjs require statements.

Regarding TS support, I was hoping there would be a compiler option in Svelte we could set, but that doesn't seem to be the case. Is svelte-preprocess essentially transpiling the TS code to regular JS that the Svelte compiler can understand? This might be something we need a bundler like esbuild for, but if you know of a way to do this without a bundler I'd be interested in hearing it. I'd definitely appreciate any brainstorming you want to share. If you're more comfortable with Node, feel free to drop some JS snippets right in this issue to demonstrate how this could work and I can try to put them within the context of Plenti to see if it's something we could integrate. Thanks!

[braebo]

Not sure if this would help: https://github.com/lukeed/svelte-preprocess-esbuild

[jimafisk]

Thanks @fractalhq, I was looking at that too. It sounds like it does what we want and Luke makes good stuff. I'm confused by the rollup example in readme though. The svelte-preprocess-esbuild module must call out to esbuild just to compile the ts. If we're going down that route, I think I'd rather just use the esbuild api directly from go. I'd be really interested in knowing if this is possible to do without using a bundler at all.

[Himujjal]

Thanks for the architechture @jimafisk . So, you are using V8go. I could understand your reason to avoid ESBuild. There is one thing that can be done though. I don't know if you will agree.

1. A Svelte Parser in Go that would separate a Svelte file into 3 parts: Template, Style, Script
2. Then use our own pre-processors like sass and TypeScript (using V8go) to compile the files to relevant Svelte like format (HTML/CSS/JS)
3. Run the Svelte compiler now.

Basically build our own Svelte pre-processor in Go.

If you say this would be fine and would like to continue, I am ready to invest time in a Go module: Svelte-preprocess with inbuilt SASS and TypeScript Out of the box. I think its time we all look into alternative Svelte compilers soon.

[jimafisk]

I think you're spot on @Himujjal, I've talked about something similar with @padraicbc as well. I thought this article did a good job of explaining what the Svelte compiler is doing under the hood in a simplified example. We'd have to break out the HTML, Style, and Scripts like you said. The hard part would probably creating the JS AST, I think Svelte uses acornjs for this, we could potentially still use v8go but getting the AST might require a modified v8. Maybe there's another way to do this without CGO entirely, but the other solutions I've tried for analyzing JS in Go were quite slow.

Preprocessing Svelte's nodejs code to work with Plenti was the easiest way for us to get something working, but it creates performance and sustainability challenges. Replicating the Svelte compiler natively in Go could greatly improve those aspects, but it might be a rather large undertaking. I'd love to hear more about the approach you're proposing, it definitely seems like something we'd be interested in.

[Himujjal]

I think a minimal parser that separates the 3 parts should be more than enough. Once we have the respective 3 strings, we just need to process them through. Then maybe use native Go preprocessors on the strings. Or utilize ones found in npm though V8. A full fledged compiler wont be required for this case.

Let me write one in a day or two though. I dont think it should be really tough.

Offtopic: Regarding a full fledged compiler for Svelte, I would rather write it in Rust or Nim though because of their fantastic extern C ABI. That way any programming language can use it.

[jimafisk]

Ok that makes sense, if we're just preprocessing the svelte htmlx code before running it in the compiler the normal way that should be something we could pull in without doing a major overhaul. I'm excited to see what you come up with! Thanks!

[braebo]

Just want to drop these in case they can be of use: Tree-sitter go bindings https://github.com/smacker/go-tree-sitter and https://github.com/Himujjal/tree-sitter-svelte.

[Himujjal]

Just want to drop these in case they can be of use: Tree-sitter go bindings https://github.com/smacker/go-tree-sitter and https://github.com/Himujjal/tree-sitter-svelte.

Yeah. tree-sitter-svelte was built by me. It parses Svelte quite well as of now. I planned the same too. Use go-tree-sitter with a tree-sitter-svelte go package to create the differentiation. Will work on it later though.

Is there a TypeScript transpiler for Go?

[padraicbc]

Just want to drop these in case they can be of use: Tree-sitter go bindings https://github.com/smacker/go-tree-sitter and https://github.com/Himujjal/tree-sitter-svelte.

Yeah. tree-sitter-svelte was built by me. It parses Svelte quite well as of now. I planned the same too. Use go-tree-sitter with a tree-sitter-svelte go package to create the differentiation. Will work on it later though.

Is there a TypeScript transpiler for Go?

The only thing I know of with any es6 or typescript support is https://github.com/dop251/goja

[Himujjal]

This issue problem can be solved easily though. I will complete this issue within this week.

Offtopic:

@padraicbc I researched a little bit. ESBuild and SWC can compile TS quite well. But they lack type checking which IMO is the major driving force behind TypeScript.

I will in future start building a full fledged Rust compiler for Svelte using tree-sitter and SWC. Anyone interested can ping me. Wont start it alone though.

[jimafisk]

I had looked at goja early on, but the benchmarks made it seem like a V8 Go binding was a decent amount faster: dop251/goja#2 (comment)

@Himujjal I would be interested in helping with a full fledged Svelte compiler if it was written in Go. By the look of it, you have a better understanding of what's involved than I do and may have solid reasons for why you'd want to use Rust instead. Although it's not the same problem, I thought it would be worth sharing Evan Wallace's response on why he chose Go instead of Rust for Esbuild:

Author here. I think the Rust vs. Go question is interesting. I actually originally wrote esbuild in Rust and Go, and Go was the clear winner.

The parser written in Go was both faster to compile and faster to execute than the parser in Rust. The Go version compiled something like 100x faster than Rust and ran at something around 10% faster (I forget the exact numbers, sorry). Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread.

The Rust version also had other problems. Many places in my code had switch statements that branched over all AST nodes and in Rust that compiles to code which uses stack space proportional to the total stack space used by all branches instead of just the maximum stack space used by any one branch: https://github.com/rust-lang/rust/issues/34283. I believe the issue still isn't fixed. That meant that the Rust version quickly overflowed the stack if you had many nested JavaScript syntax constructs, which was easy to hit in large JavaScript files. There were also random other issues such as Rust's floating-point number parser not actually working in all cases: https://github.com/rust-lang/rust/issues/31407. I also had to spend a lot of time getting multi-threading to work in Rust with all of the lifetime stuff. Go had none of these issues.

The Rust version probably could be made to work at an equivalent speed with enough effort. But at a high-level, Go was much more enjoyable to work with. This is a side project and it has to be fun for me to work on it. The Rust version was actively un-fun for me, both because of all of the workarounds that got in the way and because of the extremely slow compile times. Obviously you can tell from the nature of this project that I value fast build times :)

To be fair, I'm self interested in using Go for the benefit of this project. Either way I think folks here would be interested in hearing your progress, so even if you start something in Rust feel free to post back here!

[Himujjal]

I am thinking of choosing Rust/Nim simply because of portability not because of speed. By portability I mean to say that this library can be used by almost all the languages in the world using Rust/Nim's C ABI. Go doesn't have an external C ABI. So, let's say someone from Ruby world wants to invoke Svelte compiler someday, they will not be handicapped. In our example here, you are handicapped by Svelte compiler being written in JavaScript (i.e. you have to use V8 binding instead of a small static library).

I might as well do it in Nim since Nim is a lot like Go in terms of readability and its far easy to write in Nim. It is also a GC language. Would you join me if its in Nim? If you know Go, it would take 5 hours to learn Nim. lol. Its almost the same.

Contrary to what EvanW says, Speed won't matter much IMO as Go, Nim and Rust almost give out equivalent performance in regards to such a use case or the speed difference will be negligeable. The main aim is portability.

Anyways, I have updated go-tree-sitter to parse Svelte grammar. A little bit more work to do on the current Go compiler.

[saolof]

@Himujjal I would be interested in helping out with a Rust or Nim compiler for Svelte, for the exact reasons that you mention (i.e. being able to call it via a C ABI so that it'd be possible to wrap it in a self-contained Python or Julia package, without any dependency on a separate process). I'm currently busy with scheduling a thesis defence, but I should have more than enough free time in the fall.

Regarding Rust or Nim, I'm fine with either option, though I have a slight preference for Rust purely because it's become mainstream and should be more easy/boring to maintain in the long run, and because I'm already familiar with it.

[Himujjal]

@saolof Cool. Thanks. Lets start next week. I will open a new repo and grant you access. I already have the parser done. So, I will setup the basic to get started.

Regarding Rust or Nim or Zig. I would prefer Zig but since you want it in Rust, I would go with that. I did not want to start this alone. Thanks for the motivation.

[jimafisk]

@Himujjal & @saolof that's great that you're pursuing this! Can you share the repo link back here so interested folks can follow along? @padraicbc and I have been discussing doing something similar in Go: https://youtu.be/Ql6cZJ-Udkg. It might be cool for us all to chat at some point to share ideas even if they are for separate projects.

[Himujjal]

@jimafisk
Yeah sure. I will share. Its always good to have a competitor. That way we can share ideas. Plus using a Rust/Zig package in Go is very simple. So, there is no loss in whatever methods whoever chooses.

[saolof]

One advantage of Rust is that several groups have already built reliable high-performance Javascript parsers. One option for the javascript code transformation could be to piggyback off of SWC, which is a really impressive transpiler crate written in Rust that is intended to be a babel & TSC replacement, but also exposes an API to its parse tree that can be used to implement code transformations.

Their repo is here: https://github.com/swc-project/swc , and they have a discussion page here https://github.com/swc-project/swc/discussions where we could start a conversation about a possible implementation of a svelte compiler using their work.

This would also have the bonus that it'd be automatically compatible with an excellent transpiler, which enables targeting any individual version of JS.

[Himujjal]

https://gitter.im/sveltec/community - Svelte compiler for non-JS languages coommunity
https://gitter.im/sveltec/sveltec-rust - Svelte compiler in Rust
https://gitter.im/sveltec/sveltec-go - Svelte compiler in Go

I made a gitter channel for anyone who wish to join the development. @jimafisk @saolof added you guys. So, feel free to chat over there. Let's leave this issue alone.

[jimafisk]

@Himujjal and @saolof not sure if building a Svelte compiler in Rust is something you're still interested in, but I stumbled across another project working on this: https://github.com/pintariching/rustle. Might be cool to consolidate efforts!