RFC: Wasp IDE Integration

[craigmc08]

Motivation

One of the main point points in Wasp currently is the lack of editor integration while developing a Wasp project. This manifests in two separate ways:

1. No editor language features for .wasp files

2. Code generation breaks the existing JS IntelliSense in editors

While there does exist a Wasp extension for VSCode (https://github.com/wasp-lang/vscode-wasp), it is out of date with the current DSL version and only adds syntax highlighting. Thus, the existing solution only gives a partial solution to (1) and does nothing for (2).

Wasp is a tool that exists to make the process of developing full-stack web apps easier, and this is one place where Wasp makes it harder to write code, so these issues must be resolved.

In this RFC, we first outline what a Wasp extension would need to do generally. Then, we give more specifics on how an implementation of an extension for VSCode would look.

Requirements

1. Wasp DSL language features
   1.1. Syntax highlighting for .wasp files
   1.2. In editor parse-/type-error reporting
   1.3. Automatic formatting
   1.4. Autocompletion for variable names
   1.5. Autocompletion for dictionary keys
   1.6. Autocompletion for JS import statements

2. JavaScript IntelliSense. Support all existing editor language features for JavaScript. Working in a Wasp project should not be a downgrade from developing outside of one in terms of JS language support.

Requirements (1) and (2) are completely disjoint and can be implemented independently of each other.

Additionally, both of these requirements can be developed incrementally: even just 1 feature from the list is an upgrade from the existing tooling. The sub-requirements of (1) are ordered in decreasing importance. For (2), the most important language features are:

• Autocompletion/auto-import for generated code
• Go to definition for queries and actions

Implementation

The implementation section is split into two components. One for Wasp language features, and one for JavaScript IntelliSense.

Wasp Language Features

To fulfill requirement (1), we would implement a Wasp language server conforming to the Language Server Protocol (https://microsoft.github.io/language-server-protocol/). Through this, we can support all sub-requirements.

The language server would be written in Haskell, using the lsp package and would rely on the existing Analyzer in waspc for parsing and type checking.

Along with the language server, a small extension would need to be written for each editor to integrate the language. For VSCode, there is vscode-languageclient, which makes it extremely quick to connect a language server to VSCode.

Alternatives

An alternative is to implement syntax highlighting and formatting outside of a language server, e.g. using TextMate and prettier. Using a language server for these features is better:

Pros of a language server:

• All Wasp language features implemented in one place
• Possibility to support semantic highlighting/formatting (i.e. route declarations are usually written on one line, instead of having a newline after the opening {)
• Can use language server across multiple editors

Cons:

• Small start-up time on Wasp language server could cause a gray flash of text when opening a .wasp file
• A TextMate grammar has already been written for the existing VSCode extension

JavaScript IntelliSense in VSCode

For (2), we focus just on an extension for VSCode. This is because it appears that the most feasible/low-cost way to do this ends up being very editor specific. VSCode was chosen because it is a popular editor with a very strong community for extensions.

The main idea of the extension is to create a temporary virtual file system (virtual workspace) reflecting the code generated by Wasp to give to the TypeScript server (which is also responsible for JavaScript language features). Then, all language feature requests to the real workspace are answered by querying the virtual workspace. As an example, see the following diagram and accompanying description (code taken from Wasp TodoApp example):

1. User highlights a reference to getTasks in MainPage.js and presses F12 to go to the definition.

2. The extension intercepts this request and sends it to the copy of MainPage.js in the virtual workspace.

3. The TypeScript server responds to this request with a location in the generated query file for getTasks.

4. The extension uses source map information on the generated query file to map the result to the correct location in the source file, queries.js, using main.wasp to know this is the location of the implementation.

5. The extension responds to the user query with the modified answer, causing the editor to open the correct real file (ext/queries.js) and highlight the function.

Note: In actual generated code, @wasp/queries/getTasks contains code to make a network request. This is not reflected in the virtual workspace so that the TypeScript server can do type inference on the body of the function when you use it in the client, improving IntelliSense results.

In short, the extension:

• Builds a virtual workspace that reflects the code generated by Wasp
• Forwards language feature queries from real files to virtual files

These features would only run when the open workspace contains a main.wasp file.

Alternative Implementation Strategies

Add a jsconfig.json file to the root of a Wasp project

{
  "compilerOptions": {
    "jsx": "react-jsx",
    "lib": ["ES2015", "DOM"],
    "paths": {
      "@wasp/*": ["./.wasp/out/web-app/src/*", "./.wasp/out/server/src/*"]
    }
  }
}

This would cause the TypeScript server to look in the generated output of Wasp for imports that start with @wasp. This is uncomplicated to set up and gives OK results. There are several problems with it, however, that make it not viable. First, go-to-definition and other navigation commands could direct users into generated output, which feels clunky. Second, it doesn't support all language features (ex. auto-import when tab-completing a query). Third, it relies on wasp start to be running to stay up to date with edits.

Including a custom TypeScript language server in the extension

We could re-use parts of the VSCode TypeScript extension and customize it to bake in support for Wasp projects. This gives us complete control over the TypeScript server. The first drawback to this approach is that VSCode merges results from multiple extensions providing support for one language, so we would have duplicate results. The second is the major complexity increase of developing and maintaining a fork of parts of TypeScript, which would put a large burden on staying up-to-date.

Using a TypeScript Server plugin

It is possible to write plugins for the TypeScript server and enable them through a VSCode extension. However, it appears that we can only adjust language feature requests and responses without access to internal compiler state, so we would have to parse JavaScript files ourselves to provide extra IntelliSense, which would be very unstable. Even if we did not have to do that, we would end up with the same maintenance burden of maintaining a fork of part of a TypeScript compiler.

Changing Wasp's Generated Code

The final alternative discussed here is changing the layout of how Wasp generates code.

The first change is to keep the query and action functions called in the front-end in the same location they are imported from, i.e. import { getTasks } from '@ext/operations.js would result in the client-side operation being exported from .wasp/out/web-app/src/ext-src/operations.js. This would allow import { getTasks } from './operations.js in front-end code to work and automatically give IntelliSense (breaking change if implemented). This is how Blitz.js does it.

The second change would be to have common Wasp code as an npm module that is installed in the output. Using a jsconfig.json to direct the TypeScript server to the web app and server node_module folders for IntelliSense on dependencies. This could take a couple of forms:

• A published module, wasp. Imports would need to replace @wasp with wasp (breaking change if implemented). Additionally, this might not be successful depending on whether any of the APIs in generated code is dynamic and can vary across projects.

• A local module, @wasp. This is more flexible and yields the same quality of IntelliSense as the first method.

This has similar issues as the previous two strategies with needing the full Wasp code generator watching for changes to update IntelliSense (although only in the case of a local module). This seems like the best alternative strategy, but making changes to how Wasp works to make IDE integration easier does not seem like the way to go.

Summary

The two independent tasks this RFC results in:

1. Developing a language server for the Wasp DSL. This path is well-defined and has been done before, so there is a good amount of help online.

2. Working with JS language features in a Wasp project. There do not seem any extensions that do something similar: other projects in the same space as Wasp have a source structure that mirrors the build structure closely enough that no trickery is needed. This is the piece of the implementation that will be more time-intensive.

Proposed Next Steps

1. Implement syntax highlighting in a Wasp language server to replace the existing Wasp extension

2. Add diagnostic reporting to the Wasp language server

3. Implement the most important language features (auto-completion, go to definition) for Wasp projects

At this point, IDE integration will be in a much-improved state and the remaining features can be picked off one by one.

[Martinsos]

This is awesome @craigmc08 !!!
I quickly skimmed it, here is quick feedback based on that:

1. Yes, we can certainly proceed with Wasp Language Server (WLS) that will have syntax highlighting for start and then more advanced stuff, there is no question there.
2. We don't need to worry about breaking changes, we are in Alpha, we are all about breaking changes :D.
3. Current structure of Wasp project is not so much a consequence of optimal design choices, but more of what was easy to get working. So if we need to adjust this in order to get other stuff working, like IDE support for JS, that might be completely fine, as long as it makes sense all together.
4. Maybe relying on wasp start running is not so bad? If we can rely on that, it can make some stuff easier, right? I am tempted to say that we take it as granted that wasp start running is required for IDE tooling to work correctly. Btw we also have waspc compile command, so we can in theory run that one on file changes, we don't need to actually spin up the development servers, of course.
5. It would be great if we could get as close as we can to implementing a solution that works for many editors out of the box and is not VSCode specific. But let's see, if there is a huge difference in "price", we can go with VSCode for start. I hope not though.

I will give it a much more thorough look tomorrow and will write my thoughts on different directions that you proposed!

[shayneczyzewski]

Great writeup, Craig! 🥳 I think you covered the current situation well, and I like the idea of the directions proposed. Indeed, these sound like two distinct approaches, but both required from a DX perspective. (As you rightly said, developing apps in Wasp should not feel like a step backwards while developing!) I like starting with the Wasp language server, as it has fewer unknowns and would add real value for making .wasp files smarter across all IDEs.

The JS support does sound like we have options, but each with pros/cons. Great job researching all of them. I think a VSCode-centric approach for a first go makes total sense. The market share is there, and the tooling sounds most amenable for extension. I'm also unsure if there is a cross-IDE approach to this problem, aside from changing the generation (which also sounds reasonable).

The VSCode virtual filesystem approach sounds really neat and elegant! Could this also work in conjunction with the Wasp LSP to go to those definitions from .wasp files? Also, for one of your alternative implementation strategies you note:

Third, it relies on wasp start to be running to stay up to date with edits.

Wouldn't we still need this in the virtual filesystem case as well? I may not fully understand what the virtual filesystem is abstracting though. If it is completely independent of the generated project and just mapping across file paths that is neat too! Anyways, will think more on it but very nice RFC!

[craigmc08]

4. Maybe relying on wasp start running is not so bad? If we can rely on that, it can make some stuff easier, right? I am tempted to say that we take it as granted that wasp start running is required for IDE tooling to work correctly.

My opinion is that when you open your editor to a Wasp project, tooling should just work with no extra steps. We could run wasp start from the extension, but then the user couldn't run it in their terminal (port conflicts) to see the error messages outputted by wasp easier.

Btw we also have waspc compile command

Didn't know about this actually, could be useful for some of the alternative approaches if the main one doesn't pan out.

5. It would be great if we could get as close as we can to implementing a solution that works for many editors out of the box and is not VSCode specific. But let's see, if there is a huge difference in "price", we can go with VSCode for start. I hope not though.

Definitely agree. And with your comment about breaking changes being not too bad, the last alternative will be a good place to investigate more closely.

Looking forward to your thoughts on the different directions, @Martinsos!

[craigmc08]

Thanks for the feedback @shayneczyzewski!

The VSCode virtual filesystem approach sounds really neat and elegant! Could this also work in conjunction with the Wasp LSP to go to those definitions from .wasp files?

I think we may be able to get Go to Definition for imports in .wasp files working without talking to the virtual filesystem, since the import statement already tells us which file and symbol in the source code to find the definition.

Also, for one of your alternative implementation strategies you note:

Third, it relies on wasp start to be running to stay up to date with edits.

Wouldn't we still need this in the virtual filesystem case as well? I may not fully understand what the virtual filesystem is abstracting though. If it is completely independent of the generated project and just mapping across file paths that is neat too!

The virtual filesystem is fully separate from the generated project as currently designed, so it wouldn't need wasp start running.

[Martinsos]

As I said earlier, awesome work @craigmc08 ! Really excited about the whole thing and there are many different approaches that you described, I am confident we will figure out a good solution among all these ideas.

Below I first re-iterate the problem and general idea for solutions based on your RFC @craigmc08 , to check that I got stuff right + maybe give a bit different perspective / phrasing.

Then, I go through each approach you described and explain how I understood it.

All of this is littered with questions that ask for confirmation of my understanding or ask for more details on certain approaches / directions!

Looking forward for your comments on all this!

Problem

The reason why we don’t have JS/TS IDE stuff working for JS files in Wasp is that there is some magic being done by Wasp, specifically:

1. In Wasp project they are not a part of Npm package (no package.json) → actually there are none of those additional files like prettier or some other configs next to them. Instead they are pretty isolated, standalone. Therefore, JS/TS IDE is confused, it can’t make sense of the project. In generated project all is good, but IDE doesn’t see generated project, it sees Wasp project.
2. In Wasp project, JS files in ext/ have JS imports that are “magic” → they don’t really exist in Wasp project, however they do exist in generated project (actually they are transpiled by Wasp so they are correct in the generated project). So again, in Wasp project some JS imports just don’t exist, but in generated project they do. So JS/TS IDE can’t follow those paths, can’t figure out what is being imported, and that messes up everything.

So to recap: JS files in ext/ are missing context of the npm project package they are really in + they have JS imports that have incorrect (non-existing) paths.

When we(Wasp) generate Wasp app, we move the ext/ files to their correct places in their corresponding npm packages (client, server) and we(Wasp) rewrite JS import paths that start with “@ext” or “@wasp”. That is what generated project looks like. However IDE doesn’t know that. So the point is to somehow teach IDE about that, or to trick it to behave like it is in the generated project.

One thing we also would like to do is to make go-to-definition work a bit differently than one would expect → in Wasp, sometimes you wrap JS functions in order to get their “powered-up” Wasp counterparts, which then you again use in JS. Following up definitions of those “powered-up” counterparts wouldn’t be great as it would take us into generated code. So we would like to instead make go-to-definition take us to the wrapped(original) JS function. That is not entirely semantically correct, but it is practical.

Solutions

In all solutions, I believe we need an up-to-date generated project → I don’t see how we can get correct information otherwise. We don’t need wasp start for that, we just need Wasp to listen for file changes and re-compile on any. That we already have and it is called wasp watch - I believe it is not exposed at the moment in the CLI but we can easily expose it.

1. Redirecting

When IDE asks LSP about file F in Wasp project, we instead ask LSP about file F’ in generated project and return that answer (which we possibly also need to map again to Wasp source project). Since files are directly copied to generated project, this should be easy. Except for one small problem → we rewrite import paths. But, I believe source maps should be the solution for problems like this?

How can this be done though? Craig mentioned LSP plugins → is this it? This sounds pretty simple? I guess this should solve issue with files in wrong places, and with magic imports. Only thing that remains unsolved is go-to-definition, but that is a separate thing, can’t be solved in same manner.

2. Faking

We make it look like file F in Wasp project is in the environment we want IDE to perceive.

So we make it look as there is package.json somewhere close, there are files in paths to which those magic imports point, … .

One way we can do this is by basically writing generated project in the root dir of Wasp project. But that sounds terrible, it will cause a mess, we might have some files (.gitignore for example) collide, we would need to somehow .gitignore it all but I doubt if we even could, … → complete mess. So this doesn’t sound like a possible option.

Another way might be enabled by virtual workspaces that VSC offers → via that we can describe the whole “fake” file system environment, exactly what we need. But that will then work only for VSCode.

I have hard time imagining another way to do this: maybe via symlinks? But I don’t think we can make that work, I can’t see how? Would a bit of restructuring of Wasp project help? Maybe if we put ext/ one level deeper → have client/src/... and server/src/... and iso/src/... → then next to src/ dirs we can copy/symlink the whole generate project really (+ gitignore everything) and there we go! But that is ugly?

NOTE: Btw I can’t imagine any solution functioning without watching for file changes and regenerating project when that happens (so wasp watch functionality). I guess this could be done all in memory in theory, maybe? So FileDrafts wouldn’t be written to disk but to something else?

Future considerations

There are two things we will likely want to do in the near future that might have significant impact on choices we make here:

1. We might reorganize code from ext/ into client/, server/ and iso/.
2. We will want to support multiple Wasp files/modules. They would be able to import each other, and declarations/identifiers in them would be scoped via their module. Meaning that two wasp files could e.g. have the same query called getUser and you could also import either of those in your JS file. However, that means that JS imports of Wasp stuff can’t be global as they were so far import { getTask } from "@wasp/queries", instead they should be smth like import { getTask } from "../foo/tasks.wasp" or import { getTask } from "main.wasp" .
   I guess we could also do smth like import queries from "@wasp/queries"; const getTasksQuery = queries.foo.tasks (scope them under the module path from which they come), but that seems a bit weird.

So, the direction we choose should be able to support these!

Analysis of approaches that Craig suggested

1. VSCode Virtual Workspace

General idea of Virtual Workspace makes sense to me!
Instead of having the generated project on disk, we have it in Virtual Workspace, which I assume is in memory? How do we generate it though → we make it so that Generator, in its last step, doesn’t write FileDrafts to disk, but instead writes them to this Virtual Workspace somehow?

Also, you mention that in virtual workspace we make @wasp/queries/getTasks look like the query function from the server, instead of the actual code that does network request, since this improves IntelliSense results. But does it? I would expect the opposite → I would expect this to be confusing, since it will be showing type of one function, but in reality in your code you are dealing with another function (one that does network requests). To me it would make sense that it looks like what it is, which is a function that does network request, so that types are correct? The only thing where I would “cheat” is when sending user to the definition → then I would send them to the original JS of the Query. Or maybe to the Query declaration in Wasp hm, not completely sure. Ok that is good question → should go-to-definition of smth imported from Wasp take them to that declaration in Wasp, or to the underlying JS implementation? That was referenced via ExtImport in that declaration? I would maybe even vote for Wasp declaration.

Found this quote at https://code.visualstudio.com/api/extension-guides/virtual-workspaces:

The rich language extensions that ship with VS Code (TypeScript, JSON, CSS, HTML, Markdown) are limited to single-file language support when working on virtual resources.

Also this:

Work is under way that will add file system provider support to LSP. Tracked in Language Server Protocol [issue #1264](microsoft/language-server-protocol#1264) .

Could these be an issue?

How is this approach different from generating project on the disk and then extension forwarding language features from real files to virtual files? I guess it is the same approach really, the question is just are we writing generated project to the disk (no virtual workspace) or into memory (virtual workspace), right?
If so, shouldn’t we prefer the solution that writes to disk, since it is not editor-specific, and sounds a bit simpler anyhow? Or is it expected from language extensions that they don’t do anything on the disk? However, Wasp is somewhat specific, so maybe it is ok if we do it?

This approach should fare well with both future considerations that I mentioned (client/server/iso, different import paths), right?

Btw how does this intercepting of LSP queries work, which mechanism enables that? That is something we need to implement on the level of editor extension, right?

2. jsconfig.json

Ok, so this one makes “@wasp” import paths correct. However, that won’t work if they are rewritten somehow, right? It only helps if prefix is different?

But, what about node_modules, can it see those correctly?

Does it miss package.json file → is that something TS extension cares about?

go-to-definition → can’t we intercept that one same like we would do in approach #1?

Why doesn’t it support all language features (you mentioned auto-import)?

Reliance on wasp start → ok, that is actually probably not even an issue, since we can do smth like wasp watch, right?

I do like the approach (1) better since it sounds more rounded and correct, but this one actually sounds like it might get pretty close with very little effort (if we can resolve some of these issues)? Although, if I got it correctly it can’t support advanced path rewriting, so that is an issue in the long term.

3. custom TS LS that our extension would drive

This sounds tempting, however you are right that it sounds scary from the maintenance perspective.
On the other hand, maybe it is not so hard to maintain it? Maybe changes are pretty small?
Regarding duplicate results → can’t our Wasp extension disable the other TS extension?

However I am not sure how we would even modify it → teach it about the generated project? I guess. Sounds complex.
We need to do this for every editor also which is not good, especially taking into account the difficulty of maintainance.
So yup, certainly scary from the maintenance side.

4. TS LS plugin

Hm this is interesting. This plugin could be used to intercept queries / responses and do source mapping, right? Isn’t that exactly what we need? How would we intercept the lsp queries anyway, in approach #1, I imagined we would need smth like this?
This sounds like it would be much easier to maintain than the approach #3. But are these plugins VSCode specific thing? If not, later supporting other editors could be a pain.

5. Changing Wasp’s Generated code

Ok, so the idea is that we generate file with Wasp Query in the same path where original JS query is coming from (the one referenced via ExtImport). Then, since later JS import will be referring to the path that exists both in source project and generated project, intelisense will be working. However, generated Wasp query is not exactly the same as original JS query → so intelissense will not be correct, right? Also, if we are in the generated project generating this file in place of original file, which we also want to keep in the generated project, where is the original file? I guess that works ok in BlitzJS because they replace the file completely with new one hm and one file is one query, while in Wasp that is not so.
Sounds a bit tricky all together, I guess we could try to do some tricks but it sounds fundamentally a bit wrong, right?

General observation - source mapping

I realized that in any case, we are going to need to have some kind of source maps that map from Wasp source code to generated code, and I guess vice versa also, right? So that is a separate task really, independent both of LSP and of JS/TS IDE support → it is a task for Generator to produce the source mapping files. Which is also going to be very useful for improving error messages later. So we could have this as a completely separate feature, feature #3?

[Martinsos]

We talked about this more on the side -> conclusion was that (2) + (5) is probably the best immediate step forward because it is relatively simple to do and already gives us a lot of value.