Rethinking Wasp code organization

[Martinsos]

Related to #710 .

So far we have made decisions in Wasp like:

1. Code generated by Wasp is put in .wasp dir.
2. Code generated by Wasp is imported in JS files as "@wasp/...".
3. Generated code is structured in a certain manner (queries are under queries/, actions under actions/, ...).
4. User code is organized under ext/ directory and then just copied into the generated project.
5. And many more decisions!

We are now trying to add IDE support for user code in ext/ (specifically for JS/TS code) and one big thing that makes this somewhat tricky to do is the way our code is organized -> generated project is organized differently than what we have in the source dir, specifically in ext/, so we have to "cheat" editor/IDE into thinking stuff in ext/ is part of the project that is not really there (because IDEs, or their plugins for JS/TS, expect some standard JS project setup, like node_modules in specific place, maybe package.json to be there, tsconfig.json, ... -> and we have none of those in source project in ext/ dir, we only have it once you go to generated project).

So the idea is -> what if we rethink stuff and reorganize Wasp in such way that we maybe avoid some of these issues? Maybe make source code more similar in setup to generated project? Or at least those parts that are needed for IDE support for JS/TS?

So let's rethink stuff @shayneczyzewski @sodic @matijaSos ! Let's let our minds wander and come up with crazy ideas. Doesn't matter if they are unbaked or a bit impossible, let's just try to come up with some new directions and see where they take us.
I suggest taking an hour or two specifically for this and just trying to come up with smth.

While thinking about ideas, here are some requirements to keep in mind:

1. In the future we will want to have Wasp modules -> standalone, reusable packages of .wasp files + other JS/html/... files. Likely we will also want to mix .wasp files with other files, so they can be next to each other.
2. We might want to support some other languages next to JS/TS, for example Python, or Go:
   2.1. As languages used on backend, instead of JS/TS backend or next to JS/TS backend.
   2.2. As languages to write jobs in.
3. We want to support deploying our web apps as different architecture and on different providers.
4. We want to support multiple IDEs (even though we are now focusing on vscode).
5. Support stuff like prettier, linter, ... .
6. We want to allow organizing project by feature.

Some rough ideas that were mentioned so far:

1. What if we generated projects directly in the source project, next to files in ext/. Maybe organized as neatly as possible, so they are not too much in the way, but still they would be there.
2. What if users would not write their code in ext/ as they do now, as just JS / HTML / ... files, but instead they organized their code as complete npm packages that import Wasp code as yet another package and generated Wasp code also imports them as npm package? That way their code is an encapsulated, standalone thing that IDE knows how to deal with, and we can easily use it in the generated code.
3. Wasp could be generating more or less of the code as libraries. Some of it could be even prepared before hand as a library, if it does not depend on the source code. This is a bit orthogonal idea though, not so tightly related with what we are doing here hm but could be useful.

[Martinsos]

We did initial round of brainstorming!

Idea (1) from above:

• we concluded we might go as far as generating wasp code in user's source code, and even maybe have them commit this to their version control. So instead of hiding it in .wasp, making it more visible. This would give them more insight into what Wasp does in the background, although we are not sure how important is that. We would probably still not let them modify it, although we could also play with that in theory. If we are doing this, we could go as far as to have them write their JS/HTML/... code in the directory that is inside the generated project, which would likely help with IDE support for those.

Idea (2) from above:

• so all user code would be organized in npm packages. Wasp code used in their code would also be organized as npm packages (generated that way), while the rest of the generated code can be organized in package(s) or doesn't have to be.
  This would solve all the problems with IDE, and allow for many things out of the box, like defining tests, modifying build, adding dependencies, ... . It makes it all very standard, so a lot of stuff just works out of the box.
  Possible problem is though having user code use wasp code and vice versa, since that circularity could be an issue now that they are npm packages. Maybe we can introduce some restrictions that would enable this -> only source files in user package that aren't used in generated wasp package can import wasp package. Or maybe we can use some tricks / stubs / ... . We have yet to figure out what exactly could be the problem with circularity though, before we focus on solutions.
  There are also some extra files now introduced -> users have to deal with package.json, maybe some other files. We can scaffold these for them though. Or we can even control them to some extent -> for example Wasp says they can edit them, but some parts not, and it even parses and checks those parts, updates them, ... . We could play with different levels of control here.
  This change sounds like it could really simplify some stuff. But we might also loose a bit of control, maybe. Hard to say yet. And circularity hangs over this as apotential show-stopper.

@matijaSos mentioned that NextJS threw some errors on him that were coming from their generated code that was generated based on his code, and error messages was pretty mangled -> that sounds like they are doing stuff similar to us a bit? Maybe we should also check what they do there?

[Martinsos]

Some of the things this re-organization might fix:

1. Non-customizable build pipeline Allow our users to configure the compilation pipeline #866
2. Allow custom npm registries Wasp: allow custom npm registries #869
3. No way to install dev dependencies Consider supporting user dev dependencies #456
4. More freedom with choosing dep versions Consider allowing users to specify some NPM dependency versions themselves #645
5. No package-lock.json support Figure out Wasp's package.lock.json story #559
6. Hacky/partial IDE support Support custom TypeScript configuration #835 (+ avoid having files in source that they can't touch)
7. No support for tests Add support for tests #110

[Martinsos]

Related to #816

[Martinsos]

Current approach here: all code will be organized as one big npm package, with package.json file exposed. Both client and server code will be under this. @sodic is currently actively working on this.

[uniphonic]

Current approach here: all code will be organized as one big npm package, with package.json file exposed. Both client and server code will be under this. @sodic is currently actively working on this.

Having the package.json file exposed sounds great! When is this planned to happen? Also, what would upgrading an existing Wasp project entail? Thanks!

[Martinsos]

Current approach here: all code will be organized as one big npm package, with package.json file exposed. Both client and server code will be under this. @sodic is currently actively working on this.

Having the package.json file exposed sounds great! When is this planned to happen? Also, what would upgrading an existing Wasp project entail? Thanks!

We are planning to have this out very soon, in a couple of weeks! Most of your JS code won't need to change much, but updating will require moving some stuff around, possibly doing some db migrations (because we are also going to release improved Auth), ... . We will make sure to provide instructions / scripts on how to do the project update most easily.

[Martinsos]

whoop whoop