Figure out Wasp's node_modules story

[sodic]

Background

Wasp users write User code.

Wasp generates Generated Code. We want to split the Generated Code into:

1. SDK (library) Code - This covers all the functionalities Wasp users import and use in User code.
2. Framework Code - This covers the "shell" code. It's the code that imports and executes User code, indirectly importing and using SDK Code.

Here's how the new structure looks in the file system:

.
├── main.wasp
├── node_modules
│   ├── <other_modules>
│   └── <sdk>
├── .wasp/
│   └── <framework_code>
├── <user_code>
├── package_json
├── .eslintrc.json
└── tsconfig.json

Questions

The plan is to ensure that Wasp's SDK inside the node_modules folder includes everything Wasp users might want to import (e.g., useQuery, entities, etc.).

We must decide on how to use, organize, package, and build the SDK.

1 Organization

There are several possible ways to organize the SDK Code.

Having only a single package.json file might make more semantic sense. After all, it is a single package (Wasp SDK) and all the code should probably follow the same rules.

Having one package.json file per module might make less semantic sense, but also makes things more flexible for the future (and maybe even now). For example, having different packages for the client and the server would split the constraints for packaging and building the SDK (assuming we want to keep the same compiler output, a server project and a client project).

note
All the package.json talk depends on the decision we make in the Section 3.1: Packaging.

With that in mind, here are the options:

A single package

Requires a single package.json file for N modules.
Packages: node_modules/wasp.
Imported as: wasp/auth/useAuth, wasp/queries/getSomething

Warning
The NPM registry already has a package named wasp. Assigning the same name to a folder inside node_modules is against the rules and might cause problems.

Multiple packages

Requires N package.json files for N modules.
Packages: node_modules/wasp-queries, node_modules/wasp-auth, ...
Imported as: wasp-rpc/useAuth, wasp-auth/getSomething

Multiple scoped packages:

Requires N package.json files for N modules.
Packages: node_modules/@wasp/auth, node_modules/@wasp/queries, ...
Imported as: @wasp/auth/useAuth, @wasp/queries/getSomething

Warning
Scopes are not meant to be used for this purpose. We might want to create a real @wasp NPM organization at one point (e.g., for other wasp-related packages that aren't a part of the compiler). In that case, we would have a problem.

Organization: Decision

Still under consideration, but we are currently going with scoped packages. See #1584 (comment)

2 Usage

The IDE always tells Wasp users that their code imports and uses the SDK code from the node_modules directory. However, we must choose what happens during runtime. I see two options:

1. The SDK is used a proper NPM package - Users import and use it, the Framework Code calls the User Code. The User Code calls the SDK (exactly as the IDE claims).
2. The SDK is only used for IDE support - Users think they're importing and using the code from the SDK. However, during runtime, their code imports and uses something else (perhaps the very same SDK code, but duplicated somewhere inside the framework code).

Option 1. intuitively makes much more sense and is the one we'd prefer to go with. However, Option 2. might be easier to get working (in case we encounter more unanticipated problems).

Usage: Decision

Option 1 was possible and simple enough to get working, so we went with option 1 (i.e., SDKs are proper NPM packages).

3 Building and Packaging

This section talks about the SDK's creation process.

There are several options, we are not yet sure all of them can work.

3.1 Packaging

Packaging refers to how we "present" the SDK, we can:

1. Package it for real (i.e., create a package.json file). This includes picking a module system. Some module systems require external process calls from Haskell (e.g., CommonJS), some don't (e.g., ES Modules). This also depends on whether we're packaging SDK for the client, the server, or both (see Section 1: Organization)..
2. Copy paste the code into node_modules and rely on the IDE to figure it out. This can only work if we go with Option 2 in Section 2: Usage.

Packaging: Decision

The prototype proved that we can package our SDK code as ES Module (i.e., type: module in package.json) packages without issues (Vite handles it properly, and our server code uses ES Modules).

What we know:

• A Node runtime combined with ES Modules (i.e., import/export syntax) requires a package.json file for specifying type: module. [TESTED]
• A Vite runtime combined with ES Modules does not require a package.json file. It just works. [TESTED]
• When NPM resolves a package, it will climb up the directory tree looking for node_module folders, using the first (nearest) package definition it finds. This means that the code inside project_root/.wasp will automatically look for packages inside /project_root/node_modules [TESTED]

3.2 Building

Building refers to the steps Wasp executes to produce the SDK after detecting user changes that require updating it:

1. Build our code using TSC and emit JS code (*.js) and type declaration files (*.d.ts) on each user change. This will definitely work but will probably be slow.
2. Write the TypeScript code to the SDK dir in node_modules as-is, without a special build step and leave the compilation to the user (not sure if this could work).
3. Prebuild our SDK code with TSC and write the appropriate JS and type declarations on each user change. This is the sanest approach, but it might not be possible for every module, as some of them greatly depend on user code (e.g., query and action stuff).

Building: Decision

We must build the TypeScript code, as some users will want to use JavaScript. Still, we can get away without building anything for the time being, as Wasp currently always requires TypeScript (regardless of what the user uses).

[infomiho]

One thing to consider are local paths for npm modules: https://docs.npmjs.com/cli/v9/configuring-npm/package-json#local-paths

It enables you to specify the SDK packages in the project's package.json and put them in an sdk folder.

{
  "name": "my-project",
  "devDependencies": {
    "@wasp/auth": "file:./sdk/@wasp/auth"
  },
}

[sodic]

Relevant comment: #1584 (comment).

We are going with a single package for now. Reasons can be found in the linked discussion.

We'll be installing our sdk package using a local package installation from somewhere inside the .wasp directory and using NPM's local dependency option for installing it.

[sodic]

The SDK was implemented in #1626 and later revised in #1696.

Other issues deal with the remaining questions. For example:

• Optimize SDK build step #1840
• Split dependencies in the new structure #1595
• Investigate how our file watching works #1844
• Refactor our generated codebase #1841