Wing libraries

[staycoolcall911]

The following section should be added to the language reference:

5.1.2 Internal Libraries

Wing libraries themselves are JSII modules. They can be used in all other JSII
supported languages.

[github-actions]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days.
Feel free to re-open this issue when there's an update or relevant information to be added.
Thanks!

[github-actions]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days.
Feel free to re-open this issue when there's an update or relevant information to be added.
Thanks!

[github-actions]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days.
Feel free to re-open this issue when there's an update or relevant information to be added.
Thanks!

[Chriscbr]

Initial thoughts...

At a high level, supporting Wing libraries can be broken into two parts: packaging and publishing.

Packaging means turning a Wing module (collection of classes, interfaces, structs, etc.) into a shareable artifact. We have several options:

1. turn a Wing module (a .w file that isn't an entrypoint file) into a JSII library¹, as described in this issue's title
2. design some kind of new format?
3. skip packaging altogether (e.g., shipping un-compiled .w files)

The downsides of approaches (2) and (3) are that they also don't move us towards the goal of making Wing libraries usable in other JSII languages. (3) would also likely result in worse compilation and language server performance for users. (1) is my current recommendation.

Publishing means actually sharing the Wing library artifact in a way that others can consume it. Some options:

1. let users publish Wing/JSII libraries on npm themselves
2. host all Wing libraries (including .w and .js source code) inside a central repository like winglang/libraries, and handle publishing for users
3. host a list of all Wing libraries (pointers to other repositories) inside a central repository like winglang/libraries, and handle publishing for users

Some previous discussion about hosting libraries in a central place can be found here. In options (2) and (3), we could publish libraries to npm (under a common namespace, like @winglibs OR we could publish all libraries under a GitHub packages scope The appeal of (3) is it's similar to how homebrew operates, and has been shown to work up to a pretty reasonable scale - each formula is represented by a single file (example).

It might be worth to support support a combination of options, like (1 and 2) or (1 and 3). (1) is the easiest support since it doesn't require any extra work to implement assuming our packaging strategy involves producing an npm tarball. (2) could add some friction to library authors (for example, Wing maintainers may need to approve pull requests and library authors may be unable to run E2E tests in our library monorepo) - but it might be good enough to start with and then switch to (3) once there are more growing pains.

Footnotes

1. A jsii library is just a tarball that can be published to npm, that in addition to package.json and JavaScript code, also has a .jsii file that describes all of the exported types in a language-agnostic way. JSII libraries can be used in JavaScript/TypeScript, but they can also be converted into Python/Java/C#/Go libraries. ↩

[Chriscbr]

From @eladb on #3938:

A quick summary of our conversation about modules, libraries, directories and everything in between:

Basically, a module is any directory that has a bunch of .w files in it (excluding the .main.w files). All the declarations marked pub within this directory are exported from this module. If two (sibling) files in the same module declare the same symbol, it's a compilation error.

When we bring "./my_module" as my_module, then the my_module namespace will include the collection of all the public declarations from all the files in the module, as well as submodules (as sub-namespaces).

If a module doesn't have any pub declarations, it is considered private (same as directories in git).

A library is a module packaged into an npm package.

All the pub declarations within the .w files at the root directory of the library (next to package.json) will be at the root of the library, as well as any submodules.

Until we have pub, everything is assumed to be public.

[Chriscbr]

Aligning on the design

First, I wanted to asynchronously try to align on the motivation for this design for my own understanding, and to make the design motivations easier to understand by others for docs and so on. For me, sharing these thoughts in a GitHub comment seemed like a good enough place to make it available to everyone (and to give the opportunity to be corrected, and for others to ask questions).

Before digging into details, I want to call out that deciding how to "modules" and "libraries" are organized and authored in Wing is subjective, to an extent. I say that because users may wish to express the same kinds of libraries with the same end-user semantics (i.e. with public and private API elements, grouped into hierarchies). The main differences and tradeoffs are about the authorship experience -- like how many lines of code the user will have to write, how they're able to organize files, and so on.

Some definitions I'll be using:

• A module is a collection of API elements.
• An API element is a class, interface, struct, enum, constant, or other module (sometimes called a submodule).

So far, the existing compiler implementation treats each file as its own module. When you write a bring statement with the current syntax, a module becomes available through the foo identifier, containing all of the public API elements from foo.w:

bring "./foo.w" as foo;

If two Wing files have classes with the same names, there isn't any issue since each module is independent, so there can't be name collisions. (One consequence of this is that today, it's not possible for two Wing files to bring each other in a cycle - @eladb has pointed out in the past that cyclic dependencies can often be a source of bugs).

There isn't a way today to define submodules (e.g. I want foo to contain another module named bar that is accessed via foo.bar), or to spread the definitions of a module across multiple files (e.g. I want a module named cloud to contain the API elements from bucket.w and function.w). Some of the solutions we're choosing to pass on for now:

• pub bring ... statements - this would import another module, and make its API elements available as a submodule
• Index files - a file like foo/index.w would declare a list of other files, and then bring "./foo" as foo; would allow users to expose that folders' contents as an aggregate module.
• Module declarations - at the top of every file, a statement like module foo; or module cloud/helpers; would declare that this file's contents would be part of the given module when the Wing project is packaged as a library.
• Module statements - these would be blocks that would explicitly define a module name, and either definitions of API elements or potentially references to other modules' API elements (see appendix for an example).

In Elad's current design, each module is simply defined by a directory - so any files within the same directory will be part of the same module. Modules could be used with a similar syntax that we have today (though the syntax isn't critical):

bring "./foo" as foo; // expects `foo` to be a directory

My understanding of the benefits of this approach:

• No need for extra index files or module declarations. In many TypeScript projects (and most CDK projects), extra index.ts files were just a necessary evil required even if your files were organized in a simple structure.
• The structure of a module has a 1:1 mapping to its file system structure. If you want to move the API elements in one file to another module, it just requires moving the file. In the TypeScript world, there was always a need to update "index" files whenever moving classes/types between modules.
• Modules can be spread across multiple files. If there were no way to group multiple files into modules, then it means all code for a module must be part of a single module. This is reasonable tradeoff for some languages, but it disallows some kinds of code organization, like file-per-class organization (common in React and CDK).
• No need to re-export APIs. There can be use cases where it's helpful or necessary to re-export types from other libraries, but it's a rare enough occurrence that we're considering it out of scope for the Wing libraries MVP.

My understanding of the potential drawbacks of this approach:

• Modules no longer have a central / root file. If you want to identify where a class was defined in a module, you need to search all of the files in that module, or use an automated tool like the LSP to help you.
• More complex and potentially expensive compilation model. The compiler will need to scan your project for all .w files in order to generate your app/library's module hierarchy, and analyze dependencies not just between individual files but also dependencies between modules.
• Potential confusion when you use the same name for API elements in different Wing files. (good error messaging can help with this!)
• Less flexibility to organizing modules in "artistic" ways that don't follow the file directory structure. (would we really want that?)

Unresolved questions

How are API elements made available between files in a module?

Suppose a module contains two sibling files, bar.w and foo.w. Within a module, all API elements (even private ones) should be accessible. How would bar.w access classes from foo.w?

1. Are the types automatically / implicitly available through the global scope?
2. Does the user need bring statements? (see next question)

How do bring statements work when individual files do not represent modules?

Going back to the previous example - suppose a module contains two sibling files, bar.w and foo.w. Suppose the user writes the following statement in bar.w, assuming it's valid:

bring "./foo.w" as foo;

Previously, foo was a module. In a world where folders represent modules, what is foo? It cannot be a module since we've stated individual files are no longer modules in of themselves. Is it a module subset, or something like that? Would it make sense to only allow importing API elements directly from foo.w, e.g. bring "./foo.w" { Foo };?

Appendix

Module statement syntax

pub module Helpers {
  pub class Assertions {
    // ...
  }
  
  pub bring "./helper.w"; // re-export?
}

[Chriscbr]

Aside - how might the compiler implementation work?

Today the compiler starts by parsing all files in the Wing project. It creates a stack of files to parse, and then does the following:

1. Pop a file from the stack (skip if it was parsed already).
2. Parse the file, collecting a list of other Wing source files it references.
3. Add those files to the stack.
4. Go back to step 1.

A graph of files is built during this process -- if files "bring" each other in a cycle, this is caught during parsing.

Then, it type checks all of the files in a topological order, using the fact that each file is an independent module. Each file's symbol environment is stored in a lookup table, and when a "bring" statement is encountered, the lookup table is used to load types from a previously type checked file.

---

In the new module system, the compiler needs to support modules being spread across multiple files and directories (that might never reference each other!) Since a file like foo/bar.w might contain public API elements but might not be referenced anywhere else in the user's project (but still needs to be compiled as part of a library), it follows that the compiler needs to just find all Wing files. So parsing is now:

1. Scan the current directory and all subdirectories (ignoring directories like node_modules) for .w files.
2. Parse each file.

Then, files need to be type checked. The order type checking happens matters because it's possible a project has two sibling modules, foo/ and bar/, and one of them may depend on the other. The compiler will have to use the information from the parsed Wing files to form a topological sort of the Wing modules, and the files within each module. Then, we compile each module in topological order (or perhaps just in an inverse-depth order), and files within each module are type checked in topological order.

After compiling each module, the compiler could combine the symbol environments together, and check that no API elements with the same name are created within the same module (regardless of whether they are public or not).

From the LSP side, updating a file will cause that one file to be re-parsed, and for now the entire project will be re-type checked, though we can improve on this in the future.

[Chriscbr]

(sharing more thoughts out loud) There's one more unanswered question on my mind, which is around how we should model compilation entrypoints.

When a user opens a Wing file while using the language server today, since each file is its own module, the compiler treats that as the entrypoint for generating error diagnostics. When you open more files (perhaps they depend on the original file, or the new files depend on the first file you opened), it incrementally builds this compilation graph of the files relevant at the moment.

But with the directory-as-a-module design, we instead need to treat the file's parent directory as an entrypoint. This gets a little tricky if the file you open isn't at the root of your project -- e.g. it could be in a subfolder. We would need to pretend that file's directory is the root and compile it like that, and then change it if the compiler suddenly

All of these rules for how to incrementally compile different directories of files based on what you've opened seem a bit complex and error-prone to me. For example, it opens up the possibility for bugs that only appear if you open files in a certain order. I'm starting to think it might be simpler if we define some convention for identifying the "root" of a Wing project (like package.json or wing.toml or even just the presence of main.w). Then, when you open any Wing file, the LSP just looks find the root of the project, and just compile everything in that project.

[Chriscbr]

Closing since we've shipped #3938

Any leftover tasks should be covered by #1037 and #2171