Only export main module from package

[hendrikvanantwerpen]

#347 »

Packages would export every module as if it was the main module. This PR changes that so that only the main module is exported. If the main module is missing, index is used.

Note that for packages that specify a build artifact as the main module (along the lines of build/dist.js), the package will not export anything anymore! If this turns out to be too strict, we can look at the following alternatives for that:

• Always include index, even if the main module is explicitly specified.

• Fall back to index if the specified main module does not exist. I think we (can) have enough information in the function processing package.json to determine this.

I started documenting the package structure as well (rendered).

Fixes #337.

[BekaValentine]

I'm unclear how this addresses anything, but maybe this is interacting with other parts of the system? This only removes stack graph rules, so how does the main field of package.json give rise to stack graph edges of the appropriate sort???

[hendrikvanantwerpen]

@BekaValentine The package.json analyzer and TSG work together, but I get that it's not easy to understand how from the code. I started documenting the package structure, using an example (rendered).

So far it's only the example! I haven't had time to write text yet. At least you can already see which parts of the graph come from the different files, but I'll write more text in the coming days.

[BekaValentine]

@hendrikvanantwerpen it seems to be the case that there's a special piece of rust code that handles this stuff in a custom way? I think it's npm_package.rs? From my reading of things, it looks like npm_package.rs is called somewhere, and it processes package.json to generate a stack graph fragment which turns lookups for the package into lookups for the package's main file.

Is this correct?

If so, do you have any thoughts on how we might make it possible for external contributors to do something? One thought I have is that we might want instead to not use custom rust code, but rather use a tsg file that processes package.json, and other package config files, and then we can have some cfg file that lets contributors specify which such files to look for and process, and what to use for that. For instance, for JS, you might have

[packages]
config = "package.json"
library = "package.tsg"

or something like that, I dunno. We could also use those kinds of files to describe how packages relate to source directories, if there's a language-specific or packaging-system-specific convention, etc. We might even want to think about how packages and languages might be orthogonal or partially orthogonal issues (since multi-lingual environments beyond just the NPM ecosystem do exist and are quite widespread).

[hendrikvanantwerpen]

it seems to be the case that there's a special piece of rust code that handles this stuff in a custom way? I think it's npm_package.rs? From my reading of things, it looks like npm_package.rs is called somewhere, and it processes package.json to generate a stack graph fragment which turns lookups for the package into lookups for the package's main file.

Is this correct?

Yep! The package.json is handled by an implementation of FileAnalyzer, which is registered here:

stack-graphs/languages/tree-sitter-stack-graphs-javascript/rust/lib.rs

Line 53 in 270bb1b

FileAnalyzers::new().add("package.json".to_string(), NpmPackageAnalyzer {}),

If so, do you have any thoughts on how we might make it possible for external contributors to do something? One thought I have is that we might want instead to not use custom rust code, but rather use a tsg file that processes package.json, and other package config files, and then we can have some cfg file that lets contributors specify which such files to look for and process, and what to use for that.

So external contributors can already contribute that Rust code. This is convenient, but it is perhaps not most desirable, for two reasons: (1) contributors have to suddenly think about the lower level Rust API, instead of the higher-level DSL code, and (2) we have have externally Rust code that will run in our production system.

Having a higher-level and safer way to write these rules would be nice! Previously I was not sure using Tree-sitter and TSG was really the right approach, because it would mean pulling in a new grammar dependency for the config file, and because the TSG would need to match on concrete names (e.g., keys in JSON). But perhaps it won't be as bad, and it might be worth an experiment.

We could also use those kinds of files to describe how packages relate to source directories, if there's a language-specific or packaging-system-specific convention, etc. We might even want to think about how packages and languages might be orthogonal or partially orthogonal issues (since multi-lingual environments beyond just the NPM ecosystem do exist and are quite widespread).

Yes! The package detection stuff is all in Aleph right now, which is not nice for testing this logic. A general solution could be to add something similar to FileAnalyzer but for source trees, which computes the values of global variables for each file in the tree. But having packages/projects/something as actual concepts might also be an option. This would depend on whether there's enough shared between different languages, and whether we can still support the particular bits that don't fit the general concept.

It would be great to have an issue for this, as you suggested to write up, to develop these ideas!