Coordinating with wasm-bingden about versions and list of NPM dependencies

[alexcrichton]

This is related to https://github.com/rust-lang-nursery/rust-wasm/issues/34 and https://github.com/rust-lang-nursery/rust-wasm/issues/36, but I'd like to propose a strawman idea here.

Right now wasm-bindgen has the ability to say where you're importing functionality from:

#[wasm_bindgen(module = "foo")]
extern {
    // ...
}

If module = "..." is omitted it's assumed to come from the ambient environment (aka the browser). Otherwise foo is interpreted as an ES module directive and that's where items are imported from in the generated JS that wasm-bindgen emits.

The wasm-bindgen output is pretty "dumb" right now in that it's just bland ES module information, it doesn't currently attempt to give extra information like verison requirements for NPM packages. But that's where wasm-pack comes in! Ideally wasm-bindgen would emit information that wasm-pack later reads, parses, and emits a package.json for.

So I'd propose something like:

• Add support for version = "..." next to module = "..." in wasm-bindgen
• For all module = "..." directives that don't start with ., require that version is listed
• Emit a new section in the wasm executable in the wasm that wasm-bindgen emits in the format:
  • The section name is __wasm_pack_unstable
  • The section format is a list of elements. Each element is prefixed with 4 bytes of how long it is and then the rest of the data is a JSON payload.
  • The JSON payload for each element is currently { "module": "string", "version": "string" }
• When wasm-pack executes it slurps up the __wasm_pack_unstable section, if present, and emits these packages into the package.json that's generated.

An example of using this would look like:

#[wasm_bindgen(module = "moment", version = "2.0.0")]
extern {
    type Moment;
    fn moment() -> Moment;
    #[wasm_bindgen(method)]
    fn format(this: &Moment) -> String;
}

#[wasm_bindgen]
pub fn run() -> String {
    moment().format()
}

And wasm-pack would automatically generate a package.json with:

{ 
   "dependencies": {
        "moment": "2.0.0"
    }
}

I'm definitely up for changing anything here, this is just an initial strawman! I'd love to hear others' thoughts on this as well.

[Pauan]

Add support for version = "..." next to module = "..." in wasm-bindgen

Shouldn't that be the responsibility of package.json though?

[alexcrichton]

@Pauan indeed! Although wasm-pack is the one generating package.json, right? So it just needs to know what to fill in?

[Pauan]

@alexcrichton Please correct me if I'm wrong, but my understanding is that the plan is that Rust crates will include a package.json file at the same place where they contain Cargo.toml, and wasm-pack will then merge all of those package.jsons together to create the final package.json.

So the version information would be in the crate's package.json file, not inside of the crate's Rust code. That's what I meant by "isn't that the responsibility of package.json"?

[alexcrichton]

Oh I was thinking that intermediate Rust crates would not ship package.json, and that would explain the confusion!

Currently wasm-bindgen knows very little about Rust crates and dependencies, it purely takes one wasm file as input. It could be the case that #[wasm_bindgen] slurps up a a crate's package.json and inserts it into a custom section to later be read by wasm-bindgen, but AFAIK wasm-pack has no ability to walk the dependency graph and locate instances of package.json

I'm also not certain we'd want to ship package.json with Cargo.toml because that would imply that the crate is a standalone package for NPM which likely isn't?

[Pauan]

Hmmm... npm dependencies are rather tricky: you have dependencies, devDependencies, optionalDependencies, peerDependencies, and bundledDependencies.

And a package.json can only contain a single version per package.

And there's a bunch of other package.json metadata which is useful as well: such as the minimum Node/npm version, what platforms it supports, the CPU architecture, etc.

So I feel like having a package.json in each crate is the best way to go.

But I suppose an attribute is an okayish substitute.

I am very concerned about version conflicts, though: package.json can only contain one version per dependency, so each crate will need to carefully specify the exact same version string for the packages. (Or wasm-pack would need to use a version parser + solver to find a common version bound)

[ashleygwilliams]

@Pauan the goal was for wasm-pack to have a version parser and solver :)

@alexcrichton need to read the backscroll a little more closely but i think in general i agree with ur approach

[ashleygwilliams]

ok so, addressing a few of @Pauan's concerns

And a package.json can only contain a single version per package.

a package.json can contain a single semver expression per package.

And there's a bunch of other package.json metadata which is useful as well: such as the minimum Node/npm version, what platforms it supports, the CPU architecture, etc.

this is vaguely true but none of it is enforced by the npm client. this info could be kept anywhere (most people use a .travis.yml)

---

i'm quite strongly against requiring a package.json in a Rust crate. it would both not be functional and not aligned with the goals of the project that i clearly state in my blog post on this tool. i'm quite certain that we can avoid requiring it, a single manifest file per project is plenty

[alexcrichton]

@ashleygwilliams do you think that wasm-bindgen should do something like print out by default "please add these dependencies to your package.json"? I'd imagine that wasm-pack could pass a flag saying "don't print out that message" and otherwise it may be useful for those running wasm-bindgen manually or something like that.

[alexcrichton]

Also it's possible for wasm-bindgen to do the version resolution here maybe? It could print out itself that there are conflicting versions or otherwise union the version requirements together. That feels more wasm-pack-like though

[ashleygwilliams]

yeah i think leaving resolution of deps to wasm-pack makes the most sense here, tho if we really wanted we could do it in a crate that both tools could leverage if we wanted to

[alexcrichton]

I've opened a PR for this at wasm-bindgen at rustwasm/wasm-bindgen#161

[Pauan]

@ashleygwilliams the goal was for wasm-pack to have a version parser and solver :)

Okay, that sounds fine.

Though I wonder if wasm-pack actually needs a version solver. Given these Rust crates...

#[wasm_bindgen(module = "moment", version = "^2.0.0")]

#[wasm_bindgen(module = "moment", version = "^2.4.0")]

...couldn't wasm-pack combine all the versions together, like this:

{
  "dependencies": {
    "moment": "^2.0.0 ^2.4.0"
  }
}

That would leave the version solving up to npm/yarn/whatever.

The biggest downside I can see is that it would result in inferior error messages in the case of conflicts. But it might be good as a temporary solution.

a package.json can contain a single semver expression per package.

Indeed, that's what I meant.

i'm quite strongly against requiring a package.json in a Rust crate. it would both not be functional and not aligned with the goals of the project that i clearly state in my blog post on this tool. i'm quite certain that we can avoid requiring it, a single manifest file per project is plenty

I don't have a strong preference either way (between package.json and an attribute), though I am curious why you are strongly against package.json

[Pauan]

One of the downsides of the attribute is that it makes it very unclear what dependencies the package has.

For Rust crates you can simply look at Cargo.toml and instantly know all of the dependencies. The same is true with package.json. With the wasm_bindgen attribute you have to actually look through all of the source code to find the dependencies.

There is another possibility: specify the npm dependencies in Cargo.toml, so that way it doesn't require a separate file, and it's more intuitive for Rust programmers.

[ashleygwilliams]

@Pauan i'm confused...

{
  "dependencies": {
    "moment": "^2.0.0 ^2.4.0"
  }
}

is not a valid package.json so that's def not something we will do

[Pauan]

@ashleygwilliams What do you mean? npm's semver allows a space between versions, which it treats as logical and (i.e. set intersection).

[ashleygwilliams]

huh @Pauan i guess that is technically true! tho it's a dangerous move because i don't actually know what npm does if the intersection represents a set with no members... i think fundamentally this is something we'd probably want to sort at the wasm-pack level, mostly because i don't want to allow the publish of a package that is not installable

[edit, i was curious so i checked] npm does this:

Ashleys-MacBook-Pro-2:test ag_dubs$ vi package.json
Ashleys-MacBook-Pro-2:test ag_dubs$ npm i
npm ERR! code ETARGET
npm ERR! notarget No matching version found for lodash@^1.0.0 ^2.0.0
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'test'
npm ERR! notarget

which is not super useful. in all my years working at npm i have to be honest i never saw ANY package.json declare deps like that, so for that reason alone and my desire to not surprise people, i would opt to not do that

[Pauan]

in all my years working at npm i have to be honest i never saw ANY package.json declare deps like that, so for that reason alone and my desire to not surprise people, i would opt to not do that

Fair enough.

What about my idea of specifying the version information in Cargo.toml?

[sendilkumarn]

@alexcrichton I too agree to most part of the initial idea.

What about my idea of specifying the version information in Cargo.toml?

I think this from @Pauan makes sense too. Why cannot we port this to be in cargo.toml file. Add in a npm or whatever section inside the cargo.toml and use this to generate.

from wasm-pack 's perspective it is just a single place to look into. From user's perspective not much of a difference between defining versions and we need not beef up the rust code that users will end up in writing.

WDYT?

[alexcrichton]

Pedagogically I don't have too much of an opinion either way on where this info lives. Technically I'd prefer the attribute because it means I don't have to pull a TOML parser into wasm-bindgen.

[sendilkumarn]

Ergonomically 😜

[ashleygwilliams]

@alexcrichton let's stick with what we have for now... if it turns out that people want something else, we can build it later, based on people's actual experience, and not just guessing ;)

[Pauan]

@ashleygwilliams let's stick with what we have for now... if it turns out that people want something else, we can build it later, based on people's actual experience, and not just guessing ;)

I strongly disagree with this. A lot of people are already using wasm-bindgen and wasm-pack, and so it will quickly become a de-facto standard, which will be very hard to change later.

If we're going to experiment with different things, we should do so now. As I said, there are benefits to using Cargo.toml, so we shouldn't just dismiss it as a possibility.

[mgattozzi]

Modifying a config file for a different tool other than what that file is for does not sit well with me. Cargo.toml is used to modify how cargo works, not wasm-pack/wasm-bindgen. If you want to do build config things I highly recommend it either be in it's own config file or build.rs as is current convention for Rust projects.

[Pauan]

@mgattozzi Cargo.toml is used to modify how cargo works, not wasm-pack/wasm-bindgen.

That is incorrect: Cargo.toml has a metadata section specifically designed for non-Cargo tools which wish to put metadata into Cargo.toml

If you want to do build config things I highly recommend it either be in it's own config file or build.rs as is current convention for Rust projects.

Using a separate config file does work, though that idea was rejected.

build.rs was never intended for specifying declarative dependencies, it was intended for running Rust code to compile native C libraries (and other similar things). How do you propose to use build.rs for specifying npm dependencies?

[ashleygwilliams]

this issue has become quite contentious! and i really appreciate everyone's efforts here. because this seems like a more complex issue, @alexcrichton and i are going to work on an RFC tomorrow and hopefully post it on friday on the rust-wasm repo. the goal will be that the RFC dives deeply into the tradeoffs of this decision and gives us more structure to have this very important convo! feel free to keep chatting here and we can implement some of the thoughts that have already been shared, but this will likely be closed when we post the RFC :)

thanks again ya'll!

[mgattozzi]

@Pauan Oh I didn't even know about the metadata field! Thanks for letting me know about that :D I think at this point we'll just need to wait for the RFC and we can continue discussion there :)

[Pauan]

[...] going to work on an RFC tomorrow and hopefully post it on friday on the rust-wasm repo [...]

That sounds great! As a general principle, I think we should have RFCs for these sorts of issues that have long-term consequences for the entire community and ecosystem.

In addition to discussing the system for specifying dependencies, maybe the RFC should also discuss about whether the versions should be semver-by-default or not?

[ashleygwilliams]

closing because stale and waiting on an rfc process.