Project very similar to web-bundler

[cecton]

Hey 👋

I just found out your project today and I noticed it is very very close to what wasm-run does.

I created wasm-run because I needed a finer control on the index.html that is created so I can bundle the JS and the CSS together inside the index.html. I think web-bundler allows this which is great!

There aren't many differences with wasm-run:

• The main difference is that web-bundler uses the build.rs to generate the frontend files while wasm-run uses a cargo [bin] that the user needs to put in their crate. Example here where lib.rs is meant to be the frontend compiled in wasm and main.rs is meant to be the bin that starts a standalone frontend.
• wasm-run supports frontend-only applications and provides a small web server (tide) for that. If I'm not mistaken, I think web-bundler only supports backend+frontend scenario.
• web-bundler uses wasm-pack which benefits from all the logic of wasm-pack. wasm-run opted for wasmbindgen-cli in hope to reduce the amount of dependencies. Because of that some features are still missing in wasm-run like the JS snippets generation. Ideally I wanted to go deeper and depends only on wasmbindgen (there isn't much to do to get the snippets working).

I'm not into re-inventing the wheel and I'm all for community effort and working on common goals so I was wondering if you think we could combine our projects? Take the best ideas of both worlds and cover all the use cases. I'm not sure yet how that will translate but let me know your opinions.

(There is also the project substrate-wasm-builder that also uses the build.rs to build the WASM but it's more focused on the WASM itself.)

Best

cc @fstephany @yozhgoor

[JWorthe]

Hi @cecton! 👋

I'm keen to combine forces. It looks like wasm-run is a much more mature project, and has a bunch of features that I'd like to have in web-bundler.

We are heavily invested in the backend + frontend use case, and specifically embedding the frontend files into the backend
binary. It's been working well for us that we can run cargo build --release in our continuous delivey pipelines, and end up with one large binary to deploy with all of the frontend stuff embedded. We have enterprise users who self-host, so the simplicity of being able to give them a single binary to run is really a killer feature.

I think this is where our use cases are subtly different. We want to view the frontend and backend together as one deployable artifact. It looks to me like wasm-run makes it easy to build backend + frontend as well, but conceptually the frontend and backend are two separately deployable artifacts. This difference might make it difficult to combine all use cases naively into one project. What do you think?

How would you propose combining effort? I think there might be a good path where we have a shared crate that both web-bundler and wasm-run use, so we can expose different ways of doing things but share common logic around building and optimizing wasm files and html.

These are some of my wishlist features that it seems wasm-run already has:

• Break the dependency on wasm-pack. It was a good way to get started quickly, but as you say that means we bring in a lot of dependencies. wasm-pack also acts much more like a CLI application than a library, for example by downloading and installing other CLI applications like wasm-bindgen-cli and wasm-opt. In larger builds, we've run into trouble where multiple wasm-packs running in parallel sometimes fight over the cache directory. There's a fix for this upstream, but wasm-pack hasn't had a release in a long time.
• I'd love something like your cargo run -- serve for faster development without constantly rebuilding the backend. I'm not sure exactly how I'd make this work smoothly, but it probably involves a dev web server that proxies a running backend.
• I want to get a list of all of my frontend's Rust dependencies, so I can tell Cargo to rerun-if-changed.

Best,
Justin

[cecton]

I totally agree with you on all the points.

It is true that I designed wasm-run to have backend and frontend builds separated. I explain this better on the discussions in there: IMI-eRnD-Be/wasm-run#48 (comment) feel free to join! (Or we will join here, it might be easier.)

Maybe there is a way to make bundling through build.rs and separate (run) works. That would actually merge the 2 projects. If not, we can do as you said with a common crate that has the base logic.

[cecton]

I want to get a list of all of my frontend's Rust dependencies, so I can tell Cargo to rerun-if-changed.

This is totally doable for me. I already know how to achieve this using cargo-metadata.

I'd love something like your cargo run -- serve for faster development without constantly rebuilding the backend. I'm not sure exactly how I'd make this work smoothly, but it probably involves a dev web server that proxies a running backend.

After discussing about it with @fstephany (we're colleague) and reading your wish list, I realized that running the backend+frontend use case using the build.rs might actually be simpler (for that use case).

But I think you also made an important point: what if someone wants to work on the frontend alone. It would be faster if only the frontend is rebuilt. This is actually current the default behavior in wasm-run: the serve command can run the backend in a subprocess and only the frontend is rebuilt. There is also a feature called full-restart which also rebuilds the backend separately. The backend is ran using an entrypoint as you can see here.

Maybe a good compromise would be to drop the full-restart feature of wasm-run in favor of the build.rs like web-bundler does. Then change the run_server argument of wasm-run to use the name of a crate instead of an entrypoint, example:

#[wasm-run(backend="backend-crate-name")]

In this example, the command cargo run -p backend-crate-name would be used to start the backend in background.

In the end I think our projects are mergeable in a single project. 🎉

If we all agree to merge, there are a few questions:

• The GitHub organization: our client is the GitHub organization of wasm-run (https://github.com/IMI-eRnD-Be). We should probably go for a neutral organization.
• Name of the project (I don't have strong feelings for wasm-run).

[fstephany]

Hello here! 👋

It's good to see that bundling frontends is a thing 👍

I have another (!) use case for a bundler: the more general web development flow with a backend, frontend and sass. There's a sample project that illustrate what I'm trying to do.

I don't know if i'm totally out of scope here but I would like to be able to start my development flow in one command that could look like:

cargo run --bin run  -- serve
- (backend) Watching '/crates/backend/'
- (frontend) Watching '/crates/frontend/'
- (stylesheets) Watching '/assets/css' 
- Building backend
- Building frontend
- Building Sass
- Starting backend with $ cargo run --bin backend
....
* (backend) files have changed.
- Building backend
- Starting backend with $ cargo run --bin backend
.....
* (stylesheets) files have changed
- Building SASS from '/assets/css/app.scss'
....

The tool would watch each of the component (backend/frontend/sass) and rebuild the relevant part if something changes. When comes the time to build for a release, the necessary hooks would be exposed to customize the build process (eg., optimize the wasm, embed CSS into the html).

That would broaden the scope of web-bundler or wasm-run of course as this would not specifically focus on the frontend. I guess I see it more as a Rust Webpack than what it currently is 😅

Am I completely out of touch or does it make sense?

[cecton]

Ohhhh I just got it now. That's the thing with the 3 components you were talking about!! You just want the watch and retrigger the build separately depending on what has been touched?

Ok I think it's feasible.

[fstephany]

Yep!

That's why I think that it can be useful to have "entrypoints" and each of those entrypoint would have its own lifecycle (watch, pre-build, build, post-build).

That opens the door to multiple small-ish wasm bundle to sparkle a regular page rendered by the backend. For example, in my Rails/React days, I would typically have a rails app and only have React Components in some pages where the user interaction would be too complex for a simple HTML form. I would then have a mini react app (or sometimes a small sprinkle of JS) just for that page.

I could imagine the same with wasm bundles included in the relevant pages. I know that is not an ideal strategy right now because I would have separate wasm bundles loaded with a lot of similar glue code but it's a price I'm willing to pay to avoid the single page app.

I can try to express this idea in a PR in wasm-run.

[cecton]

Yes sorry... 😞 I should really read better. Not working properly: 🧠

The way I see it you don't even need entrypoint: we know what is the frontend crate, what is the backend crate and the CSS... we can use cargo-metadata to retrieve the list of files to watch and restart the different parts independently.

[JWorthe]

It is true that I designed wasm-run to have backend and frontend builds separated. I explain this better on the discussions in there: IMI-eRnD-Be/wasm-run#48 (comment) feel free to join! (Or we will join here, it might be easier.)

Thank you for this context. It really helped me to get a better feel for how the two compare.

Maybe a good compromise would be to drop the full-restart feature of wasm-run in favor of the build.rs like web-bundler does. Then change the run_server argument of wasm-run to use the name of a crate instead of an entrypoint

In my mental model, the backend triggers the build of the frontend with its build.rs script. I think it might be confusing to have this, and also have the frontend managing watching for changes in the backend. At least for me the circular dependency gets confusing 😅. In web-bundler land, I would probably prefer watching for frontend changes from my backend.

My current normal cargo command while I'm working on the frontend is cargo watch -x "run --bin backend". On the downside, it's slow because it rebuilds too much. On the plus side, it reliably rebuilds no matter what I changed, and our build process is relatively simple: it's just cargo build.

Another thing to consider is that needing to run in a build.rs script adds some particular weirdnesses to web-bundler. For example, it needs to create a new target directory (we call it web-target) because Cargo has a lock on the normal target directory while the build.rs script is running. We also need to clear out a bunch of environment variables that Cargo sets for build.rs scripts, since some of them might 'leak' and be accidentally used by wasm build.rs scripts that they don't apply to. For embedding, the whole frontend build process needs to be complete in the build.rs script. We can't rely on the output of a cargo run, because we need the frontend assets to all be ready for embedding (we use Rust Embed). I imagine wasm-run probably also has similar weirdnesses, and a hybrid approach might combine the weirdnesses in a way that becomes difficult to work with. Would these build.rs specific problems not add complexity to wasm-run?

If we all agree to merge, there are a few questions:

• The GitHub organization: our client is the GitHub organization of wasm-run (https://github.com/IMI-eRnD-Be). We should probably go for a neutral organization.
• Name of the project (I don't have strong feelings for wasm-run).

I also don't have strong feelings about web-bundler. Naming things is hard. It was a good name when it was part of our monorepo. It isn't a great standalone name. Regarding the Github organization, I think @stephanbuys might want to weigh in here.

Also, sorry to throw a spanner in the works, but for personal reasons I'm probably not going to be able to dedicate significant time to this in the next month or so. I'm happy to help with small changes, but rearchitecting things so that the two approaches work better together might be beyond what I can currently commit to.

[cecton]

For example, it needs to create a new target directory (we call it web-target) because Cargo has a lock on the normal target directory while the build.rs script is running.

@JWorthe can you give me more details about this? I find it weird because build.rs scripts actually tend to use OUTDIR which is in the target directory. Can you explain what is it locking exactly and how this lock behaves?

[JWorthe]

Sure. The problem comes in that, underneath wasm-pack, eventually it's calling Cargo. For simplicity, I'm going to call the two instances x86 Cargo (which it currently running the build script as part of your backend build) and wasm Cargo (which is the inner one doing the frontend build).

If you just run wasm-pack naively, then wasm Cargo will get stuck waiting for a lock on target which x86 Cargo is waiting for the build script to complete. I'm not sure exactly how the lock is implemented, but you can see it in action by running cargo build in two terminals for the same project. The one will wait with the message "Blocking waiting for file lock on build directory". So this is not good, we need to set wasm Cargo's target dir somewhere else.

As you say, it would probably be more correct to tell wasm Cargo to use something like $OUT_DIR/target, provided by x86 Cargo. If I remember right, that resolves to something like target/build/backend-{some-hash}/out/target. This will work, but if you make any changes to your backend build, it will change that hash in the path and effectively do a clean build of your frontend. It's possible that I didn't look deeply enough into this and if it's possible to make that dir more stable, so if you've seen this sort of thing handled better elsewhere I am keen to hear about it.

The compromise that we went with is to create a new web-target directory in the root for wasm Cargo to use, and then we copy the compiled .wasm file to somewhere in $OUT_DIR.

[cecton]

Ahhh so I assume wasm-pack runs a x86 build at some point... 🤔 interesting. Maybe using wasm-run's API would get rid of that issue. (The API is not open yet for you to use at the moment.)

As you say, it would probably be more correct to tell wasm Cargo to use something like $OUT_DIR/target, provided by x86 Cargo. If I remember right, that resolves to something like target/build/backend-{some-hash}/out/target.

I didn't really mean to say that but yes that's a solution. In wasm-run I download wasm-opt under target/wasm-opt. It works great and also there is no hash there. You could put things in target/web-bundler for instance. It's an idea.

Another thing to consider is that needing to run in a build.rs script adds some particular weirdnesses to web-bundler. [...]
I imagine wasm-run probably also has similar weirdnesses

Not at all actually. No locking issue, no environment variable, no nothing. wasm-run hooks itself to cargo run which is a normal binary in your project. Though... I'm lying. I found out yesterday a weirdness: if you put that binary in the same crate than the wasm frontend, then it might build the wrong dependencies in the wrong architecture on some conditions. I opened an issue on cargo for that because it looks like a bug. It's not really blocking for wasm-run because I can always advise people to use a dedicated crate for the "run" part (backend + frontend + run).

For embedding, the whole frontend build process needs to be complete in the build.rs script. We can't rely on the output of a cargo run, because we need the frontend assets to all be ready for embedding (we use Rust Embed).

I also love to embed a lot of things in my binaries. Rust-embed looks like a very interesting project as it combines the best of both world: loading during run time in debug, embedding in the binary in release. Thanks!

Would these build.rs specific problems not add complexity to wasm-run?

wasm-run has a few different entrypoints so these problems would just be handled in a specific entrypoint.

I don't think it's necessary to merge the 2 projects but I think we can easily if we want.

1. Merging would come with pros and cons:

   Pros: you and your team would benefit of a direct access to the project and you will be able to release the changes you need faster than if you would be a third-party.

   Cons: you need to maintain compatibility with the other workflows/processes of wasm-run (we will do the same the other way around).

2. web-bundler could use wasm-run as a dependency in replacement of wasm-pack:

   Pros: more freedom for everybody, the benefit of build customization of wasm-run. The only "con" for my side is that I need to open up a bit more the API but that's perfectly fine.

   Cons: changes you need on wasm-run will have to go through the validation of wasm-run which will make sure it doesn't break with the rest. It's no really a big deal, I personally have a fast response time. The team effort though is more scattered: I'm less likely to contribute to web-bundler but I guess this goes in both ways: we both have priorities for our own customers.

I'm personally all into sharing efforts and cross-companies collaboration. An OSS project only works if the maintainers work on it. Having more maintainers is only better for the survival of the project. On the other hand it reduces the hand we all have on it.

[JWorthe]

I saw your twitter thread, and wassemble is a cute name 😄

Merging seems like the right way to go. Even the con that maintaining compatibility between workflows seems like it's a pro in disguise: people will be able to jump between using the build script endpoint and the run endpoint without having to restructure how they are doing they SCSS or index.html files.

It seems like the next step would be to create a github organization and start moving things over there. When I was chatting with @stephanbuys about it this morning, he felt that the org should have a name that's broader than the single project in case we later want to bring more crates into it. My personal leaning is towards giving the org the same name as the project for now, and then change the org name later if we discover we have something else that we want to put in it. What do you think?

From our side, the people to add to the new org would be everyone in this group: https://github.com/orgs/panoptix-za/teams/oss-maintainers. That's @JWorthe, @stevedonovan, @tshepang and @stephanbuys.

[cecton]

I'm super happy we agree 😁 🎉

I don't like to disagree with people but I'm afraid I disagree with @stephanbuys 😅 There is no "cost" about having many organizations if we want for different things. Look at serde-rs, they have one organization for everything related to serde and maintained by the same team. Here our main product will be this bundler/assembler thingy so I think it's fair if we dedicate the organization to this.

Though I do agree with you: changing name later is not a problem at all. GitHub keep making redirects if that happens.

There is a problem with the name wassemble, it's already taken 😞 https://github.com/wingo/wassemble

There are alternatives:

• wasm-embed
• wasm-box
• embeddery
• pillowcase

I think I like wasm-embed the most 🤔

[stephanbuys]

Hi @cecton, pleased to "e-meet" you, I've been very glad to see how this thread is developing. When I decided to open source the bundler project I was secretly afraid that the code would just languish and not find greater/general use.

I would just like to clear something up, as I'm feeling a bit embarrassed about what's being attributed to me above. Github's automatic redirects only works for repos and it only reliably works up to the point that someone else (possibly malicious) registers the old organization's name, it's not ideal, and could create a vulnerability (docs). @JWorthe will happily testify that I often seem paranoid 😅, and to me, build tools, fall into the "abundance of caution" category.

In the discussion we had internally I pointed out the rust-wasm organization as an example, this is because I intend to release more of Panoptix's crates as open source that fit into the "rust and wasm" ecosystem (specifically, I'm thinking of framework neutral contributions), but it should have been obvious to me that the rust-wasm style of org isn't the intended model for this thread.

If this thread's intent is to create an org similar to tokio, serde, etc (as opposed to rust-wasm style organizations) then I strongly agree and you have my support.

[cecton]

Hi @cecton, pleased to "e-meet" you,

👋

I would just like to clear something up, as I'm feeling a bit embarrassed about what's being attributed to me above. Github's automatic redirects only works for repos and it only reliably works up to the point that someone else (possibly malicious) registers the old organization's name, it's not ideal, and could create a vulnerability (docs). @JWorthe will happily testify that I often seem paranoid sweat_smile, and to me, build tools, fall into the "abundance of caution" category.

That's a very good point 🤔 thanks for clearing that out!!

If this thread's intent is to create an org similar to tokio, serde, etc (as opposed to rust-wasm style organizations) then I strongly agree and you have my support.

Yes! 😁 We need to be neutral as this won't belong to a single company anymore but to an OSS organization.

[cecton]

I've discussed with the rest of my team and it is settled. Only the name of the project rename and then we can create the organization.

There is a problem with the name wassemble, it's already taken disappointed https://github.com/wingo/wassemble

There are alternatives:

* wasm-embed

* wasm-box

* embeddery

* pillowcase

I think I like wasm-embed the most thinking

[fstephany]

Thanks @cecton , @JWorthe and @stephanbuys for moving forward with this!

Like @JWorthe, my favorite name is "wassemble" I don't see the existing https://github.com/wingo/wassemble as a problem as it seems to be a solo/side project with no visible intention of being a mainstream tool.

I also like "pillowcase" because it's cute and it doesn't have any reference to wasm. Doing so does not lock the project to wasm only.

[JWorthe]

There is a problem with the name wassemble, it's already taken 😞 https://github.com/wingo/wassemble

I didn't notice that one since I only checked crates, not github. Naming is hard 😅

We could also disambiguate by spelling ours differently. wasmble seems to be free.

[cecton]

Wait, @fstephany you're right. It's a 1 star project not updated recently.

Ok let's go for wassemble. I will create the organization and invite everybody.

(@JWorthe maybe it's best your invite your people yourself as you might not want everybody to be owner)

[cecton]

(It is still time to change of name if anybody wants. wasmbl is kinda cool imo 🤔 )

I will start working on merging (stupidly) web-bundler to wasm-run and rename the project this weekend. So we have a good starting point and some time if we change our mind on something.

[tshepang]

wasmbl makes me joyous, but wasmble is fine too

[JWorthe]

I can get behind either, so going for the name that brings @tshepang joy seems good.

[cecton]

But you prefer that than wassemble?

[stephanbuys]

I'll throw my vote behind wasmbl, I'm with @tshepang in that it makes me smile. wassemble is cool but it's quite long.

[fstephany]

Looks like we've got a winner with wasmbl 👌

[cecton]

Hey everyone 👋

I finally managed to merge web-bundler to wasmbl. We can now move to this issue to continue the project: IMI-eRnD-Be/wasm-run#54 Please subscribe and please answer there on that issue from now on.

It's a very basic merge: just a module called "bundler" and that's it. The CI is working so we can start to modify things confidently with a good basis.

It took me a lot more time than I anticipated because the CI for Windows is locking during the bundler tests. I suspect this is related to wasm-pack but I don't have any evidence. Either way I suggest we just replace wasm-pack by the internals of wasm-run, then we can reactive the tests and see if they work.

I made a checklist with the things I believe we need to do and the things I noticed we could improve on the API. Don't hesitate to add things to that list or open a discussion on their points.

[cecton]

Hello 👋

I noticed there hasn't been much activity on wasmbl since we agreed on working together. I understand it's maybe not your preoccupation at the moment. So I think I will move on with the project and our collaboration.

Don't hesitate to contact me if you want to retry or do something else.

Best of luck to you all

[stephanbuys]

Hi, you're 100% correct, please go ahead and do what you need to do.

Best of luck to you too

[JWorthe]

Sorry about that 😅 Various life things have happened (some good, some bad) and February feels very long ago now.
Best of luck to you!

[cecton]

Sorry about that sweat_smile Various life things have happened (some good, some bad) and February feels very long ago now.

Thanks! No worry, I understand