Module upload format + custom build commands + durable object support #1677

xortive · 2020-12-11T19:22:18Z

remove linter and add custom build options as part of JS TargetType

JS projects with current config options will continue to work the same.
if a [builder-config] section is added, a custom build command can be configured. Wrangler will handle watching for changes and running the build command for the user.

support uploading in module format

when the script-format field in [builder-config] is set to modules, we should upload using the new modules format, uploading each file in builder-config.output_dir as a module.

The main_module is determined by looking at the module field in package.json. this should point to the bundled module in the output directory.

for service-worker builds, the behavior is the same as always: the main field in package.json is what will be uploaded, so this should be the output of your build tool.

support creating/implementing durable object namespaces
support binding/using durable object namespaces

This PR closes a lot of tickets with the same general theme (I hit xyz build edge case and I need a new feature to fix it/I want to use newer webpack version/I don't want to use webpack) that all come down to the limitations of wrangler's current build setup. Custom builds will allow users to setup their own build step, which wrangler will upload the results of unmodified to the API.

closes #206
closes #820
closes #954
partially addresses #1102
closes #1397 (just make custom build script)
closes #1550
closes #1557
closes #1558

kentonv · 2020-12-28T18:48:09Z

Is there a design doc somewhere? I don't know this codebase well enough to review the code but I'm interested to understand how this will end up looking from the user's perspective.

tillkruss · 2020-12-29T21:13:40Z

Is there a design doc somewhere? I don't know this codebase well enough to review the code but I'm interested to understand how this will end up looking from the user's perspective.

And may I request an example of how to use a simple example like the Counter in the folder structure of a Worker Site?

koeninger · 2020-12-29T21:28:06Z

@kentonv the only design doc was that google doc linked from our meeting on dec 9.

I believe Matt's plan was to share an example wrangler template, which should hopefully make things clearer. It'll be at least a couple weeks before there's more progress on this PR though.

xtuc · 2020-12-29T22:41:46Z

@koeninger would you mind sharing the document? i'd like to take a look too

ags799 · 2020-12-29T23:16:12Z

Me too.

…

On Tue, Dec 29, 2020 at 4:41 PM Sven Sauleau ***@***.***> wrote: @koeninger <https://github.com/koeninger> would you mind sharing the document? i'd like to take a look too — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1677 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWSCDNYRI5244SSCSO5D4DSXJLLNANCNFSM4UXDVOQA> .

src/deploy/durable_objects.rs

src/upload/form/mod.rs

koeninger · 2021-01-05T22:11:22Z

src/settings/toml/builder.rs

+}
+
+fn build_dir() -> PathBuf {
+    default_warning("build dir", BUILD_DIR);


It's a little confusing to warn on build dir and src dir when they aren't actually used if upload_format is set to service-worker, right?

edit - looks like src_dir is used, but only for watch? It's still confusing that I can silence the warning by putting a nonexistent directory in there, and publish fine.

I'll address this when I address @ags799 concern below about having src/build dir be ./, we don't want to allow that or allow nonexistent directories.

I ended up removing the default warning. I'll add a separate validation for the ./ and nonexistent directory cases

src/deploy/durable_objects.rs

src/build/mod.rs

src/commands/publish.rs

src/settings/toml/builder.rs

ags799 · 2021-01-07T22:49:52Z

src/settings/toml/builder.rs

@@ -0,0 +1,66 @@
+use std::env;


It would be really helpful to see an example TOML with the new additions.

The new additions would look something like this for a durable object

[builder_config] build_dir = "./build" src_dir = "./" upload_format = "modules" [durable_objects] implements = [ { class_name = "Counter" } ]

or for a regular service worker just

[builder_config] build_command = "./dummy-build.sh" build_dir = "./my-build-dir" src_dir = "./does-not-exist" upload_format = "service-worker"

Lots of underscores and duplicate info... Perhaps it could be streamlined?

[build] cwd = "." command = "sh ./dummy-build.sh" output = "./build" format = "modules" [durable_objects] # can't really avoid that one implements = [{ class = "Counter" }]

I say this because toml is pretty flexible with how you can write out tables and while you may write a table like that, I may write it like

build.cwd = "." build.command = "sh ./dummy-build.sh" build.output = "./build" build.format = "modules" durable_objects.implements = [{ class = "Counter" }]

With the long names you get

builder_config.build_dir = "." builder_config.build_command = "sh ./dummy-build.sh" builder_config.src_dir = "./build" builder_config.upload_format = "modules" durable_objects.implements = [{ class_name = "Counter" }]

Toml accepts both but shorter names look better with the second format.

I tend to prefer verbosity when it eliminates confusion, but I agree on build_command -> command.
I'll defer to @greg-mckeon on the rest

I like command, and I think that src_dir should be working_dir. And I mentioned earlier that build_dir should be upload_dir.

I'd personally want it to be configurable with reasonable defaults. So you can choose a different directory to watch, but it uses the build command's cwd by default, or perhaps just the root folder (where wrangler.toml is).

Why are we talking about configuring cwd?
Wrangler won't work anywhere but the directory with wrangler.toml
If someone needs a different cwd for their build, they can change to it and back in their build script.

Thanks for explaining @xortive. I think src_dir is good.

ah :/ I made the change to watch_dir already. It's easy to put it back, and I'm completely indifferent to which one we pick

Oh that's fine idc either way.

src/upload/form/modules_worker.rs

src/upload/package.rs

src/watch/mod.rs

ags799 · 2021-01-11T17:37:48Z

Does it make sense to support a modules project without a build step? If I had a modules project with one source file at ./index.mjs:

I would need to configure my src_dir and build_dir to be './'.
Wrangler would then try to upload all files in './' as modules, and fail on something like .gitignore because it can't parse the extension.
I would move index.mjs to a new src directory and update src_dir and build_dir.

I think we can solve this by moving upload_format from builder_config to top-level configuration. So we don't tie module support to having a build step with source and build directories.

I'll make a line comment about the .gitignore failure.

src/upload/form/project_assets.rs

koeninger · 2021-01-11T17:48:34Z

I think we can solve this by moving upload_format from builder_config to top-level configuration. So we don't tie module support to having a build step with source and build directories.

We still need to know where to look for files to upload, right? Having to specify a subdirectory seems about as inconvenient / confusing as having to specify ignore patterns. I guess we could special case single file modules using the filename from package.json module, but that seems hacky

move service_worker specific metadata to service_worker.rs

xortive · 2021-02-25T14:33:45Z

going to open a new PR with just the durable object changes. will be merging the modules + toml-edit changes into a release branch for the rc

…ate so format is preserved (#1745)

This lands support for `[wasm_modules]` as defined by cloudflare/wrangler-legacy#1677. wasm modules can be defined in service-worker format with configuration in wrangler.toml as - ``` [wasm modules] MYWASM = "./path/to/my-wasm.wasm" ``` The module will then be available as the global `MYWASM` inside your code. Note that this ONLY makes sense in service-worker format workers (for now). (In the future, we MAY enable wasm module imports in service-worker format (i.e. `import MYWASM from './path/to/my-wasm.wasm'`) and global imports inside modules format workers.)

This lands support for `[wasm_modules]` as defined by cloudflare/wrangler-legacy#1677. wasm modules can be defined in service-worker format with configuration in wrangler.toml as - ``` [wasm_modules] MYWASM = "./path/to/my-wasm.wasm" ``` The module will then be available as the global `MYWASM` inside your code. Note that this ONLY makes sense in service-worker format workers (for now). (In the future, we MAY enable wasm module imports in service-worker format (i.e. `import MYWASM from './path/to/my-wasm.wasm'`) and global imports inside modules format workers.)

This implements support for `[text_blobs]` as defined by cloudflare/wrangler-legacy#1677. Text blobs can be defined in service-worker format with configuration in `wrangler.toml` as - ``` [text_blobs] MYTEXT = "./path/to/my-text.file" ``` The content of the file will then be available as the global `MYTEXT` inside your code. Note that this ONLY makes sense in service-worker format workers (for now). Workers Sites now uses `[text_blobs]` internally. Previously, we were inlining the asset manifest into the worker itself, but we now attach the asset manifest to the uploaded worker. I also added an additional example of Workers Sites with a modules format worker.

xortive requested a review from koeninger December 11, 2020 19:22

xortive requested a review from a team as a code owner December 11, 2020 19:22

xortive force-pushed the malonso/modules-do branch 4 times, most recently from 4a42cb4 to cc82568 Compare December 11, 2020 21:14

xortive requested a review from ags799 December 11, 2020 22:54

xortive force-pushed the malonso/modules-do branch 7 times, most recently from 441c591 to 34e3164 Compare December 17, 2020 17:14

koeninger mentioned this pull request Dec 31, 2020

Custom build commands as part of wrangler dev #1557

Closed

koeninger reviewed Jan 5, 2021

View reviewed changes

src/deploy/durable_objects.rs Outdated Show resolved Hide resolved

koeninger reviewed Jan 5, 2021

View reviewed changes

src/upload/form/mod.rs Show resolved Hide resolved

koeninger reviewed Jan 5, 2021

View reviewed changes

src/deploy/durable_objects.rs Outdated Show resolved Hide resolved

ags799 reviewed Jan 8, 2021

View reviewed changes

cloudflare deleted a comment from koeninger Jan 8, 2021

koeninger mentioned this pull request Jan 11, 2021

Support uploading a pre-built WASM module #1716

Closed

ags799 reviewed Jan 11, 2021

View reviewed changes

src/upload/form/project_assets.rs Show resolved Hide resolved

xortive added 2 commits February 25, 2021 08:32

cleanup: separate out service-worker form building logic

c28e9fe

move service_worker specific metadata to service_worker.rs

support uploading module-based scripts

525eb02

xortive force-pushed the malonso/modules-do branch from b86946b to 1cf5fe6 Compare February 25, 2021 14:32

use toml-edit for modifying provided wrangler.toml for wrangler gener…

9265785

…ate so format is preserved (#1745)

xortive force-pushed the malonso/modules-do branch from 1cf5fe6 to 9265785 Compare February 25, 2021 14:34

xortive changed the base branch from master to malonso/modules-rc February 25, 2021 14:34

xortive merged commit e5b8d5b into malonso/modules-rc Feb 25, 2021

delete-merged-branch bot deleted the malonso/modules-do branch February 25, 2021 14:35

This was referenced Mar 7, 2021

Feature/monorepo support #1748

Merged

Revert #1748, update changelog-generator package.json, and release 1.15.0 #1804

Merged

dependabot bot mentioned this pull request Mar 9, 2021

Bump @cloudflare/wrangler from 1.14.1 to 1.15.0 jacobwgillespie/saved-for-later#78

Closed

manniL mentioned this pull request Apr 28, 2021

[CF] Worker output is broken nuxt/nuxt#10932

Closed

threepointone mentioned this pull request Feb 7, 2022

feat: support [wasm_modules] for service-worker format workers cloudflare/workers-sdk#409

Merged

github-actions bot mentioned this pull request Feb 9, 2022

Version Packages cloudflare/workers-sdk#407

Merged

threepointone mentioned this pull request Feb 23, 2022

feat: implement [text_blobs] cloudflare/workers-sdk#518

Merged

github-actions bot mentioned this pull request Feb 24, 2022

Version Packages cloudflare/workers-sdk#484

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Module upload format + custom build commands + durable object support #1677

Module upload format + custom build commands + durable object support #1677

xortive commented Dec 11, 2020 •

edited

kentonv commented Dec 28, 2020

tillkruss commented Dec 29, 2020

koeninger commented Dec 29, 2020

xtuc commented Dec 29, 2020

ags799 commented Dec 29, 2020 via email

koeninger Jan 5, 2021 •

edited

xortive Jan 11, 2021 •

edited

xortive Jan 25, 2021

ags799 Jan 7, 2021

koeninger Jan 8, 2021

ObsidianMinor Jan 21, 2021 •

edited

xortive Jan 25, 2021

ags799 Jan 25, 2021

ObsidianMinor Jan 26, 2021

koeninger Jan 26, 2021

ags799 Jan 26, 2021

xortive Jan 26, 2021

ags799 Jan 26, 2021

ags799 commented Jan 11, 2021

koeninger commented Jan 11, 2021

xortive commented Feb 25, 2021

Module upload format + custom build commands + durable object support #1677

Module upload format + custom build commands + durable object support #1677

Conversation

xortive commented Dec 11, 2020 • edited

kentonv commented Dec 28, 2020

tillkruss commented Dec 29, 2020

koeninger commented Dec 29, 2020

xtuc commented Dec 29, 2020

ags799 commented Dec 29, 2020 via email

koeninger Jan 5, 2021 • edited

Choose a reason for hiding this comment

xortive Jan 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ObsidianMinor Jan 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ags799 commented Jan 11, 2021

koeninger commented Jan 11, 2021

xortive commented Feb 25, 2021

xortive commented Dec 11, 2020 •

edited

koeninger Jan 5, 2021 •

edited

xortive Jan 11, 2021 •

edited

ObsidianMinor Jan 21, 2021 •

edited