Queues support #2064

jbw1991 · 2022-10-21T18:51:02Z

This is ready for review!

Notable changes:

Adds new "queues" CLI subcommand for interacting with Queues (create/delete/list)
Adds support for Queue producer/consumer configuration in wrangler.toml
Updates "publish" to both:
- Incude Queue bindings (Producers) in the script upload API call
- Make additional API calls to configure Queue Consumers
Updates "dev" to support running Queues in local mode

* Add CLI commands for Queues * Handle queue producers and consumers on wrangler publish * fixup! Handle queue producers and consumers on wrangler publish * Remove consumer commands from CLI * Update queue settings names Co-authored-by: Josh Wheeler <jwheeler@cloudflare.com>

* Support queues in wrangler dev local mode. remote mode is an error for now * fixup! Support queues in wrangler dev local mode. remote mode is an error for now * Move remote wrangler dev + queues check Co-authored-by: Josh Wheeler <jwheeler@cloudflare.com>

changeset-bot · 2022-10-21T18:51:05Z

🦋 Changeset detected

Latest commit: 726d636

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
wrangler	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2022-10-21T18:53:18Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.developers.workers.dev/runs/3394512414/npm-package-wrangler-2064

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.developers.workers.dev/prs/2064/npm-package-wrangler-2064

Or you can use npx with this latest build directly:

npx https://prerelease-registry.developers.workers.dev/runs/3394512414/npm-package-wrangler-2064 dev path/to/script.js

Additional artifacts:

npm install https://prerelease-registry.developers.workers.dev/runs/3394512414/npm-package-cloudflare-pages-shared-2064

codecov · 2022-10-21T18:59:40Z

Codecov Report

Merging #2064 (726d636) into main (02ca556) will increase coverage by 0.26%.
The diff coverage is 91.74%.

@@            Coverage Diff             @@
##             main    #2064      +/-   ##
==========================================
+ Coverage   73.16%   73.43%   +0.26%     
==========================================
  Files         129      139      +10     
  Lines        8668     8965     +297     
  Branches     2270     2327      +57     
==========================================
+ Hits         6342     6583     +241     
- Misses       2326     2382      +56

Impacted Files	Coverage Δ
packages/wrangler/src/secret/index.ts	`86.56% <ø> (ø)`
packages/wrangler/src/worker.ts	`71.42% <ø> (ø)`
packages/wrangler/src/dev/local.tsx	`27.05% <25.00%> (-0.09%)`	⬇️
packages/wrangler/src/dev.tsx	`87.73% <50.00%> (-0.36%)`	⬇️
packages/wrangler/src/config/validation.ts	`90.77% <87.03%> (-0.36%)`	⬇️
packages/wrangler/src/publish/publish.ts	`86.11% <90.32%> (+0.40%)`	⬆️
packages/wrangler/src/queues/client.ts	`97.29% <97.29%> (ø)`
packages/wrangler/src/config/index.ts	`92.59% <100.00%> (+0.28%)`	⬆️
packages/wrangler/src/create-worker-upload-form.ts	`93.47% <100.00%> (+0.14%)`	⬆️
packages/wrangler/src/dev/start-server.ts	`66.88% <100.00%> (+0.22%)`	⬆️
... and 16 more

…t they are needeed for development and early testing

packages/wrangler/src/index.tsx

penalosa

It'll be exciting to get queues support landed! I had a couple comments, mainly around import export conventions

packages/wrangler/src/__tests__/publish.test.ts

packages/wrangler/src/queues/cli/commands/consumer/add.ts

packages/wrangler/src/queues/cli/commands/consumer/remove.ts

packages/wrangler/src/queues/cli/commands/list.ts

packages/wrangler/src/queues/cli/config.ts

packages/wrangler/src/queues/client.ts

packages/wrangler/src/queues/cli/commands/list.ts

…mote mode. This is not yet supported during the beta

jbw1991 · 2022-11-02T18:55:12Z

👋 Hey @rozenmd @penalosa I think I have covered all of your comments, please let me know if there are any other desired changes or if we should pull anyone else in.

Otherwise, I think this is ready to go 😄 . I also attempted a changeset for the first time. Let me know if it needs work.

JacobMGEvans

Looking good! Anything marked as non-blocking (mainly the MSW stuff) doesn't need to be done by end of this week.

packages/wrangler/src/__tests__/publish.test.ts

packages/wrangler/src/__tests__/queues.test.ts

packages/wrangler/src/publish/publish.ts

JacobMGEvans

I don't have anything else that is jumping out at me, it looks great. Tentatively approving on @penalosa's follow up. (DM internally if need be @jbw1991, lets get this in today 😃 )

penalosa

This looks pretty good! One lingering comment, but other than that lgtm

penalosa · 2022-11-04T14:04:05Z

packages/wrangler/src/publish/publish.ts

+			const queueErr = err as FetchError;
+			if (queueErr.code === 100123) {
 				// queue_not_found
 				throw new Error(
 					`Queue "${queue}" does not exist. To create it, run: wrangler queues create ${queue}`


I missed it in the first review, but this catch block shouldn't swallow all non-handled errors—it should re-throw them

Whoops, that was the intent. Nice catch! Fixed.

Skye-31 · 2022-11-04T14:08:47Z

Fwiw, queues need to be added the the D1 Shim file for it them to work with D1, see https://github.com/cloudflare/wrangler2/blob/main/packages/wrangler/templates/d1-beta-facade.js#L165-L173

JacobMGEvans · 2022-11-04T14:51:47Z

Fwiw, queues need to be added the the D1 Shim file for it them to work with D1, see https://github.com/cloudflare/wrangler2/blob/main/packages/wrangler/templates/d1-beta-facade.js#L165-L173

I have no context on the shim, I will take @Skye-31's word on this @jbw1991

penalosa · 2022-11-04T15:28:23Z

I think @geelen might have some more context on the D1 shim? Let's not merge this until that's figured out

Skye-31 · 2022-11-04T15:31:14Z

The shim calls the normal worker when D1 is used, but as it doesn't export a queue handler, the API throws the error of "no queue consumer exported" (or something similar)

https://discord.com/channels/595317990191398933/1022981666744061972/1035613298860773387 for employees who can see it, @JacobMGEvans

jbw1991 · 2022-11-04T16:13:30Z

@Skye-31 Thanks, I created an issue for the D1+Queues bug: #2131
We'll tackle that ASAP next week after getting @geelen 's input.

jbw1991 and others added 5 commits September 8, 2022 09:13

Merge branch 'main' into queues

3240841

Treat queue producers and consumers as optional

f69c8eb

Merge branch 'main' into queues

d4f17ce

jbw1991 requested review from rts-rob, w-kuhn, OilyLime, JacobMGEvans and a-robinson October 21, 2022 18:52

jbw1991 marked this pull request as ready for review October 25, 2022 14:00

jbw1991 requested review from rozenmd and petebacondarwin as code owners October 25, 2022 14:00

Re-add consumer CLI options. These may not make the final dev exp, bu…

d1c7108

…t they are needeed for development and early testing

rozenmd reviewed Oct 28, 2022

View reviewed changes

packages/wrangler/src/index.tsx Outdated Show resolved Hide resolved

lrapoport-cf requested a review from caass October 28, 2022 15:46

penalosa requested changes Oct 31, 2022

View reviewed changes

jbwcloudflare added 2 commits October 31, 2022 15:19

Address PR feedback

868ada7

fixup! Address PR feedback

33fffa0

jbw1991 changed the title ~~[WIP] Queues support~~ Queues support Nov 1, 2022

jbwcloudflare added 2 commits November 2, 2022 13:42

Display a warning when attempting to use Queues under wrangler dev re…

0e004de

…mote mode. This is not yet supported during the beta

Include changeset for queues

cba4926

jbw1991 requested review from rozenmd and penalosa November 2, 2022 18:53

JacobMGEvans suggested changes Nov 2, 2022

View reviewed changes

Cleanup type imports and short variable names

d90bef0

Merge branch 'main' into queues

0e209cf

jbw1991 requested a review from JacobMGEvans November 3, 2022 18:32

This was referenced Nov 4, 2022

💬 [Discussion]: Should batch-timeout be in milliseconds as well? #2128

Closed

ESLint rule and have Prettier handle import type { Type } Pattern #2129

Closed

JacobMGEvans approved these changes Nov 4, 2022

View reviewed changes

penalosa approved these changes Nov 4, 2022

View reviewed changes

Rethrow error after checking for queue_not_found

726d636

jbw1991 merged commit 49b6a48 into main Nov 4, 2022

jbw1991 deleted the queues branch November 4, 2022 16:13

github-actions bot mentioned this pull request Nov 4, 2022

Version Packages #2121

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Queues support #2064

Queues support #2064

jbw1991 commented Oct 21, 2022 •

edited

changeset-bot bot commented Oct 21, 2022 •

edited

github-actions bot commented Oct 21, 2022 •

edited

codecov bot commented Oct 21, 2022 •

edited

penalosa left a comment

jbw1991 commented Nov 2, 2022

JacobMGEvans left a comment

JacobMGEvans left a comment

penalosa left a comment

penalosa Nov 4, 2022

jbw1991 Nov 4, 2022

Skye-31 commented Nov 4, 2022

JacobMGEvans commented Nov 4, 2022

penalosa commented Nov 4, 2022 •

edited

Skye-31 commented Nov 4, 2022

jbw1991 commented Nov 4, 2022

Queues support #2064

Queues support #2064

Conversation

jbw1991 commented Oct 21, 2022 • edited

changeset-bot bot commented Oct 21, 2022 • edited

🦋 Changeset detected

github-actions bot commented Oct 21, 2022 • edited

codecov bot commented Oct 21, 2022 • edited

Codecov Report

penalosa left a comment

Choose a reason for hiding this comment

jbw1991 commented Nov 2, 2022

JacobMGEvans left a comment

Choose a reason for hiding this comment

JacobMGEvans left a comment

Choose a reason for hiding this comment

penalosa left a comment

Choose a reason for hiding this comment

penalosa Nov 4, 2022

Choose a reason for hiding this comment

jbw1991 Nov 4, 2022

Choose a reason for hiding this comment

Skye-31 commented Nov 4, 2022

JacobMGEvans commented Nov 4, 2022

penalosa commented Nov 4, 2022 • edited

Skye-31 commented Nov 4, 2022

jbw1991 commented Nov 4, 2022

jbw1991 commented Oct 21, 2022 •

edited

changeset-bot bot commented Oct 21, 2022 •

edited

github-actions bot commented Oct 21, 2022 •

edited

codecov bot commented Oct 21, 2022 •

edited

penalosa commented Nov 4, 2022 •

edited