wrangler support Queues http-pull consumer config and cli #5231

w-kuhn · 2024-03-12T18:19:07Z

What this PR solves / how to test

Cloudflare Queues currently only supports worker consumers. Support for http-pull consumers is coming soon and part of that support is wranglerization.

This change extends the current worker-only consumer config (wrangler queues consumer add <queue-name> <script-name> ...options) to include wrangler queues consumer http add|remove <queue-name> ...options and wrangler queues consumer worker add|remove <queue-name> <script-name> ...options.

wrangler queues consumer

Configure Queue Consumers

Commands:
  wrangler queues consumer add <queue-name> <script-name>     Add a Queue Worker Consumer
  wrangler queues consumer remove <queue-name> <script-name>  Remove a Queue Worker Consumer
  wrangler queues consumer http                               Configure Queue HTTP Pull Consumers
  wrangler queues consumer worker                             Configure Queue Worker Consumers

and:

wrangler queues consumer http

Configure Queue HTTP Pull Consumers

Commands:
  wrangler queues consumer http add <queue-name>     Add a Queue HTTP Pull Consumer
  wrangler queues consumer http remove <queue-name>  Remove a Queue HTTP Pull Consumer

Author has addressed the following

Tests
- TODO (before merge)
- Included
- Not necessary because:
Changeset (Changeset guidelines)
- TODO (before merge)
- Included
- Not necessary because:
Public documentation
- TODO (before merge)
- Cloudflare docs PR(s): [QUEUES] HTTP Pull docs cloudflare-docs#12844
- Not necessary because:

…consumers

changeset-bot · 2024-03-12T18:19:11Z

🦋 Changeset detected

Latest commit: c7322a3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
wrangler	Minor
@cloudflare/vitest-pool-workers	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

packages/wrangler/src/queues/cli/commands/consumer/http-pull/add.ts

packages/wrangler/src/queues/cli/commands/index.ts

packages/wrangler/src/queues/client.ts

github-actions · 2024-03-12T18:47:16Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-wrangler-5231

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/5231/npm-package-wrangler-5231

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-wrangler-5231 dev path/to/script.js

Additional artifacts:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-create-cloudflare-5231 --no-auto-update

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-cloudflare-kv-asset-handler-5231

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-miniflare-5231

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-cloudflare-pages-shared-5231

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/8366124306/npm-package-cloudflare-vitest-pool-workers-5231

Note that these links will no longer work once the GitHub Actions artifact expires.

wrangler@3.36.0 includes the following runtime dependencies:

Package	Constraint	Resolved
`miniflare`	workspace:*	3.20240314.0
`workerd`	1.20240314.0	1.20240314.0
`workerd --version`	1.20240314.0	2024-03-14

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

elithrar

LGTM from a help text / CLI sub-command surface perspective!

…config

codecov · 2024-03-12T20:22:27Z

Codecov Report

Attention: Patch coverage is 92.07921% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 72.14%. Comparing base (9fd7eba) to head (c7322a3).
Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5231      +/-   ##
==========================================
+ Coverage   71.96%   72.14%   +0.17%     
==========================================
  Files         313      317       +4     
  Lines       16239    16318      +79     
  Branches     4157     4164       +7     
==========================================
+ Hits        11687    11773      +86     
+ Misses       4552     4545       -7

Files	Coverage Δ
packages/wrangler/src/config/validation.ts	`90.12% <ø> (ø)`
.../src/queues/cli/commands/consumer/http-pull/add.ts	`100.00% <100.00%> (ø)`
...rc/queues/cli/commands/consumer/http-pull/index.ts	`100.00% <100.00%> (ø)`
...c/queues/cli/commands/consumer/http-pull/remove.ts	`100.00% <100.00%> (ø)`
...ler/src/queues/cli/commands/consumer/worker/add.ts	`100.00% <100.00%> (ø)`
.../src/queues/cli/commands/consumer/worker/remove.ts	`100.00% <100.00%> (ø)`
packages/wrangler/src/queues/cli/commands/index.ts	`100.00% <100.00%> (ø)`
packages/wrangler/src/queues/client.ts	`96.42% <94.73%> (-0.87%)`	⬇️
packages/wrangler/src/deploy/deploy.ts	`90.30% <93.10%> (+0.34%)`	⬆️
...wrangler/src/queues/cli/commands/consumer/index.ts	`84.61% <80.00%> (-15.39%)`	⬇️
... and 1 more

... and 7 files with indirect coverage changes

petebacondarwin

A few suggestions. But looking pretty good.

Oh and I think this should have a changeset, right?
(Edit: I see that you have marked this, and docs, as "TODO before merge" - so we should hold you to that!)

packages/wrangler/src/deploy/deploy.ts

packages/wrangler/src/queues/cli/commands/consumer/http-pull/add.ts

packages/wrangler/src/queues/cli/commands/index.ts

packages/wrangler/src/queues/client.ts

jbwcloudflare · 2024-03-17T19:58:40Z

I think we'll want a new test that runs wrangler deploy with an HTTP consumer configured in wrangler.toml, like this one for "script" consumers:

workers-sdk/packages/wrangler/src/__tests__/deploy.test.ts

Lines 8609 to 8645 in f5e4922

    
           		it("should update queue consumers on deploy", async () => { 
        
           			writeWranglerToml({ 
        
           				queues: { 
        
           					consumers: [ 
        
           						{ 
        
           							queue: "queue1", 
        
           							dead_letter_queue: "myDLQ", 
        
           							max_batch_size: 5, 
        
           							max_batch_timeout: 3, 
        
           							max_retries: 10, 
        
           						}, 
        
           					], 
        
           				}, 
        
           			}); 
        
           			await fs.promises.writeFile("index.js", `export default {};`); 
        
           			mockSubDomainRequest(); 
        
           			mockUploadWorkerRequest(); 
        
           			mockGetQueue("queue1"); 
        
           			mockPutQueueConsumer("queue1", "test-name", { 
        
           				dead_letter_queue: "myDLQ", 
        
           				settings: { 
        
           					batch_size: 5, 
        
           					max_retries: 10, 
        
           					max_wait_time_ms: 3000, 
        
           				}, 
        
           			}); 
        
           			await runWrangler("deploy index.js"); 
        
           			expect(std.out).toMatchInlineSnapshot(` 
        
           			"Total Upload: xx KiB / gzip: xx KiB 
        
           			Uploaded test-name (TIMINGS) 
        
           			Published test-name (TIMINGS) 
        
           			  https://test-name.test-sub-domain.workers.dev 
        
           			  Consumer for queue1 
        
           			Current Deployment ID: Galaxy-Class" 
        
           		`); 
        
           		});

…config

…sumer

…config

w-kuhn · 2024-03-20T17:55:31Z

addressed feedback and it's all green - I think this is ready for final review.

ETA: We're going to need to wait to ship this until the api for updating pull consumers is available. Will need to utilize that behavior in toml config.

@jbwcloudflare I updated wrangler deploy to check for a pull consumer first then update if it exists or create if not. I can't fully test it until that config api lands.

tested deploy multiple times on pre-release build against staging api, all good 👍

jbwcloudflare · 2024-03-12T19:21:04Z

packages/wrangler/src/queues/client.ts

@@ -96,6 +97,33 @@ export async function postConsumer(
 	);
 }

+export async function postTypedConsumer(


IMO it's more confusing than helpful to try to offer a generic "typed consumer" function here.

I suggest just having a separate postWorkerConsumer and postHttpConsumer. Each one can have request/response types that apply to that consumer type.

packages/wrangler/src/queues/cli/commands/index.ts

packages/wrangler/src/deploy/deploy.ts

…config

petebacondarwin

Looks good to me, from a Wrangler team perspective. Nice work @w-kuhn

petebacondarwin · 2024-03-20T20:54:17Z

Oh actually the test failures are legit: https://github.com/cloudflare/workers-sdk/actions/runs/8364727430/job/22900763340?pr=5231#step:8:2073

Need to update some snapshots.

w-kuhn · 2024-03-20T21:35:44Z

Oh actually the test failures are legit: https://github.com/cloudflare/workers-sdk/actions/runs/8364727430/job/22900763340?pr=5231#step:8:2073

Need to update some snapshots.

Thanks! Fixed that up. Just need to open up some gates before merging tomorrow. 🥳

w-kuhn added 2 commits March 12, 2024 11:41

MQ-307 Add typed consumer types and client api

51f0941

MQ-307 Update queue consumer config to allow type, support http pull …

4213ee9

…consumers

w-kuhn requested a review from a team as a code owner March 12, 2024 18:19

w-kuhn commented Mar 12, 2024

View reviewed changes

packages/wrangler/src/queues/cli/commands/consumer/http-pull/add.ts Show resolved Hide resolved

w-kuhn commented Mar 12, 2024

View reviewed changes

packages/wrangler/src/queues/cli/commands/index.ts Outdated Show resolved Hide resolved

w-kuhn commented Mar 12, 2024

View reviewed changes

packages/wrangler/src/queues/client.ts Show resolved Hide resolved

elithrar reviewed Mar 12, 2024

View reviewed changes

w-kuhn added 2 commits March 12, 2024 15:10

MQ-307 Make queue consumer type optional in config

8b43c4e

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

f5e4922

…config

petebacondarwin reviewed Mar 12, 2024

View reviewed changes

w-kuhn added 4 commits March 18, 2024 10:15

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

8224143

…config

MQ-307 Update for PR feedback

fa53013

MQ-307 Add changeset

a82cf50

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

144f6b0

…config

This comment was marked as outdated.

Sign in to view

MQ-307 Update changeset to be minor bump

aecefc0

This comment was marked as outdated.

Sign in to view

petebacondarwin added blocked Blocked on other work internal Requires support from the Cloudflare Platform labels Mar 19, 2024

petebacondarwin self-requested a review March 19, 2024 16:11

w-kuhn added 2 commits March 19, 2024 17:13

MQ-307 PR feedback: wrangler deploy updates an existing http-pull con…

0ab450a

…sumer

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

90d4b3d

…config

This comment was marked as outdated.

Sign in to view

w-kuhn added 2 commits March 20, 2024 09:27

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

6cb13ad

…config

Use new Queues config endpoints for pull consumers

b62460a

jbwcloudflare approved these changes Mar 20, 2024

View reviewed changes

w-kuhn added 4 commits March 20, 2024 13:25

Add retry delay option for Queues http pull consumers

c253de2

Fix config api field

f77cd2d

Retry delay is always in seconds

4613164

Merge branch 'main' into wendy/MQ-307-wrangler-support-pull-consumer-…

ad4e7ca

…config

petebacondarwin approved these changes Mar 20, 2024

View reviewed changes

Update snapshot

c7322a3

RamIdeas merged commit e88ad44 into cloudflare:main Mar 21, 2024
15 checks passed

workers-devprod mentioned this pull request Mar 21, 2024

Version Packages #5326

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wrangler support Queues http-pull consumer config and cli #5231

wrangler support Queues http-pull consumer config and cli #5231

w-kuhn commented Mar 12, 2024 •

edited by lrapoport-cf

changeset-bot bot commented Mar 12, 2024 •

edited

github-actions bot commented Mar 12, 2024 •

edited

elithrar left a comment

codecov bot commented Mar 12, 2024 •

edited

petebacondarwin left a comment •

edited

jbwcloudflare commented Mar 17, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

w-kuhn commented Mar 20, 2024

jbwcloudflare Mar 12, 2024

petebacondarwin left a comment

petebacondarwin commented Mar 20, 2024

w-kuhn commented Mar 20, 2024

wrangler support Queues http-pull consumer config and cli #5231

wrangler support Queues http-pull consumer config and cli #5231

Conversation

w-kuhn commented Mar 12, 2024 • edited by lrapoport-cf

What this PR solves / how to test

Author has addressed the following

changeset-bot bot commented Mar 12, 2024 • edited

🦋 Changeset detected

github-actions bot commented Mar 12, 2024 • edited

elithrar left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 12, 2024 • edited

Codecov Report

petebacondarwin left a comment • edited

Choose a reason for hiding this comment

jbwcloudflare commented Mar 17, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

w-kuhn commented Mar 20, 2024

jbwcloudflare Mar 12, 2024

Choose a reason for hiding this comment

petebacondarwin left a comment

Choose a reason for hiding this comment

petebacondarwin commented Mar 20, 2024

w-kuhn commented Mar 20, 2024

w-kuhn commented Mar 12, 2024 •

edited by lrapoport-cf

changeset-bot bot commented Mar 12, 2024 •

edited

github-actions bot commented Mar 12, 2024 •

edited

codecov bot commented Mar 12, 2024 •

edited

petebacondarwin left a comment •

edited