Bridge: add `/events` poller #1363

svix-onelson · 2024-07-09T05:40:29Z

Helps with https://github.com/svix/monorepo-private/issues/8654

Depends on client lib updates (not yet available).

I'd call this "quick and dirty" but it was actually a bit of a hefty lift to pull together. It is however fairly dirty.

The want is to be able to pull messages from /events and feed them into a receiver output.
The flow we'd see with SenderInput paired with ReceiverOutput is what we want, but the existing typing didn't allow for this.

Refactors were needed to:

lift out the parts from the HTTP server that run payloads through transformations then forward to an output.
update config to account for a new type of receiver: a "poller".
new traits/types for the poller to differentiate it from the standard webhook receiver (to ensure we don't accidentally pass one through and try to bind it to a URL).
Lots of "connective tissue" in the form of converters between config values and concrete ones that can actually do things.

Some of the "connective tissue" exists purely to mimic bits and pieces that existed for either the other receivers or senders (remember, this case is odd in that it's similar to both).

Refactorings aside, the poller itself boasts an exponential backoff for both error cases (either from /events or from the output) as well as for the case where the /events iterator is "done."

This diff comes with a promise that we will (soon) give these additions another look at clean up the stuff that doesn't make sense or feels redundant.

bridge/svix-bridge/src/webhook_receiver/mod.rs

bridge/svix-bridge/src/config/mod.rs

svix-jplatte

Nothing looks wrong, left some cleanup notes though (can be addressed in another PR).

bridge/svix-bridge/src/config/mod.rs

bridge/svix-bridge/src/main.rs

bridge/svix-bridge/src/webhook_receiver/mod.rs

svix-onelson

I'll take a look at making the itertools change while waiting on the rest of this stuff to coalesce. 🙏🏻

bridge/svix-bridge/src/main.rs

bridge/svix-bridge/src/config/mod.rs

bridge/svix-bridge/src/main.rs

bridge/svix-bridge/src/webhook_receiver/mod.rs

svix-onelson · 2024-07-10T16:47:11Z

OSS Review: itertools

This popular extension crate adds new methods for iterators.
Useful in this case as it cleans up a previous call to fold() designed
to partition a collection into 2 collections that can't be expressed
cleanly with a plain partition().

The selected version is not the newest, but it aligns with a selected
version already included in the project transitively.

cc @svix-james @jaymell

I'd call this "quick and dirty" but it was actually a bit of a hefty lift to pull together. It is however fairly dirty. The want is to be able to pull messages from `/events` and feed them into a receiver output. The flow we'd see with `SenderInput` paired with `ReceiverOutput` is what we want, but the existing typing didn't allow for this. Refactors were needed to: - lift out the parts from the HTTP server that run payloads through transformations then forward to an output. - update config to account for a new type of receiver: a "poller". - new traits/types for the poller to differentiate it from the standard webhook receiver (to ensure we don't accidentally pass one through and try to bind it to a URL). - Lots of "connective tissue" in the form of converters between config values and concrete ones that can actually do things. Some of the "connective tissue" exists purely to mimic bits and pieces that existed for either the other receivers or senders (remember, this case is odd in that it's similar to both). Refactorings aside, the poller itself boasts an exponential backoff for both error cases (either from `/events` or from the output) as well as for the case where the `/events` iterator is "done." This diff comes with a promise that we will (soon) give these additions another look at clean up the stuff that doesn't make sense or feels redundant.

Co-authored-by: Jonas Platte <158304798+svix-jplatte@users.noreply.github.com>

OSS Review: itertools This popular extension crate adds new methods for iterators. Useful in this case as it cleans up a previous call to `fold()` designed to partition a collection into 2 collections that can't be expressed cleanly with a plain `partition()`. The selected version is not the newest, but it aligns with a selected version already included in the project transitively.

There's a new authtoken endpoint for creating fresh events subscriptions. In the response is a base64 encoded json structure including the authtoken to use, the subscription id, and the app id - everything we need to build an appropriate URL for fetching events from the stream.

This gives access to the needed beta `events_subscription` API.

There were some gaps in the initial implementation. This switches to setting a flag within the message handling loop and breaking the loop so that the sleep time penalty (or reset) can be assessed at the bottom of the loop more consistently. Mostly this is about making sure any single message that has a problem makes the whole iterator batch retry.

svix-onelson · 2024-07-11T17:09:37Z

Some relatively new stuff since last review added, starting with 47e35a7

new config using the new "subscription token" concept
revisions to the backoff/sleep handling in the loop

Looks like the `done` field doesn't signify we're at the end of the topic like I originally thought. Using the count as a proxy for this works well enough for now, but may not in the future if we expose filters like channels/event types. Follow-on from #1363

We've got 3 places where we log errors in the poller loop and this one somehow wound up logging as `trace!` instead of `error!` like the rest. Follow-on from #1363

@svix-onelson

## What's Changed * Bridge: add `/events` poller by @svix-onelson in #1363 * server: Optimize all external host dependencies for faster rebuilds by @svix-jplatte in #1365 * Bridge: log svix client errors as error, not trace by @svix-onelson in #1368 * Bump Cargo lockfiles by @svix-jplatte in #1370 * server: add configuration to for changing service name on OpenTelemetry by @LeoZ100 in #1371 * build(deps): bump openssl from 0.10.64 to 0.10.66 in /bridge by @dependabot in #1372 * bridge: Require type field for kafka input, output by @svix-jplatte in #1374 * Bump OpenAPI spec and regenerate libs by @svix-lucho in #1375 ## New Contributors * @LeoZ100 made their first contribution in #1371 **Full Changelog**: v1.25.0...v1.26.0

svix-onelson force-pushed the onelson/bridge-events branch 4 times, most recently from 681bbf1 to ba93978 Compare July 9, 2024 22:31

svix-onelson commented Jul 10, 2024

View reviewed changes

bridge/svix-bridge/src/webhook_receiver/mod.rs Outdated Show resolved Hide resolved

svix-onelson force-pushed the onelson/bridge-events branch from ba93978 to dd40495 Compare July 10, 2024 05:07

svix-onelson changed the title ~~WIP: /events poller~~ Bridge: add /events poller Jul 10, 2024

svix-onelson marked this pull request as ready for review July 10, 2024 05:09

svix-onelson requested a review from a team as a code owner July 10, 2024 05:09

svix-onelson commented Jul 10, 2024

View reviewed changes

bridge/svix-bridge/src/config/mod.rs Outdated Show resolved Hide resolved

svix-jplatte previously approved these changes Jul 10, 2024

View reviewed changes

svix-onelson commented Jul 10, 2024

View reviewed changes

bridge/svix-bridge/src/main.rs Show resolved Hide resolved

bridge/svix-bridge/src/config/mod.rs Outdated Show resolved Hide resolved

bridge/svix-bridge/src/main.rs Outdated Show resolved Hide resolved

svix-onelson dismissed svix-jplatte’s stale review via 035d2e6 July 10, 2024 16:02

svix-jplatte reviewed Jul 10, 2024

View reviewed changes

bridge/svix-bridge/src/webhook_receiver/mod.rs Outdated Show resolved Hide resolved

svix-onelson added the open source review label Jul 10, 2024

svix-onelson force-pushed the onelson/bridge-events branch 4 times, most recently from c80d904 to 4bca290 Compare July 10, 2024 23:01

svix-onelson mentioned this pull request Jul 10, 2024

Libs: bump spec #1366

Merged

svix-onelson changed the base branch from main to onelson/spec-bump July 10, 2024 23:05

svix-onelson force-pushed the onelson/bridge-events branch from 4bca290 to 6f4f67c Compare July 10, 2024 23:12

Base automatically changed from onelson/spec-bump to main July 10, 2024 23:14

svix-onelson and others added 4 commits July 11, 2024 10:02

code review: update tracing attr spellings

5ccb055

Co-authored-by: Jonas Platte <158304798+svix-jplatte@users.noreply.github.com>

svix-onelson force-pushed the onelson/bridge-events branch from 82a1413 to feb4657 Compare July 11, 2024 17:06

Bridge: bump rust sdk to 1.25

6bfccbe

This gives access to the needed beta `events_subscription` API.

svix-onelson force-pushed the onelson/bridge-events branch from feb4657 to 5536e9a Compare July 11, 2024 17:07

svix-onelson requested a review from svix-jplatte July 11, 2024 17:11

svix-james approved these changes Jul 11, 2024

View reviewed changes

svix-onelson merged commit 269153d into main Jul 11, 2024
6 checks passed

svix-onelson deleted the onelson/bridge-events branch July 11, 2024 18:45

svix-onelson mentioned this pull request Jul 11, 2024

Bridge: log svix client errors as error, not trace #1368

Merged

svix-onelson mentioned this pull request Jul 12, 2024

Bridge: use batch count=0 to signal backoff, not "done" #1369

Closed

svix-lucho mentioned this pull request Jul 30, 2024

Bump version to 1.26.0 and update changelog #1376

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bridge: add `/events` poller #1363

Bridge: add `/events` poller #1363

svix-onelson commented Jul 9, 2024 •

edited

Loading

svix-jplatte left a comment

svix-onelson left a comment

svix-onelson commented Jul 10, 2024

svix-onelson commented Jul 11, 2024

Bridge: add /events poller #1363

Bridge: add /events poller #1363

Conversation

svix-onelson commented Jul 9, 2024 • edited Loading

svix-jplatte left a comment

Choose a reason for hiding this comment

svix-onelson left a comment

Choose a reason for hiding this comment

svix-onelson commented Jul 10, 2024

OSS Review: itertools

svix-onelson commented Jul 11, 2024

Bridge: add `/events` poller #1363

Bridge: add `/events` poller #1363

svix-onelson commented Jul 9, 2024 •

edited

Loading