fix(examples): Rewrite examples using the chain style API #147
Conversation
Rewrite the examples to use the chain style API. The billing example is migrated in a separate PR. Remove some of the helper files since they weren't being used outside of the example and weren't very complicated.
|
I couldn't get the typing to work for the |
| @dataclass | ||
| class Span: | ||
| span_id: int | ||
| trace_id: int | ||
| duration: int | ||
| timestamp: int | ||
|
|
||
| def to_dict(self) -> dict[str, int]: | ||
| return { | ||
| "span_id": self.span_id, | ||
| "trace_id": self.trace_id, | ||
| "duration": self.duration, | ||
| "timestamp": self.timestamp, | ||
| } | ||
|
|
||
|
|
||
| def build_span(value: Message[bytes]) -> Span: | ||
| """ | ||
| Build a Span object from a JSON str | ||
| """ | ||
|
|
||
| d: dict[str, Any] = json.loads(value.payload) | ||
|
|
||
| return Span(d["span_id"], d["trace_id"], d["duration"], d["timestamp"]) |
There was a problem hiding this comment.
Let's use the schema from sentry-kafka-schemas and a Parser rather than a custom event type.
There was a problem hiding this comment.
I thought we were trying to move away from sentry-kafka-schemas?
There was a problem hiding this comment.
Yes, but the replacement will have an interface that will be very close to that one. It should be a quick swap, we can use it till we replace it.
Yeah it seems the flatMap implementation is just plain broken and we are noticing it only now because the old The issue is the following:
So I think there are two fixes to make. And I would do them in a separate PR:
What do you think ? |
|
@fpacifici It mostly makes sense, just to clarify:
This is confusing me a bit, I thought FlatMap was allowing a 1->N style of processing. So one message in can produce one or more messages out. If this is the case I agree that it shouldn't enforce a Batch style input. I'm reluctant to have the output be a Generator however. I would think we'd want a more concrete data structure, in case the next step is written in Rust. |
|
I removed the changes to FlatMap from this PR, I'll migrate that example in a separate PR. |
We still generally wrap the function provided by the user into more python code before running rust. Example: https://github.com/getsentry/streams/blob/main/sentry_streams/sentry_streams/adapters/arroyo/rust_arroyo.py#L217-L235 This means we can still have an abstract structure like an Iterable and materializing it into something rust would understand. |
|
@fpacifici Could you take another look at this? The flatmap changes went into a separate PR, and I'd like to get this merged soon so it doesn't conflict too much with #152 |
See my comment on |
Rewrite the examples to use the chain style API. The billing example is migrated in a separate PR. Remove some of the helper files since they weren't being used outside of the example and weren't very complicated.
Also skip the
alerts.pyfile since it uses a FlatMap, and that has to be implemented in a separate PR.