Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop using Kafka's Serde and add a real, composable Serializer/Deserializer #30

Closed
iravid opened this issue Sep 27, 2019 · 2 comments
Closed
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@iravid
Copy link
Member

iravid commented Sep 27, 2019

We should hardcode the consumer to K = Array[Byte] and V = Array[Byte], and provide data types that describe serialization/deserialization to and from byte arrays.

These data types should also be derivable from Kafka's existing Serde to ease migration.

As part of moving to these codecs, we should consider what interface to provide on the consumer for handling de/serialization failures. It's easy to just send a stream of eithers, but that is not very performant or ergonomic.

@iravid iravid added enhancement New feature or request help wanted Extra attention is needed good first issue Good for newcomers labels Sep 28, 2019
@svroonland
Copy link
Collaborator

I can see three desirable ways of handling deserialization failures:

  1. Fail the stream
  2. Skip the message
  3. Allow the client to handle it explicitly

1 should be easy (already the case?)
3 can be done by sending Eithers like you mentioned.
Maybe 2 can be implemented with some sort of onDeserializationError: DeserializationFailure => UIO[Unit] parameter on the plainStream and partitionedStream methods?

@iravid
Copy link
Member Author

iravid commented Oct 7, 2019

Resolved by #37. I'll open a follow-up for the failure handler

@iravid iravid closed this as completed Oct 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants