Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #252 kafka sink user error #476

Merged

Conversation

breezykermo
Copy link
Contributor

Closes #252 by communicating the error to the controller (following the pattern in the Kafka source).

When running in development with a basic job that plugs data from the nexmark source to a (badly configured) kafka sink, i.e.:

insert into badkaf
select process_string(person.name) from nex where person.name is not null;

(where process_string simply ensures a String is always returned, rather than an Option<String> -- as my badkaf sink expects a raw string.)

Arroyo seems to immediately attempt to go into a checkpoint, and eventually (after several minutes) times out through a failed self.flush. When this happens, the error is appropriately displayed in the UX in my setup:

pic-selected-240102-0422-39

I am not familiar enough with the lifecycle to understand how to test the error code path in self.process_element given that the setup described seems to immediately go into a checkmark. I would appreciate guidance on how to do this, and will be happy to update the PR accordingly.

I haven't added a test to catch this error/panic specifically, as from what I can tell the testing in connectors only confirms happy path.

Copy link
Member

@mwylde mwylde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, this looks great!

@mwylde mwylde merged commit 261a0bf into ArroyoSystems:master Jan 8, 2024
4 checks passed
@breezykermo breezykermo deleted the fix/252-kafka-sink-user-error branch January 8, 2024 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kafka sink should produce UserErrror when it cannot connect to cluster
2 participants