Skip to content

Forcefully emit batches on input timeout error #916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Jun 12, 2020
Merged

Forcefully emit batches on input timeout error #916

merged 11 commits into from
Jun 12, 2020

Conversation

dominiklohmann
Copy link
Member

Note: This does not fix the underlying issue completely, but is a decent workaround that improves the latency between source and importer.

@dominiklohmann dominiklohmann added the bug Incorrect behavior label Jun 11, 2020
@dominiklohmann dominiklohmann force-pushed the topic/force-emit-batches branch from d935836 to d8d5731 Compare June 11, 2020 09:25
@dominiklohmann dominiklohmann force-pushed the topic/force-emit-batches branch from d8d5731 to 7bb3866 Compare June 11, 2020 09:59
@dominiklohmann dominiklohmann force-pushed the topic/force-emit-batches branch from ea42b26 to af00012 Compare June 11, 2020 13:09
@dominiklohmann dominiklohmann marked this pull request as ready for review June 12, 2020 08:17
lava
lava previously requested changes Jun 12, 2020
View reviewed changes
Copy link
Member

@lava lava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good; the change request is for printing an error on parser failure, the other items are mostly discussion points.

tobim
tobim requested changes Jun 12, 2020
View reviewed changes
Copy link
Member

@tobim tobim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the time being this is a reasonable workaround, I hope we'll be able to revert the changes to the source in the future.

@dominiklohmann dominiklohmann requested review from lava and tobim June 12, 2020 10:02
@dominiklohmann dominiklohmann force-pushed the topic/force-emit-batches branch from b7f8027 to e69d3c1 Compare June 12, 2020 10:18
tobim
tobim requested changes Jun 12, 2020
View reviewed changes
@dominiklohmann dominiklohmann force-pushed the topic/force-emit-batches branch from e69d3c1 to 6578ed3 Compare June 12, 2020 10:44
@dominiklohmann dominiklohmann requested a review from tobim June 12, 2020 10:45
tobim
tobim approved these changes Jun 12, 2020
View reviewed changes
Copy link
Member

@tobim tobim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we should have an integration test for this, but we need to add parallel execution capabilities to the testing tool first.

@dominiklohmann dominiklohmann merged commit dbf1a79 into master Jun 12, 2020
@dominiklohmann dominiklohmann deleted the topic/force-emit-batches branch June 12, 2020 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mavam mavam mavam left review comments

@tobim tobim tobim approved these changes

@lava lava Awaiting requested review from lava

Assignees
No one assigned
Labels
bug Incorrect behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dominiklohmann @lava @mavam @tobim