Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not drop batches. #231

Merged
merged 1 commit into from Nov 20, 2019
Merged

Do not drop batches. #231

merged 1 commit into from Nov 20, 2019

Conversation

frankmcsherry
Copy link
Member

Previously, if the advance frontier for a trace reached the empty set, indicating that no further requirements exist for correct accumulation, the trace would drop all batches. This has some problematic interactions with read_upper() and map_batches() which cannot tell that things are broken. It seems that there is no urgency to drop the batches, as they will be dropped the instant the last trace handle is dropped. If they are not dropped this way, it is because some weirdo is holding on to a defunct trace handle. I'm sure we will find out that I am such a weirdo, but it seems like the correct fix in that case is to drop the handle rather than monkey around with the implementation.

This also updates the logging for spine_fueled_neu.rs to correctly record batch drops when the trace is dropped.

cc: @umanwizard

Previously, if the advance frontier for a trace reached the empty set, indicating that no further requirements exist for correct accumulation, the trace would drop all batches. This has some problematic interactions with `read_upper()` and `map_batches()` which cannot tell that things are broken. It seems that there is no urgency to drop the batches, as they will be dropped the instant the last trace handle is dropped. If they are not dropped this way, it is because some weirdo is holding on to a defunct trace handle. I'm sure we will find out that I am such a weirdo, but it seems like the correct fix in that case is to drop the handle rather than monkey around with the implementation.

This also updates the logging for `spine_fueled_neu.rs` to correctly record batch drops when the trace is dropped.
Copy link
Contributor

@umanwizard umanwizard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a sense of what the memory impact would be in typical use?

@frankmcsherry
Copy link
Member Author

My hope is that there is zero impact on "idiomatic" uses, meaning those in which mere moments after one expresses that they do not plan on reading from a trace they also drop the trace. It is very believable that there are non-idiomatic uses, including ones that make total sense and reveal that I am ret-conning "idiomatic" to mean "make sense to me at the moment".

@frankmcsherry
Copy link
Member Author

Well, let's try it out and hear if there are any regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants