Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not drop batches. #231

Merged
merged 1 commit into from Nov 20, 2019

Conversation

@frankmcsherry
Copy link
Member

frankmcsherry commented Nov 19, 2019

Previously, if the advance frontier for a trace reached the empty set, indicating that no further requirements exist for correct accumulation, the trace would drop all batches. This has some problematic interactions with read_upper() and map_batches() which cannot tell that things are broken. It seems that there is no urgency to drop the batches, as they will be dropped the instant the last trace handle is dropped. If they are not dropped this way, it is because some weirdo is holding on to a defunct trace handle. I'm sure we will find out that I am such a weirdo, but it seems like the correct fix in that case is to drop the handle rather than monkey around with the implementation.

This also updates the logging for spine_fueled_neu.rs to correctly record batch drops when the trace is dropped.

cc: @umanwizard

Previously, if the advance frontier for a trace reached the empty set, indicating that no further requirements exist for correct accumulation, the trace would drop all batches. This has some problematic interactions with `read_upper()` and `map_batches()` which cannot tell that things are broken. It seems that there is no urgency to drop the batches, as they will be dropped the instant the last trace handle is dropped. If they are not dropped this way, it is because some weirdo is holding on to a defunct trace handle. I'm sure we will find out that I am such a weirdo, but it seems like the correct fix in that case is to drop the handle rather than monkey around with the implementation.

This also updates the logging for `spine_fueled_neu.rs` to correctly record batch drops when the trace is dropped.
Copy link
Contributor

umanwizard left a comment

Do we have a sense of what the memory impact would be in typical use?

@frankmcsherry

This comment has been minimized.

Copy link
Member Author

frankmcsherry commented Nov 19, 2019

My hope is that there is zero impact on "idiomatic" uses, meaning those in which mere moments after one expresses that they do not plan on reading from a trace they also drop the trace. It is very believable that there are non-idiomatic uses, including ones that make total sense and reveal that I am ret-conning "idiomatic" to mean "make sense to me at the moment".

@frankmcsherry

This comment has been minimized.

Copy link
Member Author

frankmcsherry commented Nov 20, 2019

Well, let's try it out and hear if there are any regressions.

@frankmcsherry frankmcsherry merged commit e6d7181 into master Nov 20, 2019
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.