Fixed-size: return each batch to at most one current-batch query. #2391
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Part of DAP-09 implementation (#2389). With this change, Collectors are expected to remember the collection job IDs they generate before sending the collection request. If the Collector encounters a transient failure, it can then resend the request (with the same collection job ID), and rely on request idempotency for things to work out. The upside of this change is that Collectors no longer need to have logic to deduplicate multiple current-batch requests that are mapped to the same underlying batch, and they can generate current-batch requests at higher rates without risking more duplicated collections.
|
(we should wait to merge this until we have created a DAP-07 branch & main is for DAP-09, unless we want to backport this change to DAP-07) |
|
I think whether we want to backport depends on whether we want our WIP stateful collector to support DAP-07. This change is necessary for it to be optimally implemented #2254 (comment). The changeset looks simple enough to backport (i.e. merge as-is), although I still am giving it a close read. |
inahga
approved these changes
Jan 5, 2024
|
We should take this once #2468 lands |
12 tasks
|
We've now branched for |
inahga
pushed a commit
that referenced
this pull request
Jan 18, 2024
) Part of DAP-09 implementation (#2389). With this change, Collectors are expected to remember the collection job IDs they generate before sending the collection request. If the Collector encounters a transient failure, it can then resend the request (with the same collection job ID), and rely on request idempotency for things to work out. The upside of this change is that Collectors no longer need to have logic to deduplicate multiple current-batch requests that are mapped to the same underlying batch, and they can generate current-batch requests at higher rates without risking more duplicated collections.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of DAP-09 implementation (#2389). With this change, Collectors are expected to remember the collection job IDs they generate before sending the collection request. If the Collector encounters a transient failure, it can then resend the request (with the same collection job ID), and rely on request idempotency for things to work out.
The upside of this change is that Collectors no longer need to have logic to deduplicate multiple current-batch requests that are mapped to the same underlying batch, and they can generate current-batch requests at higher rates without risking more duplicated collections.