Skip to content

Conversation

@stankiewicz
Copy link
Contributor

UnboundedReader is controlled with UnboundedReaderMaxReadTimeSec and UnboundedReaderMaxElements.
For sub second latency user has to set UnboundedReaderMaxElements to 1 which is not flexible as there may be multiple situation where there are more elements.

In this change UnboundedReaderMaxReadTimeSec becomes double. Change is backward compatible.
Unlike #31036, this doesn't introduce additional flag.

@github-actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey added as fallback since no labels match configuration

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey
Copy link
Contributor

CC: @kennknowles

void setReaderCacheTimeoutSec(Integer value);

/** The max amount of time an UnboundedReader is consumed before checkpointing. */
@Description(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioning that the representation can be in fractions of seconds is probably a good call-out here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. added additional comment.

new Duration(beforeReading, afterReading).getStandardSeconds(),
lessThanOrEqualTo(maxReadSec + 1));
new Duration(beforeReading, afterReading).getMillis(),
lessThanOrEqualTo((long) (maxReadSec * 1000L)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're short a second on this comparison looking at the old unit test, although I do not know if the increased resolution to milliseconds renders the +1 second useless or not

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch. fixed.

Copy link
Contributor

@jrmccluskey jrmccluskey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jrmccluskey jrmccluskey merged commit bb310e7 into apache:master Apr 18, 2024
jrmccluskey added a commit that referenced this pull request Apr 23, 2024
jrmccluskey added a commit that referenced this pull request Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants