New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Issues with seqNo and Eventhub Retention #408

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@Fokko
Copy link
Contributor

Fokko commented Nov 10, 2018

It feels like we're having some issues with the retention of the Event Hub.
When starting a Spark job that reads from a consumer group that already has some pruned messages
because then run out of retention.

java.lang.IllegalStateException: In partition 1 of eventhubdev, with consumer group consumergroup-1,
request seqNo 1921738 is less than the received seqNo 1924294.
The earliest seqNo is 1924293 and the last seqNo is 1944993

First I need to fix another issue with the Simulator, which I'll do in another PR to keep everything nicely separated. Currently, the latest seqNo is implemented by taking the size() of the messages within a certain partition. This is not right, we want to take the max of the seqNo of the actual messages:

Thanks for contributing! We appreciate it :)

For a Pull Request to be accepted, you must:

  • Run scalafmt on your code using the .scalafmt.conf present in this project
  • All tests must pass when you run mvn clean test

Just in case, here are some tips that could prove useful when opening a pull request:

  • Read the Contributor's Guide
  • Make the title of pull request is clear and informative.
  • There should be a small number of commits, all with informative messages.
  • The pull request shouldn't introduce any breaking changes (unless will occur on the next release)
  • Any public code should be properly documented
  • Be sure to write tests for any changes in the pull request
  • The code should build without any errors
@Fokko

This comment has been minimized.

Copy link
Contributor

Fokko commented Nov 10, 2018

Please merge #409 first

Issues with seqNo and Eventhub Retention
It feels like we're having some issues with the retention of the Event Hub.
When starting a Spark job that reads from a consumer group that already has some pruned messages
because then run out of retention.

java.lang.IllegalStateException: In partition 1 of eventhubdev, with consumer group consumergroup-1,
request seqNo 1921738 is less than the received seqNo 1924294.
The earliest seqNo is 1924293 and the last seqNo is 1944993

@Fokko Fokko force-pushed the Fokko:fd-write-test branch from a00aef4 to a25fd1f Nov 10, 2018

@sabeegrewal

This comment has been minimized.

Copy link
Member

sabeegrewal commented Nov 29, 2018

@Fokko sorry for the hold up, #409 is merged. Can you rebase when you get the chance? Also, is this ready for review?

as always, thanks for taking the time.

@sabeegrewal

This comment has been minimized.

Copy link
Member

sabeegrewal commented Jan 2, 2019

Hey, I'm going to close this for now. If you continue to have issues after the upcoming release, then we should revisit this PR.

@sabeegrewal sabeegrewal closed this Jan 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment