New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job activation can miss older jobs #927

Closed
ThorbenLindhauer opened this Issue Jun 5, 2018 · 4 comments

Comments

1 participant
@ThorbenLindhauer
Member

ThorbenLindhauer commented Jun 5, 2018

Steps to reproduce:

  1. Create job of type a
  2. Create job of type b
  3. Subscribe to type b and receive subscribed job
  4. Subscribe to type a

Current behavior:

  • The job of type a is never received

Expected behavior:

  • The subscriber receives the job

Reason:

  • The ActivateJobStreamProcessor instances interfere each other's reprocessing because they share the same producer ID
  • In particular, the stream processor for type a considers the events stream processor for type b produced when it searches for the position from which to continue

Issue discovered via #925.

@ThorbenLindhauer

This comment has been minimized.

Member

ThorbenLindhauer commented Jun 6, 2018

When detecting the position of the last source record in the stream for which a follow-up event was produced by the stream processor, we could make use of the record filter. I.e. only select a record as the source record, if its follow-up event matches the stream processor id AND the source event matches the record filter. The record filter must then include the job type.

@ThorbenLindhauer

This comment has been minimized.

Member

ThorbenLindhauer commented Sep 3, 2018

This issue blocks testing #1230.

@ThorbenLindhauer

This comment has been minimized.

Member

ThorbenLindhauer commented Sep 3, 2018

@menski I think the same problem applies when reprocessing with multiple exporters, and there the idea sketched in #927 (comment) does not work. Let's discuss this.

@ThorbenLindhauer

This comment has been minimized.

Member

ThorbenLindhauer commented Sep 4, 2018

We will go for a hackfix here, because we plan to change how job activation works in a fundamental way in the not-so-far future (towards a state query approach), where we will likely not have individual stream processors per job type anymore (and therefore no longer have this problem).

The hackfix is to generate the stream processor id based on job type's hash value.

ThorbenLindhauer added a commit that referenced this issue Sep 4, 2018

fix(broker): reliably activate jobs
- fixes an issue such that job activation would miss jobs if subscriptions
  were opened in a certain order
- fix is to make it very likely that different job activation processors
  receive different stream processor ids, so their produced records
  do not interfere during stream processing
- this is a hackfix because the ids are based on a hash function, which may
  have collisions; a proper solution is discussed in issue #927

@wafflebot wafflebot bot added needs review and removed in progress labels Sep 4, 2018

menski added a commit that referenced this issue Sep 4, 2018

fix(broker): reliably activate jobs
- fixes an issue such that job activation would miss jobs if subscriptions
  were opened in a certain order
- fix is to make it very likely that different job activation processors
  receive different stream processor ids, so their produced records
  do not interfere during stream processing
- this is a hackfix because the ids are based on a hash function, which may
  have collisions; a proper solution is discussed in issue #927

bors bot added a commit that referenced this issue Sep 4, 2018

Merge #1244
1244: fix(broker): reliably activate jobs r=menski a=ThorbenLindhauer

- fixes an issue such that job activation would miss jobs if subscriptions
  were opened in a certain order
- fix is to make it very likely that different job activation processors
  receive different stream processor ids, so their produced records
  do not interfere during stream processing
- this is a hackfix because the ids are based on a hash function, which may
  have collisions; a proper solution is discussed in issue #927

closes #927


Co-authored-by: Thorben Lindhauer <thorben.lindhauer@camunda.com>

bors bot added a commit that referenced this issue Sep 4, 2018

Merge #1244
1244: fix(broker): reliably activate jobs r=menski a=ThorbenLindhauer

- fixes an issue such that job activation would miss jobs if subscriptions
  were opened in a certain order
- fix is to make it very likely that different job activation processors
  receive different stream processor ids, so their produced records
  do not interfere during stream processing
- this is a hackfix because the ids are based on a hash function, which may
  have collisions; a proper solution is discussed in issue #927

closes #927


Co-authored-by: Thorben Lindhauer <thorben.lindhauer@camunda.com>

@bors bors bot closed this in #1244 Sep 4, 2018

@wafflebot wafflebot bot removed the needs review label Sep 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment