-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expected to find job with key, but no job found #6016
Comments
On a first look, this looks like a bug. If it is not, then we should lower the log level. |
Duplicate of #3765 |
I don't think this is duplicate of #3765. |
I would categorize it as a bug. I couldn't find a case where this is expected to happen. But I would assign severity low as the error went away shortly and we did not observe any other issues related to it. |
This occurred on the gameday cluster, so postponed until tomorrow after the game day in case it's related. Let's not solve the incident before the gameday occurs 😅 |
Not sure it is related to the game day. It is not deliberate. I can tell that much Please wait for Game Day. After Game Day I can also give full sources, which might help in diagnosing the problem |
So from the client side there is nothing out of the ordinary. I think this is a Zeebe-internal phenomenon. Would be interesting to see a stack trace, in particular whether it is called from |
Please verify whether we're creating garbage (i.e. leaving dangling references) or we're actually "losing" jobs or inserting the wrong references. The latter is high priority, but the former isn't. In case of doubt I would assume the worst, and then we can verify and reprioritize (or just fix it if the cause is obvious at that point). |
Unfortunately there is not stack trace available to determine if it is called from forEachTimedOutEntry or forEachActivateableJob. |
As a user, Peter did not observe any issues. We also don't see any stuck workflows in Operate. So I think we can safely assume that the severity is low. |
Occurred in the last CW09 benchmark: https://console.cloud.google.com/errors/CKe9pf-Tqe-ZZw?service=zeebe-broker&version=medic-cw-09-cf1627167-benchmark&time=P1D&project=zeebe-io Seems to occur in bursts, but the cluster seems healthy and the workers aren't reporting any errors. |
I'm closing for now as we haven't noticed any negative effects, so I wouldn't invest time here at the moment. |
We are experiencing this on 8.3.3 Was it ever discovered what this is about? Should we be worried (as in silent job loss)? |
This can happen normally if your jobs time out while another worker is working on them. So for example, I have a job of type This is not an error from Zeebe's point of view because Zeebe is an at-least-once system. |
I see, thank you. It might be consistent with the fact that we have attempted to run some tests with a high value for |
Bumps [org.apache.maven.plugins:maven-javadoc-plugin](https://github.com/apache/maven-javadoc-plugin) from 3.6.2 to 3.6.3. - [Release notes](https://github.com/apache/maven-javadoc-plugin/releases) - [Commits](apache/maven-javadoc-plugin@maven-javadoc-plugin-3.6.2...maven-javadoc-plugin-3.6.3) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-javadoc-plugin dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Describe the bug
Version 0.25.3
Report location: visitJob (JobState.java:257)
logs
Environment:
The text was updated successfully, but these errors were encountered: