OptimisticLockingFailureException updating step execution after commit failure #1826
Comments
|
Dave Syer commented You need to flush JPA changes before Spring Batch tries to commit its own meta data. The best way to do that is in the ItemWriter. Any reason you can't do that? |
|
Quinton McCombs commented We are already using the JpaWriter supplied with Spring Batch which performs the flush. We have a requirement to audit all database changes and identify the changes made during a single transaction. For this reason, we implemented our auditing using an Eclipselink pre-commit hook (Hibernate can do this as well). While the transaction level auditing does not make much sense for a batch job, it is what is used by the rest of the application. |
|
Dave Syer commented If you are using transaction hooks you are going to have to tread very carefully. You need to somehow ensure that the vendor hook runs before the Spring transaction synchronization, or else not use it. The easiest way would be to run your audit code in an ItemWriter or ItemWriteListener, but if you find a cleverer way and it can be exposed as a framework feature, please let us know. If you can provide a sample project (preferably self-contained, with an in-memory database, and built using maven, but we'll take whatever you have), that would help. Or if you want to make a suggestion about a patch you can fork on github and send a pull request (the README has more information). |
|
Ramkumar Kirhsnan commented We've also got the same failure after the one of the steps failed to commit due to database exception thrown while writing into the database. Following is the stack trace when i tried to run the job for only one item |
|
Dave Syer commented I think that problem is unavoidable if you don't flush the Hibernate session in your ItemWriter. You have made life complicated by using a partition step, but the same failure mode is easy to see in a single threaded normal step. Can you just do the flush? |
|
Kristof Buts commented maven project for reproducing the bug |
|
Kristof Buts commented In our current project, we are also experiencing this problem in some of our batches, but until yesterday we couldn't reproduce it. So, what did I figure out? Consider the following scenario for a batch running 3 threads: So, the problem is that copying the oldVersion variable is done outside the synchronized block. By putting it inside the block (say at line 407 instead of line 380, just after setting lock to true), I think the bug would be solved (I already tested this by debugging my test with a conditional breakpoint in eclipse which called this code at the right place and it worked). Is it possible to get this fix in a patched version for spring-batch-core-2.1.8.RELEASE? We cannot upgrade to a higher spring version since we are only one month away from a big release. Kind regards, |
|
Kristof Buts commented Added the scenario in a more readable form |
|
Philippe Mouawad commented Hello, Is there some plan to take into account what Kristof Buts. Stacktrace: |
|
Philippe Mouawad commented Please find attached the spring batch classes put in INFO and DEBUG fororg.springframework.batch.core.step.tasklet. As you can see before failure we have this: |
|
Philippe Mouawad commented I have tested with last 2.2.1 and I have the same problem. |
|
Philippe Mouawad commented Hello, Hope you can take it into account soon. |
|
Philippe Mouawad commented Hello, Could you shed some light on this ? Thank you |
|
Michael Minella commented I added a comment about the pull request. In short, moving the copy to where it is won't work because the StepExecution could have already been modified in the step. There are a couple tests that are failing with that update that illustrate this. With regards to why the copy method restores the version number...why wouldn't it? The intent is that there was a rollback so the state should match what was there before the rollback. |
|
Philippe Mouawad commented Thanks for feedback. Further testing this morning on real world project showed me fix did not correct all cases, my real world case is still broken although attached test case was fixed by it. Does it mean you confirm this bug is a real one ? Test project attached reproduces it and my real world also does. It seems anyway that the stepexecution is modified somewhere before restoring which generates the case. |
|
Kristof Buts commented
|
|
Gérald Quintana commented We has this OptimisticLockingFailureException, when we noticed this log just before: "Commit failed while step execution data was already updated. Reverting to old version." . Turning org.springframework.batch.core.step.tasklet and org.springframework.transaction loggers level to debug helped us find a bug in our tasklet code. In our case:
|
|
zyro commented i think i am currently facing this (or at least sth. similar) in the following scenario:
|
|
member sound commented Is this issue still active? I'm also getting the OptimisticLockFailureException. The cause for me was an insert on a column whose length was not big enough for the inserted data, thus throwing SQL Exception and thereby the rollback: |
|
Mohamed commented Hi, Is there any way to discover the origin of this error? |
|
Adam commented Same thing as for |
|
Gérald Quintana commented
|
|
Mohamed commented Hi, Thanks in advance |
|
Gérald Quintana commented
Having logs of commit failure would help, but it's not the case. |
|
David J. M. Karlsen commented Any movement on this issue? |
|
Sarvesh Katariya commented Any updates/workaround on this issue ? |
|
Marcus Mattern commented We did a simple workaround for us to avoid this problem. {{
}} Since then our problem was gone. Marcus |
|
member sound commented Any updates on this fix? I got the same problem... |
|
James Home commented
Regarding the comment: "With regards to why the copy method restores the version number...why wouldn't it? The intent is that there was a rollback so the state should match what was there before the rollback." The thing is, it's not actually a rollback in the database sense. You're really just resetting the counts the their previous values. It's a new update in a db sense, so surely it should use the appropriate version number. The policy of 'rolling back' the version number in a db update seems to be incompatible with the principle of version based optimistic locking that's being used by the dao. |
|
zyro commented following the suggestion to flush the session in the writer, i think there is still a (slight..) chance for an exception being thrown (e.g. |
|
I am also getting |
|
I got the same exception |
|
No, I opened a PR with my attempt to fix the problem, see #591. I believe this is the most daunting issue in the backlog at the moment. The PR is not merged yet because we need to test the changes in depth. I added 3 integration tests to cover the case (which are passing without regressions) but I want to gather feedback from the community before applying the fix. If you want to help moving things forward, I invite you to give the PR a try and share your feedback. |
|
Is this issue resolved now : https://stackoverflow.com/questions/53959121/spring-batch-exception-handling ? |
|
benas - Your PR still in Open status . When can we expect this fix ? even we are also getting same error . org.springframework.batch.core.step.FatalStepExecutionException: JobRepository failure forcing rollbackrr - Job[at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:463)r at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)r at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)r at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271)r at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)at org.springframework.batch.repeat.support.TaskExecutorRepeatTemplate$ExecutingRunnable.run(TaskExecutorRepeatTemplate.java:262)r at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:268)r at java.lang.Thread.run(Thread.java:748)rCaused by: org.springframework.dao.OptimisticLockingFailureException: Attempt to update step execution id=4465436 with wrong version (1), where current version is 2r at org.springframework.batch.core.repository.dao.JdbcStepExecutionDao.updateStepExecution(JdbcStepExecutionDao.java:256)r at org.springframework.batch.core.repository.support.SimpleJobRepository.update(SimpleJobRepository.java:191)r at sun.reflect.GeneratedMethodAccessor853.invoke(Unknown Source)r at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)r at java.lang.reflect.Method.invoke(Method.java:498)r at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)r at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)r at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)r at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)r at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)r at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)r at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)r at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)r at com.sun.proxy.$Proxy70.update(Unknown Source)r at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:456)r ... 7 common frames omittedr |
|
@snagunoori As mentioned in my previous comment, the fix will be merged when we gather enough feedback from the community. For now, only one person tested the fix and gave feedback. If you want to help moving things forward, I invite you to give the PR a try and share your feedback as well (on the issue related to the PR, not here). |
|
@benas We are also hit by the this exception, today. If you provide me the exact version to take for testing of it then we can also provide the feedback for this fix. |
That would be really appreciated! The fix is in #591 , it is not released so I can't give you an exact version to try out. You can apply the fix to the master branch, build the project and import the snapshot version in your project for testing. Please note that fixing the problem does not mean the Thank you upfront. |
|
Hi @benas is there any help? |
|
hi @benas when i commented the code works successfully and the status of job became Completed and you know the exception give a sense of meaning as there is a concurrent access to locked object which i think the [StepExecution] as i tried to update it and in the same time my stop jobOperator.stop(jobExecution.getId()); try to update it also, am i right ? |
Hi, at my workplace we have seen this issue now and then over the last few years (currently on Spring 4.3.29, Spring Batch 3.0.10), but usually it has not been consistently present or we have worked around it. Last week it started to hit all the time, so I tried a patched version today - with success! |
fmbenhassine
added
the
has: backports
fmbenhassine
removed
the
has: backports
fmbenhassine
changed the title
|
hope this issue can be fixed |
Hello, Could you consider to release this "soon" ? That would allow us to migrate from 3.0.10 to 4.X |
|
my 2c worth, when working on a mainframe using IMS, that also uses the concept of transactions (syncpoints), we had a similair system that we wrote inhouse to manage restartability. Instead of using a database(table) for writing the job information, which causes all sorts of issues because it participates in the syncpoint, we wrote the job information to a flat file which is associated with the job. This keeps the job restart data outside the (syncpoint) and simplifies the management of the job restart data. |
|
Adding ".faultTolerant()" to the StepBuilder helped here. |
Quinton McCombs opened BATCH-1767 and commented
It appears that if the commit fails, spring batch will get an OptimisticLockingFailureException when it tries to revert the changes to the step execution. In my particular case, I have a callback through EclipseLink to update history tables before a transaction is committed. If a failure occurs during this callback, the commit fails.
From looking through the code and the attached log file, the step execution is updated and committed before the main transaction is committed. When the commit fails, the old values for the step execution (including version) are updated to the values before the chuck started. When control returns to AbstractStep.execute(), the OptimisticLockingFailureException is thrown when the step execution is updated with the failed status because the new version had already been committed to the database.
Affects: 2.1.7
Attachments:
Referenced from: pull request #591
23 votes, 31 watchers
The text was updated successfully, but these errors were encountered: