New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Javers fails under load in distributed applications #657
Comments
thanks for reporting. Indeed, there are problems with id generator for distributed apps. We need to invent sth better than RANDOM. Any suggestions? |
Hi Bartosz! Thanks for the reply. We have a couple suggestions/improvements, so:
-- |
That's how CommitIdGenerator.RANDOM works. You can use it but it would break Shadow queries
The main problem is -- ShadowQueryRunner sorts by commitId, so CommitIdGenerator must produce the monotonically increasing sequence |
Maybe we should use a DB sequence instead? |
The DB sequence might be a bottleneck for highly loaded applications (but this definitely very helpful for medium size systems), also this is not a good choice for MongoDB. Anyway we can make CommitIdGenerator pluggable and create Pull Request. Can we make ShadowQueryRunner to sort by, for example, creation timestamp instead of commit_id? Is the commit_id used for anything else except sorting? |
Pluggable CommitIdGenerator will add more complexity. What we need is good CommitIdGenerator implementation for distributed apps, and it should be available in JaVers out of the box. CommitId is used for sorting (by time) in Shadow queries (see I think about some mixed apporoach, majorId could be generated as in SYNCHRONIZED_SEQUENCE, so head +1 and minor Id as random, for example
This would give us almost perfect uniqueness and good time-based orderability. Should be good enough for Shadow queries. What do you think? |
Bartosz
For example, if there is entity com.app.Foo {id=123, name=”Bla-bla-bal”} then every commit might have the following ID:
This way we can have unique numbers for commits and provide good support of distributed mode. Thanks |
I'm not insisting on DB sequence, in fact SYNCHRONIZED_SEQUENCE is not using DB sequence, the implementation is DB independant. I like the idea to use commitDate (timestamp) for sorting. We already have it so we can us it. So my suggestion:
|
Seems like it will work. Unfortunately, I don't have enough knowledge about ShadowQueryRunner and Javers internals, so I'm afraid I cannot create right Pull Request. Could you please make these changes in the next weeks? |
Will try at weekend |
Thanks, Bartosz! |
the PR is ready https://github.com/javers/javers/pull/661/files |
ok, the fix is released in 3.9.1 |
Try it, and let me know if it works for you. |
Hi Bartosz! Thanks for the changes you've made. I found a couple issues regarding that:
|
Hi, So this commitId - |
Concerning the first issue, please provide a runnable test. I need to reproduce it on my side. |
I tested that on single app instance, not a cluster. 6624663025432992769.0 was generated by RANDOM generator, when I switched to SEQUENCE generator which increased the last commit id to 6624663025432992769.1 |
I don't understand why you are switching to SEQUENCE. The whole issue is about using RANDOM generator. |
Released in 3.9.1 Javers javers = javers().withCommitIdGenerator(CommitIdGenerator.RANDOM)
.build(); |
Hi Bartosz,
-- |
|
I've added a test - gbondarchuk9@75467e0, but it works. Will try to find a way to get it failed. What I do is: execute two commits inside one transaction (just one by one):
Either I modify one of commit dates by second (+/-), or do the same step as above under SEQUENCE generator - it works fine. |
So you are updating an object twice in the same millisecond? How Javers would know which snapshot is the last one? Remember that commits are sorted by commitDate when RANDOM is used. The whole algorithm will work as soon as you don't update on object from two different threads (or two javers instances) exactly in the same millisecond. |
Yes, I have such functionality that requires two object states (which is saved in the same second, not millisecond). Sure, Javers won't know which snapshot will be last and which will be first, but I supposed it had to return both snapshots with the same commitDate instead of one, as commitDate is used only for sorting, didn't I? I'm using single instance app, not a cluster. Is it possible to widen commitDate to use milliseconds? -- |
commitDate already has milliseconds precision |
Please let me know when you come up with a failing test with the reproduction of the issue you are writing about |
Hi Bartosz, As far as I know in MySQL to have fractional precision (milliseconds, microseconds), column type should be TIMESTAMP(1) and higher (TIMESTAMP(3) in our case, I think), but currently we have pure TIMESTAMP type for commitDate. -- |
could be, I've checked only Mongo and Postres. Feel free to contribute a PR with the fix here https://github.com/polyjdbc/polyjdbc/blob/master/src/main/java/org/polyjdbc/core/dialect/MysqlDialectTypes.java |
I've created PR |
MySql issue is moved here #664 |
I am getting this error while using Javers 3.6.3 with SQLServer and 2 JVMs which writes to same database |
@a-mujthaba321 start from bumping javers version to latest, and please don't comment on closed issues. Open a new one, if you are sure that you found a bug |
Javers fails under load in distributed applications during commit: org.javers.common.exception.JaversException: CANT_SAVE_ALREADY_PERSISTED_COMMIT: can't save already persisted commit '37140.0'.
We use MySQL database, and Spring integration module.
Seems like the following situation occurs:
_We saw that CommitIdGenerator contains both: SYNCHRONIZED_SEQUENCE and RANDOM (for distributed applications), but RANDOM generator is deprecated, do we have some alternative generator for distributed applications? What can we undertake to fix the issue?
Full Stacktrace
stacktrace.txt
The text was updated successfully, but these errors were encountered: