Javers fails under load in distributed applications

[gbondarchuk9]

Javers fails under load in distributed applications during commit: org.javers.common.exception.JaversException: CANT_SAVE_ALREADY_PERSISTED_COMMIT: can't save already persisted commit '37140.0'.

We use MySQL database, and Spring integration module.

Seems like the following situation occurs:

1. On the first application instance we called javers.commit(...) in transaction - it retrieved max commit_id from database and then increased it.
2. On the second application instance we also called javers.commit(...) in another transaction - it retrieved the same max commit_id as in step Initial diff #1 and also increased it by one. (first transaction is not committed yet)
3. First application instance committed transaction - data successfully saved to database.
4. Second application instance committed transaction - we got exception as above, as data with the same commit_id already exist in database after first transaction completion.

_We saw that CommitIdGenerator contains both: SYNCHRONIZED_SEQUENCE and RANDOM (for distributed applications), but RANDOM generator is deprecated, do we have some alternative generator for distributed applications? What can we undertake to fix the issue?

Full Stacktrace
stacktrace.txt

[bartoszwalacik]

thanks for reporting. Indeed, there are problems with id generator for distributed apps. We need to invent sth better than RANDOM. Any suggestions?

[gbondarchuk9]

Hi Bartosz!

Thanks for the reply.

We have a couple suggestions/improvements, so:

1. We propose to make CommitIdGenerator pluggable and available for passing appropriate implementation thru JaversConfiguration, so we can provide our own generator. It requires to make a general interface for generators. (Exactly the same approach as for ObjectAccessHook).
   So we would like to have the following structure:

public interface CommitIdGenerator {
   public CommitId nextId();
}

public class CustomCommitIdGenerator implements CommitIdGenerator {
   public CommitId nextId() {
      return ...;
   }
}

JaversBuilder builder = TransactionalJaversBuilder.javers()
...
.withCommitIdGenerator(new CustomCommitIdGenerator())
...

2. Create CommitId based on UUID or md5 hash of current timestamp + some unique identifier for application instance (it can be current thread number, process id, etc), and changed commid_id data type from decimal (22,2) to varchar(30), for example.

--
thanks

[bartoszwalacik]

That's how CommitIdGenerator.RANDOM works. You can use it but it would break Shadow queries

    /**
     * @deprecated RANDOM CommitIdGenerator is deprecated because it don't play along with Shadow queries. Only the default algorithm (SYNCHRONIZED_SEQUENCE) is supported.
     * @since 2.6
     */
    @Deprecated
    public JaversBuilder withCommitIdGenerator(CommitIdGenerator commitIdGenerator) {
    

The main problem is -- ShadowQueryRunner sorts by commitId, so CommitIdGenerator must produce the monotonically increasing sequence

[bartoszwalacik]

Maybe we should use a DB sequence instead?

[gbondarchuk9]

The DB sequence might be a bottleneck for highly loaded applications (but this definitely very helpful for medium size systems), also this is not a good choice for MongoDB. Anyway we can make CommitIdGenerator pluggable and create Pull Request.

Can we make ShadowQueryRunner to sort by, for example, creation timestamp instead of commit_id? Is the commit_id used for anything else except sorting?

[bartoszwalacik]

Pluggable CommitIdGenerator will add more complexity. What we need is good CommitIdGenerator implementation for distributed apps, and it should be available in JaVers out of the box.

CommitId is used for sorting (by time) in Shadow queries (see JaversExtendedRepository.getHistoricals()).
It should be monotonically increasing. Creation timestamp is not very good for that because you can have diverged clocks among servers.

I think about some mixed apporoach, majorId could be generated as in SYNCHRONIZED_SEQUENCE, so head +1 and minor Id as random, for example

1.7489099
2.3424455
3.4425523

This would give us almost perfect uniqueness and good time-based orderability. Should be good enough for Shadow queries. What do you think?

[sergey-derugo]

Bartosz
If you have diverged clocks among servers then most likely you have plenty of other problems so usually distributed application significantly relies on time synchronization.
Also using DB sequence introduces issues for highly loaded systems, in particular:

1. DB sequence might become a bottleneck for the hole system because all transactions constantly need query next value.
2. Some databases doesn’t support sequence, for example, MySQL has AUTO_INCREMENT option while Oracle support sequences.
3. Implementation of sequence might be tricky for the MongoDB because usually people prefer avoid synchronization mechanism and requirement about monotonic numbers makes the things much more complex.
   My suggestion is that we should define Commit Id as following:

	Major = “Entity Class @ Entity Id” or hash(“Entity Class” + ”Entity Id”)
	Minor = “select now()” or “current timestamp” (if NoSQL database is used)

For example, if there is entity com.app.Foo {id=123, name=”Bla-bla-bal”} then every commit might have the following ID:

	Major = com.app.Foo @ 123
	Minor = 23456789 

This way we can have unique numbers for commits and provide good support of distributed mode.

Thanks

[bartoszwalacik]

I'm not insisting on DB sequence, in fact SYNCHRONIZED_SEQUENCE is not using DB sequence, the implementation is DB independant. I like the idea to use commitDate (timestamp) for sorting. We already have it so we can us it. So my suggestion:

• un-deprecate CommitIdGenerator.RANDOM
• ShadowQueryRunner would use commitDate for sorting (instead of CommitId)
• maybe db index should be added on commitDate field

[gbondarchuk9]

Seems like it will work.

Unfortunately, I don't have enough knowledge about ShadowQueryRunner and Javers internals, so I'm afraid I cannot create right Pull Request.

Could you please make these changes in the next weeks?

[bartoszwalacik]

Will try at weekend

[gbondarchuk9]

Thanks, Bartosz!

[bartoszwalacik]

the PR is ready https://github.com/javers/javers/pull/661/files

[bartoszwalacik]

ok, the fix is released in 3.9.1
It will be available in a few hours in Central.
RANDOM generator is un-deprecated and there are changes in ShadowQueryRunner around commits sorting.

[bartoszwalacik]

Try it, and let me know if it works for you.