Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQLMaxValueIncrementer is not multi-master replication-safe [SPR-14418] #18989

spring-projects-issues opened this issue Jun 29, 2016 · 1 comment
in: data type: enhancement


Copy link

@spring-projects-issues spring-projects-issues commented Jun 29, 2016

Tim Gokcen opened SPR-14418 and commented

The Spring subsystem is used by Spring Batch, among others, to generate increasing "unique" IDs.

However, is not safe to use in multi-master MySQL or MariaDB environments, e.g. master-master replication or MySQL/Galera cluster. This is because of the way in which it always kicks forward last_insert_id() by getCacheSize(), which will result in duplicate IDs across hosts and result in collisions if those IDs are used to, for example, insert rows into a table that has a unique key on the ID.

The global MySQL variable auto_increment_increment is available and used by clustering systems to ensure that AUTO_INCREMENT columns do not collide across the cluster (see the Galera documentation). The same value could also be used by MySQLMaxValueIncrementer to avoid collisions.

In multi-master environments, the cacheSize would also have to always be equal to 1; there is no way to pre-allocate IDs without guaranteeing that they were not already selected by another host.

Alternatively, the whole idea of using the backing RDBMS to generate increasing IDs could be abandoned in favour of generating non-linearly increasing "probably-unique" IDs, as for example in in Twitter Snowflake, which uses a combination of "timestamp + host-ID + thread-ID + counter" to produce a very-likely-to-be-unique 64-bit integer.

Affects: 4.2.6

Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Apr 3, 2018 commented

Spring Batch does not seem to provide support to inject/override custom implementations as they not annotated beans that you can override using @Primary at runtime. However, src/main/resources/data-source-context.xml already has externalized this class based on src/main/resources/ based on the ENVIRONMENT

<!--  Use this to set additional properties on beans at run time -->
<bean id="overrideProperties" class="org.springframework.beans.factory.config.PropertyOverrideConfigurer"
   <property name="location" value="classpath:batch-${ENVIRONMENT}.properties" />

We probably have to implement a custom class and override this properties file in classpath if I am not wrong. Worst case, we were able to fork and inject our own Incrementer to resolve this problem in our Galera cluster across DCs besides thinking about stopping replication completely.

@spring-projects-issues spring-projects-issues added in: data type: enhancement labels Jan 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
in: data type: enhancement
None yet

No branches or pull requests

1 participant