ChangeSet applied twice #36
Comments
|
This seems like a problem in Liquibase itself (not the lock implementation). You'll notice these lines in pod #3: Either Liquibase should not release the lock between the read and execute, or it should reread from the DATABASECHANGELOG table. Are you able to upgrade to the latest version of Liquibase, to see if the problem is still present in the latest version? It would be nice to include an integration test for your scenario, but I also realize it might be hard to implement. |
|
Thank you for your answer, after some more testing, debugging and digging through Liquibase release notes, I think I found why the lock is released after readnig the For some reason without The releasing of the lock was fixed in Liquibase v4.4.1, so I upgraded our Spring Boot version to It was just a coincidence, that after adding the Thank you again for your answer, I'll close this issue. |
Last week I ran into the problem of
DATABASECHANGELOGLOCKgetting stuck, and a service failing to restart, because Liquibase times out waiting for the lock. The service is running on Kubernetes with multiple replicas, a replica was restarted by the system, which is normal in Kubernetes, and for some reason it terminated in the middle of checking Liquibase change log, leaving the lock in the DB, so after that no new replicas could start.I started to look for solutions to this, and this library seems like the perfect solution for the problem. I created a test scenario to reproduce the error I ran into. At first adding the dependency seemed to resolve the issue as expected, but on the second run a change set I added for testing that runs slow on purpose was applied twice by two different replicas.
Version info:
Spring Boot:
2.5.8Liquibase:
4.3.5liquibase-sessionlock:
1.6.3mysql-connector-java:
8.0.27MySQL:
5.7.42The test scenario:
3replicas, withRecreatestrategy, which means, all 3 replicas will start at the same time, and wait for the 3 replicas to start successfullyWhithout liquibase-sessionlock the lock in
DATABASECHANGELOGLOCKgot stuck, so no replicas could start successfully, so I could reproduce the problem.I deleted the slow change set from the
DATABASECHANGELOGtable, added the liquibase-sessionlock dependency, and deployed a new version.After adding liquibase-sessionlock when the replica that started the migration was terminated, the other two replicas tried to acquire the session lock, both saw that it was successful, both read the
DATABASECHANGELOGtable, then one of them returned to wait for the lock, the other applied the change set, and released the lock, the first one acquired the lock again, and applied the same change set, then released the lock, the third replica that started after the termination, acquired the lock, did not apply any changes and then released the lock.The on purpose slow change set:
The logs after adding liquibase-sessionlock:
The first replica, that was terminated:
The second replica, that took over first, and applied the change set:
The third replica, that applied the change set again:
The last replica, that was created, because of the termination:
Conclusion
I believe this library could be the ideal solution for the initial problem, but there seems to be a concurrency issue, and the lock can somehow be acquired by two replicas at the same time, and both can read the
DATABASECHANGELOGtable, to determine which change sets to apply.Any help is appreciated, I'll try to find the issue, and submit a PR, if I'm successful.
The text was updated successfully, but these errors were encountered: