Clone this wiki locally
FileLock provides an easy way to ensure a single block of code runs at a given time on a given server (= no overlap), something which is often needed for ETL jobs. For instance:
- You may have long-running incremental data synchronisation jobs, which will store a
last_updated_attimestamp on disk. You would not want this timestamp to be modified by two concurrent jobs (or the data synchronisation would not work as expected).
- You may query a remote resource (server etc), for which you may need to avoid more than one connection at a given time.
FileLock solves this problem in the case you are running your tasks on a single server.
FileLock will protect you against:
- Cron jobs which may overlap (e.g. 1st run becomes long for some reason, 2nd run starts in the middle).
- Background jobs that may end up running concurrently in a way you didn't expect.
FileLock is not a distributed lock. A Postgres-based lock will be shipped in a next version of Kiba Pro for that purpose.
Currently tested against: MRI Ruby 2.4-2.7, Linux/Ubuntu, Mac OS X. Other platforms may work (please send inquiry for verification).
FileLock, wrap your ETL job run with this code:
require 'kiba-pro/middlewares/file_lock' Kiba::Pro::Middlewares::FileLock.with_lock(lock_file: 'tmp/process.lock') do job = prepare_job Kiba.run(job) end
When this code runs, the middleware will attempt to immediately acquire the lock on the file (creating it if it does not exist). If the acquirement is successful, the code in the block will run. Otherwise,
LockFailedException will be raised, and the code won't run.
This will cover against other processes locking against the same file, but also against threads in the same process, or forks of the same process.
A few important points must be noted:
Do not ever delete/unlink the lock file. Doing so will prevent
FileLockfrom working properly, as a race condition will be able to occur. If you really need to delete lock files, turn off all your processes and crons (e.g. maintenance mode), make sure nothing is running nor can be started, then proceed. See DON'T unlink section on this page for more background.
- If you use Capistrano-styled deployment, make sure to store your lock files in a folder which is shared between deployments (typically under
shared). The lock works on a given file, so if the file is not the same, the lock will not work!
This wiki is tracked by git and publicly editable. You are welcome to fix errors and typos. Any defacing or vandalism of content will result in your changes being reverted and you being blocked.