Commits on Nov 2, 2015

1. Context manager to acquire Postgres advisory locks …

   This Postgres feature is invaluable when dealing with synchronisations,
   especially when importing concurrently records from a system.
   
   When we export a record, we are able to acquire a lock on the exported record
   to prevent 2 jobs to export it at the same time. This is different when we
   import a record for the first time and with several jobs running in parallel,
   chances are high that 2 jobs will import the same record at the same moment.
   
   The Postgres advisory lock comes handy there for they allow to acquire an
   application lock.  Usually we'll acquire the lock at the beginning of an import
   (beginning of ``Importer.run()``) and we'll throw a ``RetryableJobError`` if
   the lock cannot be acquired so the job is retried later. The lock will remain
   in place until the end of the transaction.
   
   Example:
    - Job 1 imports Partner A
    - Job 2 imports Partner B
    - Partner A has a category X which happens not to exist yet
    - Partner B has a category X which happens not to exist yet
    - Job 1 import category X as a dependency
    - Job 2 import category X as a dependency
   
   Since both jobs are executed concurrently, they both create a record for category X.
   With this lock:
   
    - Job 1 imports Partner A, it puts a lock for this partner
    - Job 2 imports Partner B, it puts a lock for this partner
    - Partner A has a category X which happens not to exist yet
    - Partner B has a category X which happens not to exist yet
    - Job 1 import category X as a dependency, it puts a lock for this category
    - Job 2 import category X as a dependency, try to put a lock but can't, Job 2
      is retried later, and when it is retried, it sees the category X created by
      Job 1
   
   See http://topopps.com/implementing-postgres-advisory-locks/ for the article
   where I learned about the computation of the hash for this purpose.

   

   guewen committed Nov 2, 2015
   Copy the full SHA

   6e23779 View commit details

   Browse the repository at this point in the history