aptly 0.9.5 "ERROR: can't open database: resource temporarily unavailable"

[dankegel]

I'm running a bunch of instances like this (context is stress testing unrelated part of CI system;
each build slave ssh's to the apt repo machine and runs aptly to suck the new debs in):

aptly repo add -force-replace=true -remove-files=true repo-buildhost4.oblong.com-xenial-dev incoming-unique.tmpdir.13432

I think I noticed that aptly doesn't like multiple instances running, so It's wrapped with a shell script to not run when any other aptly process is present. The shell script is racy and sometimes lets more than one instance through, and aptly fails with:

Loading mirrors, local repos, snapshots and published repos...
ERROR: can't open database: resource temporarily unavailable

Is this expected? Isn't aptly supposed to do its own locking?

(Also, oddly, when this happens, aptly doesn't seem to exit with nonzero status, so my script doesn't notice the problem.)

[smira]

I think aptly exit code should be non zero, but the error itself is quite expected: aptly does locking on DB and fails to get it.

In general, I would recommend running aptly in API mode for cicd.

[katcaola]

This sounds like database contention.

I've encountered this several times when I use Jenkins to update various debs and push to aptly. I only have one aptly instance and when jobs try to update aptly at the same time, they fail because aptly only lets the job that got to it first have the mutex. The solution was to have one job that consumes artifacts of all jobs and pushes to aptly so that way it's one job continually using aptly and not multiple racing for access.

Having multiple slaves try to ssh and edit aptly at the same time might be causing your problem.
If your script is racy, that is also a contributor.

[smira]

We could have tried to add an option to try acquiring DB lock for some time before giving up, should be relatively easy. That way we could avoid simple failures like that. Thoughts?

[katcaola]

As long as the time allowance for trying to access the DB can be set manually, this could work.

The only problem I can see is with timing is if the timeout isn't long enough. In the Jenkins example I mentioned, the current deb and package library is extremely large (I'm currently in the middle of coming up with solutions to purge and make it smaller) and so if you do any kind of update to aptly or the db, it takes upwards of tens of minutes (especially aptly db cleanup). If the time to acquire the lock is preset and not able to be set manually, this could prove problematic, as "amount of time to keep trying" varies based on the problem trying to be solved. Some people are fine with having the attempts last for 45 minutes, others are not. Flexibility is always a plus.

[smira]

Yes, I'm looking towards something like -db-retry-lock=10m flag available for all the aptly commands (or it could go to config as well).