[Improvement] During batched import, offer to resume on failed file import

[dalgibbard]

Our Production DB is pretty big/fiddly (around 7million documents+), and when re-importing this, I've sometimes hit CouchDB errors halfway or so through - meaning I have to delete the DB, clean up the files, and start afresh. It would be useful if it detects failure, and offers the user to retry a few times (enabling the user to restart the DB or whatever as appropriate before retrying) before failing - when our imports are taking up to 3hrs to complete, a failure halfway means 1.5hrs of lost time... :(

[dalgibbard]

Have got this in my next pending release. Example output (captured by stopping couchdb!):

[root@server ~]# ./couchdb-backup.sh -R -H 127.0.0.1 -u admin -p pass -d db6 -f db.short.json
... INFO: Separating Design documents
... INFO: Duplicating original file for alteration
... INFO: Stripping _design elements from regular documents
... INFO: Fixing end document
... INFO: Inserting Design documents
... INFO: Successfully imported 0 Design Documents
... INFO: Small dataset. Importing as a single file.

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 1/3 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 2/3 - Retrying...

curl: (7) couldn't connect to host
... ERROR: Import of db.short.json failed.
See above for any errors

[dalgibbard]

Added the -a flag to allow for user config of the number of retries to perform

[dalgibbard]

Tested:

./couchdb-backup.sh -R -H 127.0.0.1 -u admin -p pass -d db6 -f db.short.json -a 10
... INFO: Separating Design documents
... INFO: No Design Documents found for import.
... INFO: Small dataset. Importing as a single file.

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 1/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 2/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 3/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 4/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 5/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 6/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 7/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 8/10 - Retrying...

curl: (7) couldn't connect to host
... WARN: Import of db.short.json failed - Attempt 9/10 - Retrying...

curl: (7) couldn't connect to host
... ERROR: Import of db.short.json failed.
See above for any errors

Value validation works also:

./couchdb-backup.sh -R -H 127.0.0.1 -u admin -p pass -d db6 -f db.short.json -a cheese
... WARN: Retry Attempt value of "cheese" is invalid. Disabling Retry-on-Error.
<SNIP>

./couchdb-backup.sh -R -H 127.0.0.1 -u admin -p pass -d db6 -f db.short.json -a 0
... WARN: Retry Attempt value of "0" is invalid. Disabling Retry-on-Error.
<SNIP>

[danielebailo]

I leave it open for a while just to remember we have to change documentation accordingly.

[dalgibbard]

Documentation has already been done... I just forgot to commit it! sigh I'll sort it in a little while :)

[dalgibbard]

All done!