Backup & Restore database with docs with binary attachments

[Telephonic]

Great work on these scripts! Thank you for writing these and sharing. It's great to be able to just grab a copy of a whole CouchDB database.

Did you design the scripts in mind to handle binary attachments? We seem to able to download a full database that includes binary attachments. The backup file and the original database size are similar. However, when I try to restore this database dump with binary files, I get the following error:

{"error":"bad_request","reason":"invalid UTF-8 JSON"}

Have you come across the above error before?

I have been able to restore pure JSON (non binary) databases without any trouble.

Thank you!

Graham
CTO, Telephonic

[danielebailo]

Hi Graham,
I've not tested the script with binaries.
I know that it has some bug when the database contains strings with the " character, but I didn't have time to test it yet.

Bynaries,if I'm not wrong, are usually attached to the object as strings of chars.
Can you make a test, e.g. creating a database with a couple of docs, one with an attachment, and then donwload it?
If you send me the output, I could then do some work on this test case.
best
Daniele

Il giorno 26/set/2013, alle ore 20:48, Telephonic ha scritto:

Great work on these scripts! Thank you for writing these and sharing. It's great to be able to just grab a copy of a whole CouchDB database.

Did you design the scripts in mind to handle binary attachments? We seem to able to download a full database that includes binary attachments. The backup file and the original database size are similar. However, when I try to restore this database dump with binary files, I get the following error:

{"error":"bad_request","reason":"invalid UTF-8 JSON"}

Have you come across the above error before?

I have been able to restore pure JSON (non binary) databases without any trouble.

Thank you!

Graham
CTO, Telephonic

—
Reply to this email directly or view it on GitHub.

[danielebailo]

This issue should be solved.
Can you check it?

[danielebailo]

This issue should be solved.
Can you check it?

[Telephonic]

Hi Daniel,

I'm very sorry for the slow reply here.

I've tested again and the attachments aren't included when using couchdb-dump.sh. There is no binary string within the downloaded file.

Were you able to get this to work in your tests?

thanks,
Graham

[danielebailo]

Dear Graham,
unfortunately at the moment I've no time to work harder on this project.
Sorry for this
best
Daniele

Daniele

http://www.danielebailo.it/ http://danielebailo.wordpress.com/

2014-08-17 0:59 GMT+02:00 Telephonic Communications Inc. <
notifications@github.com>:

Hi Daniel,

I'm very sorry for the slow reply here.

I've tested again and the attachments aren't included when using
couchdb-dump.sh. There is no binary string within the downloaded file.

Were you able to get this to work in your tests?

thanks,
Graham

—
Reply to this email directly or view it on GitHub
#2 (comment)
.

[dalgibbard]

FYI; I looked at this issue whilst contributing some other fixes; the main issue is that requesting the docs output doesn't 'print' the binary content; example:

{"docs":[
{"_id":"4685ca2ba841112ed63ba386c0001e56","_rev":"2-e1432ed19858cdbc8b5c0c0bbf38f76a","value":"this is my binary document","_attachments":{"esxi.tgz":{"content_type":"application/x-compressed-tar","revpos":2,"digest":"md5-K1F3CiyZT2AjIaR3n7dFSQ==","length":1697995,"stub":true}}}
]}

In order for these to be managed too, I suspect you'd need to scan your entire output for references to attachement and then, programatically download the binary separately.
Then, during restore, you'd again need to monitor for import of the document, strip the attachment stanza, and post-successful-import, push the binary file to the document again. (If you try and import the document without stripping the attachement segment, it will fail with "missing_stub" as the referenced file doesn't exist)

It's a bit long winded, and unfortunately out of scope for the changes that i've been looking to implement.

[danielebailo]

fixed here #8