support for blob or binary attachments

I think it is essential to be able to store raw binary data either as a blob or attachments (as couchdb does).
The current workaround is to Base64 encode which actually bloats the size with aprox. 130%.
You end up having to shuffle more bytes and perform additional decoding which wastes cpu cycles (and energy).
So please add support for attachments or other means to store binary data in the database.

[luebbert42]

Hi kaerus, I am helping the team with mostly non-technical tasks, e.g. I am collecting use cases for ArangoDB and extract requirements for the roadmap. Can you tell me a bit more what you do with ArangoDB so that we can find a solution for you? You can reach me here or via dorthe@arangodb.org. Thanx.

[fceller]

What kind of binary data do you have in mind regarding the size?

Do you have any opinion/ideas on how the Server / Client protocol should look like? JSON has no support for binary data as far as I can see. If using attachments, one could forgo any encoding and send the binary data in the HTTP body. If using blobs within a document, I have no idea how to encode this.

Hi, thanks for the quick reply. My concerns are mainly with compressed binary data such as images but also other file formats such as PDF etc. However I would like to have the option to compress arbitrary content (in a CMS system I'm currently developing) since many web clients today support compressed content (accept-encoding: gzip, deflate). Compression, in my case, increases the overall performance, especially when accessed over slow or long latency networks. In short, being able to store and retrieve data in a raw binary format would make the user experience better in my case.

fceller: Regarding the protocol, I was referring to couchdb attachments for that specific reason. I have not put any more thought if that is the optimal way to do it, but it is atleast simple to interface with.
But perhaps BSON could be a better alternative http://bsonspec.org/

Size of data varies depending on usage, but anything from small jpegs (4KB) to streamable movies (1GB) I guess. :)

Thinking about my use case a little more, smaller blob's (< 16KB) could be embedded as base64 encoded data.
Larger files could be wired as multipart binary content and stored as document attachments. Another option is to store attachments directly on disk and only maintain references in the database. However I prefer to have everything in one place for simplicity.

fceller: What is your take on this. Is it better to keep things as they are and workaround binary attachments with alternate storage i.e on local disk or NAS or is it feasible/practical to store binaries directly in arangodb?
I don't know how this could affect memory utilization e.t.c in the db, perhaps a simple pro vs con list from your side would clear things up regarding this matter.

[fceller]

I've to think about this: Storing very large files in the database seems will in principle work, but might not be as performant as storing such files on a local disk and only store the file-name.

On the other hand, I might be nice to just have one server to which a client can talk to.

I need to check how couchdb attachments work.

The more I think about this the more I lean towards just keeping references to files in arangodb. Why?
Since my users could just as well store their attachments in other remote servers such as amazon cloud servers etc. I will prototype a solution based on this approach meanwhile you guys come up with a solution. :)
I still think the possibility to store binaries is essential and I like the idea to be able to implement quota restrictions using the journalSize feature.

[fceller]

This is indeed a good argument. There are specialized distributed file-server out there, which will do a better job when dealing with files.

For smaller files: yes, it should be possible. Question still remains how to implement this.

[fceller]

I've discussed the issue with Martin:

Da wir insgesamt versuchen Sachen effizient zu machen, sollten
solche unstrukturierten grossen Dinge binär übertragen und dann
auch binär gespeichert werden.

Damit man mit solchen Blobs effizient umgehen kann muss man
damit aber anders umgehen als mit anderen Dokumenten. So
will man die Semantik anders haben (sprich: man will für solche
grossen Dinge keine synchrone Replikation haben). Man will
sie anders speichern (weil unsere normale Copying Garbage
Collection für solche grossen Dinge eher ineffizient ist).

Und ich wollte nicht versuchen diese unterschiedliche
Behandlung vor dem Entwickler zu verbergen (was bei
anderer Replikationssemantik auch gar nicht ginge). Daher
sollte es also eigene Funktionen zum Speichern, Abfragen
und Löschen von solchen Blobs geben.

Translated:

As we try to do things efficiently in total, should
Such unstructured binary large objects and then transferred
are also stored in binary form.

In order to deal efficiently with such blobs have to
handle it differently but as with other documents. so
want to have different semantics (ie: you want for such
great things have no synchronous replication). They want to
they store different (because our normal copying garbage
Collection of such great things is more efficient).

And I did not want to try this different
To conceal the treatment from the developer (which is
other replication semantics would not). hence
So it should own functions for storing, querying
Delete and type of such blobs.

From my point of view the best approach would be to create a separate document for each file referencing the 'parent document'. That way you avoid having to do a merge if you get revision collisions when a user is updating document and at the same time uploading attachments. A graph relation could be used I guess, will attempt that.

I'm adding this quote from another thread.
"I think that arangodb should be data agnostic. Perhaps you need to create a binaries api and have a separate process that handles binary file attachments."

Although that is not really agnostic (blame json), you can at least store data in any format.
Do you know if there is any way making json work with multipart http content ?
I'm wondering if it's possible to send both {json stuff} and ----- files ------ bundled in the same request.

Adding these for reference.
Multipart/form-data:
http://www.ietf.org/rfc/rfc2388.txt
FormData:
https://developer.mozilla.org/en/XMLHttpRequest/FormData
https://developer.mozilla.org/en/DOM/XMLHttpRequest/FormData/Using_FormData_Objects