I've a ton of JSON that really, really hates to follow a schema

[faddat]

It's blockchain data, and there's a key that contains an array, and the keys in that array are almost always different. This blockchain is advancing rapidly, and I can't really say if things will even out too soon (which would enable me to make some kind of struct for this array). So, I noticed the cool trick with the JSON import:

Will this work for any and all JSON?

Say I have 5 gigs of JSON.... what is the best way to feed it to elias?

[krotik]

Hi,

5 Gigs of data is quite a bit. I would, in any case, add it in multiple smaller chunks. I am assuming that the data is stored in one or multiple files on disk.

I can think of 3 different ways on how the data could be added:

• Adding it through the command line using the -import argument. I would imagine a script which creates import files and then runs EliasDB's import.
• You could use the REST API. However, for this amount of data it is probably too slow and just the wrong mechanism.
• You could write a Go program which embeds EliasDB. It could read the JSON data in chunks and use transactions to feed it into the datastore. (I think this is the best alternative.)

In general: the power of a graph database lies in its ability to store highly connected data. It sounds as if your data uses primarily keys which might be better stored in a normal key-value store. However, if you do want to store it as a graph (e.g the advancement of the blockchain could be modelled as edges) you would need to form small objects of key value pairs. In EliasDB the keys must be strings while the values can be anything. The value of the "key" and "kind" attribute should also be a string. (It is currently a bug that the "key" attribute value must be a string).

Node values can contain any (and all) JSON data. If you embed EliasDB in an own Go program you can even store slices and object structures. In fact anything which can be serialized with gob.

[faddat]

aha!

Okay that was a really huge tip right there about storing slices and object
structres in embedded elias. That's actually what had drawn me towards
it. Of couse I guess I could be getting led back to a non-graph-database
by my next point, because... I suppose if I did something like:

blocknum:returnfromcurl that wouldn't really get me anywhere in terms of it
being a graph database, really.

I'd need to break it down into more granular units like:

blocknum contains userid
blocknum contains trxid
trxid contains operations {
one
two
three
}

or....?

But that's really, really good to know about golang embedded DBs. I'm
definitley a noob when it comes to go.

Jacob Gadikian
E-mail: faddat@gmail.com
SKYPE: faddat
Phone/SMS: +84 167 789 6421

On Mon, Oct 3, 2016 at 7:47 PM, Matthias Ladkau notifications@github.com
wrote:

Closed #6 #6.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#6 (comment), or mute the
thread
https://github.com/notifications/unsubscribe-auth/AGz6iWLss3202ShJx-jdiqhA4dScbM5Pks5qwPl7gaJpZM4KLsKn
.