Generate blank UIDs based on JSON fields on mutations and loaders

[MichelDiz]

Experience Report

What you wanted to do

Import JSON data made by any other DB but having sure that the nodes will be unique.
Informing Bulkload what are the Keys of my JSON are that represent uniqueness.

And also be able to setup a direct injection from API's serving JSON.

something like this for locally JSON
dgraph bulk --jsons="./json.json" --setunique="mission_name, rocket_id, payload_id, site_id"

That way Bulkload would generate a "uid" key throughout JSON before sending it to Dgraph. This ensures that objects are unique.

Also, it would be interesting to have an option to import JSON offered by API. This can help a lot in other situations. Like in Database Migration. Lots of people could use this to migrate from old DBs to Dgraph.

From API's serving JSON:
dgraph bulk --bulkapi="https://api.spacexdata.com/v3/launches/, https://api.spacexdata.com/v1/nasa_launches/" --setunique="mission_name, rocket_id, payload_id, site_id"

Any external references to support your case

e.g: I have this API https://api.spacexdata.com/v3/launches/ . It provides me with a dataset. However, since there is no "uid: _:blank" key (with a blank node) all the rockets in this object of this API will not be unique in Dgraph when I import it.

This was discussed with @codexnull

[manishrjain]

Didn't we already build support for JSON in bulk loader?

[MichelDiz]

Yes we do, after talking to Javier late 2018 we believe that this enhancement would be plausible.

[codexnull]

Yes, the bulk loader already supports JSON. I believe this request is for supporting a way to automatically add a uid field to the data. Currently it requires it to be present already.

[manishrjain]

Ok. After talking to Javier, my understanding is that this would create a blank node for the UID field in the JSON map (if UID field isn't present), based on the fields mentioned in --setunique. That way, all the records holding the same mission_name and/or rocket_id, would get the same UID.

This seems like an easy and useful change.

[campoy]

For reference, in order to load all the data from pokeapi.com I had to play around with UIDs in order to make this work.

In addition to generating blank UIDs from existing IDs (id, name, etc) I also had to make some fields into objects.

For instance given:

{
  "url": "/pokemon/1234",
  "name": "pikachu",
  "types": [
    "electric"
  ]
}

I had to modify it into:

{
  "uid": "_:pokemon1234",
  "url": "/pokemon/1234",
  "name": "pikachu",
  "types": [
    {
      "uid": "_:electric",
      "name": "electric"
    }
  ]
}

You can see the code in github.com/campoy/pokegraph.