Graph API

[pudo]

The data in aleph is stored as documents in ElasticSearch, with a schema that can best be expressed in RDF. However, we want to be able to show network graph visualisations of parts of the corpus. In particular:

• We want to be able to show a network graph focussed on an individual entity or set of entities.
• We want a higher level abstraction (e.g. a "view" or "canvas") in the context of a casefile which makes a network diagram of a set of entities or documents into a persistent object, preserving entities and layout.

To show aleph data as a network diagram, we will need to transform it into a property graph-style representation. For simplicity and performance, this should take place on the backend.

Fundamentally, the transformation from aleph ftm data to property graph can be done on three levels:

1. By transferring entity links between entities into edges (e.g. an Asset has a link to an Ownership, the Ownership has a second link to a LegalEntity as the owner).
2. By transforming some of the entities into links themselves (e.g. an Ownership would be broken down into an edge between asset and owner).
3. By reifying (making into pseudo-entities) tags of entities into nodes. That would include entity properties such as email addresses, postal addresses, phone numbers, and possibly even names, which would be presented as nodes in the network diagram.

Additionally, there is some need for extra computation that may be required to make this API usable by a UI client:

• Include the degree in metadata for each node, possibly disaggregated by type (i.e. "this entity is linked to 15 assets, 4 email addresses, and 2 postal addresses")
• Allow querying adjacent edges and nodes for a particular entity by edge type.

[sunu]

#464 is the first attempt at creating a graph API. The nodes and relationships are based of FTM schema and they roughly follow the 1st and 3rd point in the original comment. i.e.

1• By transferring entity links between entities into edges (e.g. an Asset has a link to an Ownership, the Ownership has a second link to a LegalEntity as the owner).
and
3• By reifying (making into pseudo-entities) tags of entities into nodes. That would include entity properties such as email addresses, postal addresses, phone numbers, and possibly even names, which would be presented as nodes in the network diagram.

One of the shortcomings in the representation is that we store reverse relationships for only a few types of relations because not every relationship has a reverse defined in FTM schema.

That means it is not possible to make queries like: "get all companies with the address x" efficiently because we only store company->address relationship in the index and not address->company.

May be it is worth storing some of these selected reverse relations even though they are not defined in FTM schema; e.g. for properties like addresses, names? Storing the fingerprints or some kind of normalized form of these properties would be even better for reverse querying.

[pudo]

Agreed on the inverse FtM properties, I've created a ticket over there. But as for the inverse lookup problem I think it's worth us solving this systematically. Fundamentally, we have four types of graph expansion:

• From an entity to all entities that it references (entity-forward), i.e. expand all the properties that have entity IDs as values.
• From an entity to all entities which reference it (entity-reverse), i.e. find all the entities where the root entity is mentioned in properties (there's a denormalised entities field in the index to make this easier).
• From an entity to all tags/selectors (names, phones, emails, ...) which it mentions (tag-forward)
• From a tag/selector to all entities in which it is mentioned (tag-reverse)

My current thinking about the basic graph API is that we should come up with simple APIs for all these four that operate against ElasticSearch. They should all return an iterator over (s, p, o) tuples. We can also think about annotating reverse predicates with a star to annotate them:

• doc:1827 -> names -> name:"Banana King"
• name:"Banana King" -> names* -> doc:1827

We can use redis sets as a sort of cache for their results to accelerate lookups. (Interestingly, this seems pretty close to what Facebook does with it's TAO social graph).

One big aspect we haven't figured out yet is how to intersect all of this with authz -- which is also the reason I don't feel we should merge #464 as an increment for now: it's a security hole.