Graph DB: Requirements

[joeflack4]

Broad requirements

1. Transactionality: Need to be able to do CRUD (Creates, Reads, Updates, Deletes). Siggie also wrote: The value set data changes frequently -- by the minute or hour -- and needs to be up-to-date in the graph database.
2. Web API: Should be able to run on a separate server. It will be used by TIMS and TermHub, and should have its own infrastructure and lifecycle.
3. Query complexity: Should be very flexible and rich, as we do not know what our future use cases will be.
4. Data variety: We need much more data than we've attempted to load in OAK.
   i. Additional structures: In addition to edges for vocabulary structures, we need concept metadata, relationship metadata, full N3C/OMOP vocabulary tables, versions of these tables;
   ii. ValueSets (Graph DB: Requirement: CRUD on graph database for value sets #515): We need value sets (OMOP/N3C, eventually ATLAS and VSAC) in the graph database.
   iii. ?: Currently we do our graph analysis using a graph tool -- NetworkX, which could be replaced by OAK -- but it forces our application to do most data retrieval in Postgres, including a lot of graph-like analysis, and a small part of the complex stuff in the graph tool. (Joe: Not sure what the issue is here, per OAK or otherwise)

Specific requirements

1. Graph DB: Requirement: Subgraph from concept IDs #509

Options considered

Details

1. TCCM: One option is to continue development on TCCM, which already has a Neo4J graph DB backend, and already has a lot of code written out, e.g. loaders for some ontologies and REST endpoints.
2. TriplyDB: May be able to use this as a free option, combine multiple RDF files, and upload them here.
3. OAK
4. Gremlin (JanusGraph)
5. Gremlin (Neo4J)
6. Cypher (Neo4J)
7. networkx

Related

• Future: Graph-based backend #170 (some old discussion)

[joeflack4]

@Sigfried Closing this, as we have a good solution for networkx and the link to TIMS here appears to be no longer.