-
Notifications
You must be signed in to change notification settings - Fork 869
Data Import and Export #1299
Comments
This has now been released in the latest version of |
Hi, |
So If I want to import from AirTable, what would be my best bet, prisma + AirTable's API wrapper, or export to csv and use graphql-cli-load? @marktani Sorry for reopening too! |
Overview
Graphcool should support importing data from various data sources including:
From a high level the process looks like this:
NDF (Normalized data format)
Requirements
design goals
not important
Files
The NDF (Normalized data format) is stored in a series of files named nodes-x.json, lists-x.json and relations-x.json where x is an incrementing number.
Each file is roughly 10 mb in size. The Graphcool server accepts arbitrarily large files, but files significantly larger than the 10 mb default may cause memory exhaustion.
Structure
The import json object that the endpoint expects must have the following format
Where type can be one of the three import object types:
nodes
,lists
orrelations
. One file can only contain one type of import objects.Nodes
Listvalues
Scalar Lists could contain values that exceed the current size limit of 10mb per file. To circumvent this limitation, they can be split up in multiple lines and therefore multiple files. They will then be concatenated by the server in the order they are provided.
Relations
Graphcool 1.0 allows for optional Back Relations. This means that one model in the relation does not have a field associated with the relation. When such a relation is exported Graphcool 1.0 will generate a format like this:
Both the Framework and Graphcool 1.0 will accept this format. But the Framework of course still needs to have two fields defined on each relations. We will just infer the proper relation to create using the one provided field.
A format like this will also be accepted by both versions of Graphcool.
Value Representations
(new Date()).toJSON()
)"{\"key\": 42}"
)Process
Transform
The cli should support transforming data from various data sources to the NDF (Normalized data format). Source Mappers can be implemented as CLI plugins or build into the CLI.
Verify
The cli will get the schema from the backend and perform basic validation on the type of all data.
Additionally referential integrity can be checked for all relations, as well as identifying nodes with violated required relations.
Import
The backend exposes a new dedicated import endpoint that accepts individual files as described above. A single file is processed at a time. each element in the array is treated individually. Required relations are not verified during import of nodes. If any node fail to import, the index and reason is returned in the response from the import endpoint. The cli can then decide to retry or show the error to the user. Violation of a unique constraint is a likely error.
Other considerations
A similar process could support data export to various formats.
On the server we should perform json parsing in a streaming manner to not clog the CPU: https://github.com/circe/circe/tree/master/examples/sf-city-lots
Open questions
The text was updated successfully, but these errors were encountered: