/
json-import.liquid
292 lines (241 loc) · 9.15 KB
/
json-import.liquid
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
---
converter: markdown
metadata:
title: "[DEPRECATED] Data Import using JSON"
description: Detailed description of JSON data import with examples.
---
This topic describes **data import** using JSON format
{%- capture content -%}Import using JSON format is deprecated. We recommend [importing using CSV format](/developer-guide/data-import-export/import){%- endcapture -%}{%- render 'alert/deprecated', content: content -%}
Data to be imported needs to be in JSON format, it is an array of [user data](/api-reference/graphql/userimport.html) and array of [records data](/api-reference/graphql/customizationimport.html):
```json
{
"users": [...],
"records": [...]
}
```
You can generate data from an existing Instance with the `pos-cli data export` command ([read more about this here](/developer-guide/data-import-export/export#cli-export)). It will generate the `data.json` file. You can also import data from other sources (e.g. old stack, other systems) once you generate the proper JSON file. It is fairly easy to convert, for example, a CSV file into a JSON file.
## General information
- Before import, your Tables need to be deployed.
- When importing data, the `id` fields will be changed. Relations will be kept. If you have your own relations defined in properties, you need to first describe them in Table files - [read more about this here](/developer-guide/data-import-export/record-relations).
- If you test your imports and need to clean Instance data, use the `pos-cli data clean` command. Note, that this command will work only on staging Instances.
- When you run import twice, it will try to match previously imported data without removing existing changes. Example:
1. You have a record `shirt` with properties: `color`, `size`
2. First import adds new `shirt` with JSON: `{ "id": 1, "properties": { "color": "red" }}`
3. User sets attributes of `shirt`, now `shirt` has the following properties: `{ "color": "green", "size": "large" }`
4. Second import with the same file changes `shirt` object to: `{ "color": "red", "size": "large" }`
- Import will only do basic validations, such as the existence of properties and type validation. It doesn't check any of the validations defined on forms.
## Data Import with the CLI
With pos-cli data, you can export/import Instance data (information entered by the users).
platformOS CLI provides the `pos-cli data import` command for uploading and importing data from a JSON file or ZIP archive with CSV files. It accepts one option:
- --path (short: -p): A path to a JSON or ZIP files to be imported
Under the hood, the CLI uses the GraphQL import API that's described in the next section. However, using the CLI provides some major benefits:
- Uploading file from disk
- Leveraging the CLI's authentication mechanism (i.e. you don't need to manually send your authentication token)
### Input format
The required data format is a JSON file with users and records keys that have an array of maps. Each of those collections needs to be in the same format as a corresponding GraphQL mutation. Under the hood import is just a set of mutations: [`import_users()`](/api-reference/graphql/rootmutation.html), [`import_records()`](/api-reference/graphql/rootmutation.html). When importing data using the CLI, the file should contain data in a format that matches the following structure:
```json
{
"users": [<UserImport],
"records": [<CustomizationImport>],
}
```
### Example
Example data import file `data.json`
```json
{
"users": [
{
"id": 1000,
"email": "tom@example.com",
"first_name": "Tom",
"slug": "tom"
},
{
"id": 1001,
"email": "mike@example.com",
"first_name": "mike",
"slug": "mike",
"created_at": "2010-10-10 10:00:00",
"profiles": [
{
"type_name": "student",
"id": 2001,
"properties": {
"age": 21
},
"attachments": [
{
"name": "document",
"url": "files/document.pdf"
}
],
"images": [
{
"name": "avatar",
"url": "files/avatar.png"
}
]
}
]
}
],
"records": [
{
"id": 1,
"user_id": 1000,
"type_name": "house",
"customizable_id": 1000,
"customizable_type": "User",
"properties": {
"color": "red"
}
},
{
"id": 2,
"user_id": 1001,
"type_name": "house",
"customizable_id": 1001,
"customizable_type": "User",
"properties": {
"color": "green",
"buyer_id": 1000,
"beach_house_id": 1
},
"addresses": [
{
"name": "foo_address",
"address": "Warsaw",
"postcode": "01-001"
}
]
},
{
"id": 3,
"user_id": 1001,
"type_name": "house",
"properties": {
"buyer_id": 1000,
"beach_house_id": 2
},
"attachments": [
{
"name": "driver_licence",
"url": "files/licence.pdf"
}
],
"images": [
{
"name": "headshot",
"url": "files/headshot.png"
}
]
}
]
}
```
To import the data from a file, run the following command in the terminal:
<pre class="command-line" data-user="user" data-host="host"><code class="language-bash">
pos-cli data import --path=data.json staging-instance
</code></pre>
## Data Import with GraphQL mutations
It is possible to import data using GraphQL mutations. This gives you more flexibility, and even allows you to create your own import tool as a website feature. Please remember to handle authorization properly.
There are two main mutations for importing [`import_users()`](/api-reference/graphql/rootmutation.html) and [`import_records()`](/api-reference/graphql/rootmutation.html). You can create a page that will import sent data.
##### app/graphql/import_data.graphql
```graphql
mutation import_data(
$users: [UserImport!]!,
$records: [CustomizationImport!]!
) {
import_users(users: $users) { ids }
import_records(records: $records) { ids }
}
```
##### app/views/pages/import_data.json.liquid
{% raw %}
```liquid
---
method: post
---
{% graphql res = "import_data",
users: context.params.users,
records: context.params.records
%}
{{ g }}
```
{% endraw %}
<pre class="command-line" data-output="2-6" data-user="user" data-host="host"><code class="language-bash">
curl -d @transformed.json \
-H "Content-Type: application/json" \
-X POST https://example.com/import_data.json
</code></pre>
## Data Import with raw import API
Our CLI uses the HTTP API to import a data file. You can use it in your own tool.
Please note, that you need to provide your authentication token in the HTTP Authorization header of the request.
<pre class="command-line" data-output="2-6" data-user="user" data-host="host"><code class="language-bash">
curl -F "import[data]=@data.json" \
-H "Authorization: Token token=[YOUR API TOKEN]" \
-X POST https://example.com/api/app/imports
</code></pre>
{% include 'alert/note', content: 'You can find your API Token in [Partner Portal](https://partners.platformos.com/me) under "Access Key".' %}
## Importing Images and Files
Records and Users can store images and attachments in properties (Visit [User Uploads](/developer-guide/user-uploads/user-uploads)). To import them you need to have the proper data record in [*.yml files](/developer-guide/properties). Here is an example record with images and attachments:
```yml
name: car
properties:
- name: color
type: string
- name: car_document
type: file
- name: car_photo
type: image
```
## Uploading files with CLI
To import files with `pos-cli`, prepare a JSON file with paths to existing files. You need to have those files on your disk. Read more about the required structure in our [reference docs](/api-reference/graphql/customizationimport.html). Here is an example JSON file that assumes you have files in the `import_files/` directory:
```json
{
"records": [
{
"id": "id",
"type_name": "car"
"user_id": null,
"properties": {
"color": "red"
},
"attachments": [
{
"url": "import_files/offer.pdf",
"name": "car_document"
}
],
"images": [
{
"url": "import_files/car.png",
"name": "car_photo"
}
]
}
]
}
```
During the import, `pos-cli` will upload the files to our storage and match them with corresponding records.
## Record relations
With `properties`, you can create your columns for storing the data. In the properties, you can also store the `id` of an existing object. Then you can fetch the data with `related_record(join_on_property: String!)` or the `related_user(join_on_property: String!)` field in GraphQL.
In order to export/import data with relations preserved, you need to define them in Table files using the `belongs_to` attribute.
##### app/schema/house.yml
```yaml
name: house
properties:
- name: address
type: string
```
##### app/schema/cars.yml
```yaml
name: cars
properties:
- name: color
type: string
- name: house_id
type: integer
belongs_to: house
```
This works only with JSON import/export and hence is deprecated.