/
781.txt
653 lines (460 loc) · 26.2 KB
/
781.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
[1] [[改行]]区切りの [[JSON]] 値の列が [[JSON]] によって表現可能なデータの交換に使われることがあります。
これを[DFN[[[改行]]区切りの [[JSON]]]]、[DFN[[[LDJSON]]]]、[DFN[[[NDJSON]]]]、
[DFN[[[JSON Lines]]]] などと呼ぶことがあります。
;; [85] 統一的な名前はなく色々な呼ばれ方をしています。
特に名称なしで利用されていることもあります。
* 構文
[2] 0個以上の [[JSON]] 値を[[改行]]で区切ります。
[FIG(railroad)[
= ?
== [[JSON]] 値
== *
=== [[改行]]
=== [[JSON]] 値
]FIG]
[3] 連続した[[改行]] (= [[空行]]) やファイル末尾の[[改行]]をどう解釈するかは、実装によります。
[4] [[改行]]が [[LF]] か [[CRLF]] か、いずれでもよいかは実装によります。
;; [38] [[LF]] を区切りとして解釈すれば、 [[CR]] は [[JSON]] の[[空白]]なので、
自然に無視されます。
[5] [[JSON]] 値には[[字句]]間の[[空白]]として[[改行]]を含めることも本来は可能ですが、
本形式では使用しません。これにより、1行 = 1 [[JSON]] 値となり、処理しやすくなります。
;; [6] [[JSON]] 値における[[空白]]は[[人間]]の可読性の向上以外に意味を持たないので、
情報交換用には不要です。なお[[文字列]]中の[[改行]]は [[JSON]] 値内では必ず[[エスケープ]]が必要なので、
元々[[改行]]そのままの形で現れることはありません。
(ですから、[[改行]]が含まれる [[JSON]] 値から何も考えずに[[改行]]を除去すれば、
ここで使える[[JSON]]値となります。)
;; [7] [[JSON]] 値内部の[[改行]]をそのままにしても、意図通り[[構文解析]]することは可能ではあります。
行指向の処理はできなくなりますが、[[JSON]]値の終わりが現れる行までを読み込んで1つの
[[JSON]]値として処理するだけです。
なお [[Wikipedia]] はそのようなものも含め、区切りを含まないものを
[[Concatenated JSON]] [SRC[>>44]] と呼んでいます。
[37] [[JSON]] 値としてすべて認めるものと、[[オブジェクト]]や[[配列]]など特定の形のもののみ認めるものがあります。
[50] [[文字コード]]としては、 [[UTF-8]] を使うのが普通です。
* 処理
[86] 入力を[[行]]ごとに、それぞれ [[JSON]] として[[復号]]していきます。
[87] [[行指向]]の処理は [[Unix]] などで一般的に用いられており、
ほとんどの[[プログラミング言語]]の[[標準ライブラリー]]で簡単に実装できます。
[88] [[JSON]] の処理も、特別なことはせず、単に [[JSON]] から内部データ構造に変換する一般的な[[ライブラリー]]を呼び出すだけで十分です。
[89] 後は、利用する場面に応じて[[空行]]を受信した時の処理、
不正な [[JSON]] を受信した時の処理、
入力ストリームの末端に到達した時の処理を実装し忘れないように注意しましょう。
;; [90] こうした処理をまとめて行う専用の[[ライブラリー]]もいくつもありますが、
その場その場で都度実装しても良いくらい単純な処理で済みます。
* MIME 型
[51] [[NDJSON]] 仕様は [[MIME型]]として [DFN[[CODE(MIME)@en[[[application/x-ndjson]]]]]]
を規定しています [SRC[>>23]]。
[56] [[Wikipedia]] は [[MIME型]]として [DFN[[CODE(MIME)@en[[[application/x-ldjson]]]]]]
を使うべきと述べています [SRC[>>14]]。
[62] たまに [CODE(MIME)@en[[[application/json]]]] が使われることもあります。 (正しくはありません。)
* 実装
[10] [[jq]] は、結果が複数の [[JSON]] 値となるとき[[改行]]区切りで出力します。
ただし、標準では [[JSON]] 値に[[改行]]を含む[[空白]]を入れて[[人間]]が見やすい形にします。
[CODE[-c]] / [CODE[--compact-output]] オプションを指定すると、[[空白]]のない、
本項の形式となります。
[12] [[Twitter]] は [[CRLF]] 区切りの [[JSON]] 値を使っています [SRC[>>292]]。
[[ツイート]]に [[LF]] が含まれる可能性がある ([[CR]] が含まれる可能性はない) [SRC[>>292]]
とされていますが、[[文字列]]に生の [[LF]] が含まれることがあるのか、それとも字句間に含まれることがあるだけなのかは不明です (前者は [[JSON]] 仕様上あり得ないはずですが)。
[13] [[Twitter]] はオプション形式として、[[JSON]]値の前に [[JSON]]値の長さを[[十進数]]で出力する形式も用意しています [SRC[>>292]]。
[[CRLF]] の出現を探すのではなく、指定文字列長分を読み込む方式の方が都合がいい実装のために用意されているようです。
[REFS[
- [292] [CITE[Processing streaming data | Twitter Developers]] ([TIME[2015-03-05 11:53:33 +09:00]] 版) <https://dev.twitter.com/streaming/overview/processing>
]REFS]
[83] >>82 は、[[改行]]区切りにされることがよくあるが、
他の[[空白]]区切りでも (曖昧でなければ[[空白]]なしでも) 受け付ける、としています。
この形式は他の実装では [[Concatenated JSON]] と呼ばれています。
[11] その他次の実装が[[改行]]区切りの [[JSON]] 値列を使っています。
[REFS[
- [291] [CITE[Bulk API]]
([TIME[2015-02-24 04:40:27 +09:00]] 版)
<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html>
]REFS]
[63] いろいろな [[Webアプリケーション]]の [[Web API]] が本項の形式 (の何らかのバリエーション)
を採用しています。
[57] [[Chrome]] の[[開発者ツール]]は[[応答]]が [[JSON]] の列の時も [[JSON]] 値の[[配列]]のように表示できます。
* 関連
[8] [[JSON]] 値の列全体を [[JSON]] [[配列]]とすることも可能ですが、
末尾まで読み終えてから処理するのでない ([[ストリーミング]]処理したい)
場合に扱いづらいのが問題です。
[84] [[JSON]] をストリーミング形式で構文解析することも可能で、
そのような実装も存在はしますが、本方式の方がずっと簡単に実現できます。
[WEAK[(ストリーミングに対応していない普通の [[JSON]] の実装を流用できます。)]]
[9] [[JSON text sequences]] は本項の方式から派生したもののようですが、
特殊な[[制御文字]]を使っています。本項の方式と比べてそこまで複雑でもありませんが、
メリットも特にありません。本項の方式の方が [[Unix]] その他の[[行指向]]の既存の[[ライブラリー]]等は高そうです。
* 歴史
[81] [[NDJSON]] は、昔は [[LDJSON]] と言っていました [SRC[>>80]]。
[REFS[
- [80] [CITE@en[Update specification_draft2.md · ndjson/ndjson-spec@c658c26]] ([TIME[2016-02-21 20:56:12 +09:00]] 版) <https://github.com/ndjson/ndjson-spec/commit/c658c26ba15bea718228b5a3d54f0d4db22ec0ba>
]REFS]
* メモ
[14] [CITE@en[Line Delimited JSON - Wikipedia, the free encyclopedia]] ([TIME[2015-02-24 09:07:32 +09:00]] 版) <http://en.wikipedia.org/wiki/Line_Delimited_JSON>
[15] [CITE[standard content-type for "streaming JSON" (newline delimited JSON objects)? - Google グループ]]
([TIME[2015-03-13 19:49:54 +09:00]] 版)
<https://groups.google.com/forum/#!topic/nodejs/0ohwx0vF-SY>
[16] [CITE[websocket - Choice of transports for JSON over TCP - Stack Overflow]]
([TIME[2015-03-13 19:47:01 +09:00]] 版)
<http://stackoverflow.com/questions/6573870/choice-of-transports-for-json-over-tcp>
[17] [CITE[How We Built Filmgrain, Part 2 of 2 | Filmgrain Blog]]
([TIME[2013-09-15 11:20:31 +09:00]] 版)
<http://blog.filmgrainapp.com/2013/07/02/how-we-built-filmgrain-part-2-of-2/>
[18] [CITE@en[ldjson-stream]]
([TIME[2015-03-13 19:51:08 +09:00]] 版)
<https://www.npmjs.com/package/ldjson-stream>
[19] [CITE@en[maxogden/ndjson]]
([TIME[2015-03-13 19:51:19 +09:00]] 版)
<https://github.com/maxogden/ndjson>
[55] [CODE[[[mongoimport]]]] [SRC[>>53]] / [CODE[[[mongoexport]]]] [SRC[>>54]]
における [[JSON]] とは原則として[[改行]]区切りの [[JSON]] の列のことを言っています。
[REFS[
- [53] [CITE@en[mongoimport — MongoDB Manual 3.0.1]] ([TIME[2015-03-28 05:46:01 +09:00]] 版) <http://docs.mongodb.org/manual/reference/program/mongoimport/>
- [54] [CITE@en[mongoexport — MongoDB Manual 3.0.1]] ([TIME[2015-03-28 05:45:59 +09:00]] 版) <http://docs.mongodb.org/manual/reference/program/mongoexport/>
]REFS]
[20] [CITE@en[ld-jsonstream]]
([TIME[2015-03-13 19:51:24 +09:00]] 版)
<https://www.npmjs.com/package/ld-jsonstream>
[21] [CITE@en[timkuijsten/node-ld-jsonstream]]
([TIME[2015-03-13 19:51:34 +09:00]] 版)
<https://github.com/timkuijsten/node-ld-jsonstream>
[22] [CITE[how to parse a large, Newline-delimited JSON file by JSONStream module in node.js? - Stack Overflow]]
([TIME[2015-03-13 19:51:46 +09:00]] 版)
<http://stackoverflow.com/questions/15121584/how-to-parse-a-large-newline-delimited-json-file-by-jsonstream-module-in-node-j>
[23] [CITE@en[NDJSON - Newline delimited JSON - Data Protocols - Open Knowledge Foundation]]
([[Open Knowledge Foundation and others]] 著, [TIME[2015-03-04 16:29:27 +09:00]] 版)
<http://dataprotocols.org/ndjson/>
[24] [CITE[Storing Data as Newline Delimited JSON Entries - Will Anderson]]
([[Will Anderson]] 著, [TIME[2015-02-27 16:14:27 +09:00]] 版)
<http://willi.am/blog/2014/07/16/storing-data-as-newline-delimited-json-entries/>
[25] [CITE[JSON Lines]]
([TIME[2015-02-25 07:23:03 +09:00]] 版)
<http://jsonlines.org/>
[26] [CITE[ndjson]]
([TIME[2015-01-02 21:04:22 +09:00]] 版)
<http://ndjson.org/>
[FIG(quote)[
[FIGCAPTION[
[27] [CITE[Streaming API | Plotly Developers]]
([TIME[2015-03-13 19:59:16 +09:00]] 版)
<https://plot.ly/streaming/>
]FIGCAPTION]
> Once the stream has been connected over http, write to the request stream with newline separated JSON.
> httpRequestSocket.write('{ "x": 3, "y": 1 }\n')
> The newline is extremely important. Without this delimiter the Streaming Endpoint will not delineate your data, and will terminate the stream. You can send multiple streams to the same plot by nesting stream tokens within the corresponding data trace object. Similarly you can use the same token for multiple traces in a plot (they will show the same stream, so this is useful only in when using subplots).
]FIG]
[52] [CITE@en[draft-hallambaker-jsonl-01 - JSON Log Format (JSON-L)]]
( ([TIME[2014-05-07 04:59:18 +09:00]] 版))
<http://tools.ietf.org/html/draft-hallambaker-jsonl-01>
[28] [CITE[Lesspress.net]]
([[Christopher Brown, io@henrian.com]] 著, [TIME[2014-12-04 07:34:04 +09:00]] 版)
<http://lesspress.net/#38>
[29] [CITE@en[ndjson/ndjson-spec]]
([TIME[2015-03-13 20:44:22 +09:00]] 版)
<https://github.com/ndjson/ndjson-spec>
[FIG(quote)[
[FIGCAPTION[
[30] [CITE@en[dat/rest-api.md at master · maxogden/dat]]
([TIME[2015-03-13 20:46:00 +09:00]] 版)
<https://github.com/maxogden/dat/blob/master/docs/rest-api.md#post-apibulk>
]FIGCAPTION]
> Data can be either newline-delimited JSON (with content-type application/json) or CSV (with content-type text/csv).
> The response will be a stream of newline-delimited JSON objects.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[31] [CITE@en[Bubbles: Python ETL Framework (prototype) - Open Knowledge Labs]]
([TIME[2015-03-10 23:46:38 +09:00]] 版)
<http://okfnlabs.org/blog/2014/09/01/bubbles-python-etl.html>
]FIGCAPTION]
> JSON newlline delimited file in a store represented by a directory of JSOND files
]FIG]
[FIG(quote)[
[FIGCAPTION[
[32] [CITE[logstash - open source log management]]
([TIME[2015-03-03 06:25:28 +09:00]] 版)
<http://logstash.net/docs/1.4.2/codecs/json_lines>
]FIGCAPTION]
> This codec will decode streamed JSON that is newline delimited. For decoding line-oriented JSON payload in the redis or file inputs, for example, use the json codec instead. Encoding will emit a single JSON string ending in a ‘\n’
]FIG]
[FIG(quote)[
[FIGCAPTION[
[33] [CITE[Storing alerts as JSON — OSSEC 2.8.1 documentation]]
([TIME[2015-03-09 23:01:50 +09:00]] 版)
<http://ossec-docs.readthedocs.org/en/latest/manual/output/json-alert-log-output.html>
]FIGCAPTION]
> With the json output, you can write alerts as a newline separated json file which other programs can easily consume.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[34] [CITE[Class: Traject::MarcReader — Documentation for traject/traject (master)]]
([TIME[2015-03-13 20:50:41 +09:00]] 版)
<http://www.rubydoc.info/github/traject/traject/master/Traject/MarcReader>
]FIGCAPTION]
> "json" The "marc-in-json" format, encoded as newline-separated json. (synonym 'ndj'). A simplistic newline-separated json, with no comments allowed, and no unescpaed internal newlines allowed in the json objects -- we just read line by line, and assume each line is a marc-in-json.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[35] [CITE[log-defer-viz - search.cpan.org]]
([TIME[2015-03-13 20:52:16 +09:00]] 版)
<http://search.cpan.org/dist/Log-Defer-Viz/bin/log-defer-viz>
]FIGCAPTION]
> INPUT FORMAT
> $ log-defer-viz --input-format=json ## default is newline separated JSON
]FIG]
[36] [CITE[python-stream-json]]
([TIME[2013-12-04 07:00:02 +09:00]] 版)
<http://www.enricozini.org/2011/tips/python-stream-json/>
[39] [CITE@en[Feed exports — Scrapy 0.24.5 documentation]]
([TIME[2015-02-26 06:37:57 +09:00]] 版)
<http://doc.scrapy.org/en/latest/topics/feed-exports.html#topics-feed-format-jsonlines>
[40] [CITE[Add support for the JSON lines format. ''''''[''''''#937474'''''']'''''' | Drupal.org]]
([TIME[2015-03-13 21:55:28 +09:00]] 版)
<https://www.drupal.org/node/937474>
[FIG(quote)[
[FIGCAPTION[
[41] [CITE@en[OpenCPU - High performance JSON streaming in R: Part 1]]
([[Jeroen Ooms]] 著, [TIME[2015-03-12 16:39:22 +09:00]] 版)
<https://www.opencpu.org/posts/jsonlite-streaming/>
]FIGCAPTION]
> The json streaming format
> Because parsing huge JSON strings is difficult and inefficient, JSON streaming is done using lines of minified JSON records.
]FIG]
[42] [CITE@en[bitdivine/jline]]
([TIME[2015-03-13 22:03:56 +09:00]] 版)
<https://github.com/bitdivine/jline>
[FIG(quote)[
[FIGCAPTION[
[43] [CITE[API Basics — Scrapinghub documentation]]
([TIME[2015-03-05 23:55:58 +09:00]] 版)
<http://doc.scrapinghub.com/api.html#data-formats>
]FIGCAPTION]
> To better support streaming with many popular JSON parsers, we provide JSON Lines format by default, but JSON and CSV are also available.
]FIG]
[44] [CITE@en[JSON Streaming - Wikipedia, the free encyclopedia]]
([TIME[2015-03-21 21:13:36 +09:00]] 版)
<http://en.wikipedia.org/wiki/JSON_Streaming>
[FIG(quote)[
[FIGCAPTION[
[45] [CITE@en[Streaming | Flowdock API]]
([TIME[2015-03-29 18:06:53 +09:00]] 版)
<https://www.flowdock.com/api/streaming>
]FIGCAPTION]
> JSON Stream (application/json)
> The JSON stream returns messages as JSON objects that are delimited by carriage return (\r). Newline characters (\n) may occur in messages, but carriage returns should not.
> Parsers must be tolerant of occasional extra newline characters placed between messages. These characters are sent as periodic "keep-alive" messages to tell clients and NAT firewalls that the connection is still alive during low message volume periods.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[46] [CITE[The JSON Streaming Record (JSRec) data format « Huy Nguyen]]
([TIME[2014-07-23 13:00:27 +09:00]] 版)
<http://www.huyng.com/posts/json-streaming-record-data-format/>
]FIGCAPTION]
> Files of this format have .jsrec as their file extension
> Each line in the file is a json hash map
> Empty lines and lines beginning with ‘#’ are considred comments and ignored during parsing
]FIG]
[FIG(quote)[
[FIGCAPTION[
[47] [CITE@en[Docker Remote API v1.17 - Docker Documentation]]
([TIME[2015-03-26 12:53:56 +09:00]] 版)
<https://docs.docker.com/reference/api/docker_remote_api_v1.17/>
]FIGCAPTION]
> Content-Type: application/json
> {"stream": "Step 1..."}
> {"stream": "..."}
> {"error": "Error...", "errorDetail": {"code": 123, "message": "Error..."}}
]FIG]
[FIG(quote)[
[FIGCAPTION[
[48] [CITE@en[Extend your community - Meetup]]
([TIME[2015-03-29 18:43:17 +09:00]] 版)
<http://www.meetup.com/meetup_api/docs/stream/2/open_events/>
]FIGCAPTION]
> A response message is one HTTP chunk, the body of which is a single json object, described below, terminated by a newline.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[49] [CITE@en[Streaming]]
([TIME[2015-02-21 05:45:15 +09:00]] 版)
<http://developer.oanda.com/rest-live/streaming/>
]FIGCAPTION]
> All data written to the stream are encoded in the JSON format. Events sent to the stream are either heartbeats (every 15 seconds) to ensure that HTTP connection remains active or transactions reporting the following events:
]FIG]
[58] [CITE@en[Talk:Line Delimited JSON - Wikipedia, the free encyclopedia]]
([TIME[2015-02-26 19:59:49 +09:00]] 版)
<http://en.wikipedia.org/wiki/Talk:Line_Delimited_JSON>
[FIG(quote)[
[FIGCAPTION[
[59] [CITE[Catmandu::Importer::JSON - search.cpan.org]]
([TIME[2015-03-29 20:19:56 +09:00]] 版)
<http://search.cpan.org/dist/Catmandu/lib/Catmandu/Importer/JSON.pm>
]FIGCAPTION]
> multiline switches optionally between line-delimited JSON and multiline JSON or arrays. the default is line-delimited JSON.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[60] [CITE@en[Farsight Security/Blog/NMSG and JSON encoding]]
([[Mike Schiffman]] 著, [TIME[2015-06-02 08:28:56 +09:00]] 版)
<https://www.farsightsecurity.com/Blog/20150506-mschiffm-nmsg-nmsgjsontool/>
]FIGCAPTION]
> Currently, the only type of JSON njt expects and emits is newline-delimited JSON. Each record is expected to be terminated with a literal \n and no \n's can appear inside the JSON.
]FIG]
[61] [CITE@en[Re: Please help with writing spec for async JSON APIs]]
([TIME[2015-08-18 01:44:10 +09:00]] 版)
<https://www.mail-archive.com/es-discuss@mozilla.org/msg36495.html>
[FIG(quote)[
[FIGCAPTION[
[64] [CITE[Streaming API | DataSift Developers]]
([TIME[2015-08-28 21:50:10 +09:00]] 版)
<http://dev.datasift.com/docs/api/streaming-api>
]FIGCAPTION]
> The response body contains a list of JSON objects separated by new lines.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[65] [CITE[pull | DataSift Developers]]
([TIME[2015-08-28 23:07:20 +09:00]] 版)
<http://dev.datasift.com/docs/api/rest-api/endpoints/pull>
]FIGCAPTION]
>
> X-DataSift-Format: json_new_line
> Content-Type: application/ldjson
]FIG]
[FIG(quote)[
[FIGCAPTION[
[66] [CITE@en[Preparing Data for BigQuery - BigQuery — Google Cloud Platform]]
([[Google]] 著, [TIME[2015-09-17 06:50:38 +09:00]] 版)
<https://cloud.google.com/bigquery/preparing-data-for-bigquery>
]FIGCAPTION]
> One JSON object, including any nested/repeated fields, must appear on each line.
> The following example shows sample nested/repeated data:
> {"kind": "person", "fullName": "John Doe", "age": 22, "gender": "Male", "citiesLived": '''['''{ "place": "Seattle", "numberOfYears": 5}, {"place": "Stockholm", "numberOfYears": 6}''']'''}
> {"kind": "person", "fullName": "Jane Austen", "age": 24, "gender": "Female", "citiesLived": '''['''{"place": "Los Angeles", "numberOfYears": 2}, {"place": "Tokyo", "numberOfYears": 2}''']'''}
>
]FIG]
[FIG(quote)[
[FIGCAPTION[
[67] [CITE[ニコニコ動画 『スナップショット検索API』 ガイド]]
([TIME[2015-09-15 12:49:32 +09:00]] 版)
<http://search.nicovideo.jp/docs/api/snapshot.html#toc5>
]FIGCAPTION]
> 【検索レスポンス】
> {"dqnid":"c0676eea-cc77-4317-b442-d626c5f34558","type":"hits","values":'''['''{"_rowid":0,"cmsid":"sm13208019","title":"【DIVA 2nd】 鏡音八八花合戦 【EDIT PV】","view_counter":9999},{"_rowid":1,"cmsid":"sm12215733","title":"メイドイン俺でミニゲーム その5","view_counter":9997},{"_rowid":2,"cmsid":"sm3495465","title":"【初音ミクオリジナル】~プレゼント~【Independence Free】","view_counter":9997}''']'''}
> {"dqnid":"c0676eea-cc77-4317-b442-d626c5f34558","type":"stats","values":'''['''{"_rowid":0,"service":"video","total":213353}''']'''}
> {"dqnid":"c0676eea-cc77-4317-b442-d626c5f34558","endofstream":true,"type":"hits"}
> {"dqnid":"c0676eea-cc77-4317-b442-d626c5f34558","endofstream":true,"type":"stats"}
> 検索レスポンス仕様
> 改行(\n)区切りを一つのチャンクとして、いくつかのチャンクに分かれて返ってきます。
]FIG]
[68] [CITE@en[thenativeweb/json-lines-client]]
([TIME[2015-10-09 23:01:06 +09:00]] 版)
<https://github.com/thenativeweb/json-lines-client>
[FIG(quote)[
[FIGCAPTION[
[69] [CITE[API Basics — Scrapinghub documentation]]
([TIME[2015-10-09 21:51:46 +09:00]] 版)
<http://doc.scrapinghub.com/api.html>
]FIGCAPTION]
> To better support streaming with many popular JSON parsers, we provide JSON Lines format by default, but JSON and CSV are also available.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[70] [CITE@en[mattdesl/garnish]]
([TIME[2015-11-29 22:52:44 +09:00]] 版)
<https://github.com/mattdesl/garnish>
]FIGCAPTION]
> Typically, you would use bole or ndjson to write the content to garnish. You can also write ndjson to stdout like so:
]FIG]
[FIG(quote)[
[FIGCAPTION[
[71] [CITE@en[rvagg/bole]]
([TIME[2015-11-29 22:53:23 +09:00]] 版)
<https://github.com/rvagg/bole>
]FIGCAPTION]
> Newline separated JSON output to arbitrary streams
]FIG]
[72] [CITE[JSON Lines | Hacker News]]
([TIME[2015-11-29 22:59:09 +09:00]] 版)
<https://news.ycombinator.com/item?id=10280483>
[73] [CITE[Storing Data as Newline Delimited JSON Entries - Will Anderson]]
([[Will Anderson]] 著, [TIME[2015-09-15 10:20:59 +09:00]] 版)
<http://willi.am/blog/2014/07/16/storing-data-as-newline-delimited-json-entries/>
[FIG(quote)[
[FIGCAPTION[
[74] [CITE[json(1) - JSON love for your command line]]
([TIME[2014-08-25 16:28:55 +09:00]] 版)
<http://trentm.com/json/>
]FIGCAPTION]
> "Adjacent" objects means objects separated by a newline, or by no space at all. Adjacent arrays means separate by a newline. These conditions are chosen as a balance between (a) not being ambiguous to parse with a simple regex and (b) enough to be useful for common cases.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[75] [CITE@en[Everyone loves R markdown and Github; stories from the R Summit, day two | 4D Pie Charts]]
([TIME[2015-11-29 23:04:26 +09:00]] 版)
<http://4dpiecharts.com/2015/06/28/everyone-loves-r-markdown-and-github-stories-from-the-r-summit-day-two/>
]FIGCAPTION]
> jsonlite also supports ndjson, where each line is a JSON object. This is important for large files: you can just parse one line at a time, then return the whole thing as a list or data frame, or you can define a line specific behaviour.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[76] [CITE[R: Streaming JSON input/output]]
([TIME[2015-11-29 23:05:45 +09:00]] 版)
<http://finzi.psych.upenn.edu/library/jsonlite/html/stream_in.html>
]FIGCAPTION]
> Because parsing huge JSON strings is difficult and inefficient, JSON streaming is done using lines of minified JSON records, a.k.a. ndjson. This is pretty standard: JSON databases such as dat or MongoDB use the same format to import/export datasets. Note that this means that the total stream combined is not valid JSON itself; only the individual lines are. Also note that because line-breaks are used as separators, prettified JSON is not permitted: the JSON lines must be minified.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[77] [CITE@en[stephenplusplus/jsonl]]
([TIME[2015-11-29 23:09:11 +09:00]] 版)
<https://github.com/stephenplusplus/jsonl>
]FIGCAPTION]
> Transform a stream of JSON into a stream of Line Delimited JSON
]FIG]
[FIG(quote)[
[FIGCAPTION[
[78] [CITE[embulk-parser-jsonlをリリースしました! - Qiita]]
([TIME[2015-11-29 23:09:57 +09:00]] 版)
<http://qiita.com/shun0102/items/8989e6ed2ee0f46a0fa9>
]FIGCAPTION]
> 僕がリリースしたのはjsonlということで、1行につき1つのJSONというフォーマットを読み込むためのparserです。
]FIG]
[79] [CITE@en[axiaoxin/vim-json-line-format]]
([TIME[2015-12-29 18:12:22 +09:00]] 版)
<https://github.com/axiaoxin/vim-json-line-format>
[FIG(quote)[
[FIGCAPTION[
[82] [CITE[Orchestrate]]
([[Orchestrate]] 著, [TIME[2016-03-08 10:22:00 +09:00]] 版)
<https://orchestrate.io/docs/apiref#bulk>
]FIGCAPTION]
> Or alternatively, to send a JSON stream, set the Content-Type header to "application/orchestrate-export-stream+json". JSON stream data is often sent with one JSON object per line, using newline characters to delimit adjacent objects, but this isn't strictly necessary with Orchestrate. You can format the data with arbitrary whitespace (or put them direclty adjacent to one another, as long as the stream can be parsed as a sequence of valid JSON objects.
]FIG]
[FIG(quote)[
[FIGCAPTION[
[91] [CITE@en[Event endpoint · Appuri]]
( ([TIME[2016-08-24 16:22:38 +09:00]]))
<http://developer.appuri.com/v1/docs/event-endpoint>
]FIGCAPTION]
> Events are sent via an HTTPS POST request with the content type set to application/x-ldjson (Line-Delimited JSON).
]FIG]
[FIG(quote)[
[FIGCAPTION[
[92] [CITE@en[REST API]]
( ([TIME[2016-02-27 07:16:28 +09:00]]))
<http://quasar-analytics.org/docs/restapi/>
]FIGCAPTION]
> The following values are supported for the Accept header:
> Value Description
> None “Human-Readable” results, one result per line. Note: not parseable as a single JSON object.
> application/json Nicely formatted JSON array
> application/ldjson;mode=precise One result per line
]FIG]
[FIG(quote)[
[FIGCAPTION[
[93] [CITE@en[REST API]]
( ([TIME[2016-02-27 07:16:28 +09:00]]))
<http://quasar-analytics.org/docs/restapi/>
]FIGCAPTION]
> Note that if you remove the Accept header, then you will receive Precise JSON, which is the default format. The response would then look like this:
> { "city": "WESTOVER AFB", "state": "MA", "pop": 1764, "loc": '''[''' -72.558657, 42.196672 ''']''' }
> { "city": "CUMMINGTON", "state": "MA", "pop": 1484, "loc": '''[''' -72.905767, 42.435296 ''']''' }
]FIG]