-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
780 lines (626 loc) · 39.6 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
Youtube-Data
~~~~~~~~~~~~
Stefan Vargyas, stvar@yahoo.com
Jan 18, 2018
Table of Contents
-----------------
0. Copyright
1. The Youtube-Data Program
2. Prerequisites of Youtube-Data
3. Use Cases of Youtube-Data
4. Appendix: Building Json-Type, 'youtube-data.so' and
'youtube-data-litex.so'
5. Appendix: Using Shell Function 'youtube-data'
6. References
0. Copyright
============
This program is GPL-licensed free software. Its author is Stefan Vargyas. You
can redistribute it and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
You should have received a copy of the GNU General Public License along with
this program (look up for the file COPYING in the top directory of the source
tree). If not, see http://gnu.org/licenses/gpl.html.
Additional note: the files '*.json' in the top directory of the source tree
are subject to the same copyright and license provisions as the rest of the
Youtube-Data is.
1. The Youtube-Data Program
===========================
Youtube-Data program's vocation is that of an elaborated reader of the data
associated to video content published by YouTube web site. As is well-known,
YouTube structures its public data around the so-called *resources* -- the
most preeminent: channels, playlists and videos -- and *content* attached
to resources.
Youtube-Data is querying the Google's API [2] from the 'bash' command line
prompt for the content associated to each of the above three categories of
resources and it is displaying the resulting data as formatted UTF8 text on
standard output.
Youtube-Data is using an own timestamping cache mechanism to minimize the
number of actual calls made to the remote service. Consequently, the user
is free to process locally the data received from the remote party the way
needed, without worrying that he might exceed the quota limits allocated.
There is a feature of Youtube-Data that makes this program stand out compared
to others like it (e.g. in contrast with YouTube's own web site): through its
caching mechanism, Youtube-Data is able to produce diffs between old and new
content associated to each resource published.
For example, when one has a list of favorite channels, he may very easily
check each of these channels for to see whether new video materials have
been published until the last time queried (see below section 3, Use Cases
of Youtube-Data, for a couple of relevant 'youtube-data' command lines).
Youtube-Data consists of a quite involved 'bash' shell script 'youtube-data.sh',
which exports, upon being sourced at the command line prompt, its main function
'youtube-data' -- along with a few auxiliary ones.
2. Prerequisites of Youtube-Data
================================
The first prerequisite of using the shell script 'youtube-data.sh' is that of
obtaining a so called *YouTube Data API key* from Google [1]. Upon achieving
that, edit the shell script file 'youtube-data.sh' for to assign the obtained
string value to the environment variable '$YOUTUBE_DATA_APP_KEY'.
The second prerequisite is also about editing 'youtube-data.sh': one must set
a proper value for '$YOUTUBE_DATA_HOME'. The value assigned to this environment
variable must point to the top directory of Youtube-Data's source tree.
Note that if one does not want to touch the file 'youtube-data.sh' for to assign
own values to the two environment variables above, then he may instead do these
assignments for example in user's own '~/.bashrc' file.
A more involved prerequisite of Youtube-Data refers to the third-party programs
required for the well-functioning of the shell functions in 'youtube-data.sh' --
specifically that of its main one, 'youtube-data'. Youtube-Data's 'bash' shell
script makes use of the following programs:
--------------------------------------------------------------------
GNU/Linux Power Tools
--------------------------------------------------------------------
GNU awk 3.1.8
GNU bash 3.2.51
GNU coreutils 8.12 (cat, cut, date, mktemp, rm, sort, tee, tr,
truncate, uniq)
GNU diffutils 2.8.7 (diff)
GNU grep 2.7
GNU sed 4.1.5
GNU wdiff 1.2.1
GNU wget 1.14
--------------------------------------------------------------------
Free Software Tools
--------------------------------------------------------------------
Cache-DB 0.2 (cache)
Json-Type 0.12.1 (json 0.9.0, json.so 0.12.1, json-litex.so 0.1.1)
--------------------------------------------------------------------
The version numbers above indicate which packages and, by case, binaries were
used developing Youtube-Data.
Any mainstream GNU/Linux installation includes all the above GNU/Linux power
tools. The last two tools listed above -- Cache-DB and Json-Type [3] -- must be
build from their sources. Cache-DB's source tree is included within the source
tree of Youtube-Data in 'tools/cache-db' directory. Building Cache-DB is easy,
as per the instructions in 'tools/cache-db/README'. Note that Cache-DB itself
needs an additional GNU/Linux tool: GDBM, the GNU Database Manager library.
For what it concerns Json-Type, the first appendix section below presents in
details how to obtain the newest Json-Type source release package and how to
make the required binaries out of that package's source tree. Note that the
version of Json-Type must be at least 0.12.1.
The last prerequisite of Youtube-Data is that of type library 'youtube-data.so'
and of path library 'youtube-data-litex.so'.
Youtube-Data uses Json-Type for parsing, validating and transforming each JSON
text is receiving from Google's remote service of *YouTube Data API* [2].
The validation of remote service's JSON response texts is done with the help of
files '*.json' to be found in the top directory of the source tree. These text
files contain JSON type definitions in the sense of Json-Type and, respectively,
JSON litex definitions in the sense of Json-Litex, each corresponding to certain
YouTube Data API endpoint accessed by 'youtube-data'. Each time the shell script
accesses a certain endpoint, it feeds into 'json' program the JSON text received
and asks 'json' to validate that according to endpoint's corresponding type and
litex definitions.
Now, instead of passing to 'json' type definitions in form of JSON text files,
'youtube-data' may function by passing to 'json' a compiled type library which
encompasses in binary form all the type definitions contained in the JSON text
files. Using the compiled type library 'youtube-data.so' will make 'json' and,
consequently, 'youtube-data' run faster. This is so because, when given type
definitions in text form, 'json.so' needs to parse and validate that text for
constructing internal data structures. When 'json.so' gets type definitions in
form of the compiled type library 'youtube-data.so', it has nothing more to do
than simply use that: the library 'youtube-data.so' has all of 'json.so's data
structures constructed beforehand.
Similarly, instead of providing to 'json' litex definitions in form of JSON text
files, 'youtube-data' may provide to 'json' the library 'youtube-data-litex.so',
which encloses all the respective litex definitions in binary form. The compiled
path library 'youtube-data-litex.so' will make 'youtube-data' run faster. (This
is so because, when getting litex definitions in form of a compiled path library,
'json-litex.so' will have all the data structures it needs already constructed.)
The first appendix section below gives all the details for one to build the type
library 'youtube-data.so' and the path library 'youtube-data-litex.so' out of the
JSON files contained in the top directory of the source tree.
3. Use Cases of Youtube-Data
============================
This section is devoted to the presentation of main use-cases of Youtube-Data.
Note that the command line examples to be seen below require the fulfillment of
all preconditions of previous section -- for to have a workable 'youtube-data'
shell function sourced in the current 'bash' environment from the shell script
'youtube-data.sh':
$ . youtube-data.sh
First, let have the '.shortcuts' file contain two entries -- corresponding to
two very popular C++ channels, "Meeting C++" and "CppCon":
$ cat <<EOF >.shortcuts
meeting-cpp UCJpMLydEYA08vusDkq3FmjQ
cppcon UCMlGfpWw-RUdWX_JbLCukXg
EOF
The most useful action option of 'youtube-data' is `--diff-recent[=NUM]': it
produces on stdout the 'diff' between previous and current most recent NUM
(5 when NUM is not given) video entries of the resource specified -- either a
channel or a playlist. Below, the syntax `--id=@meet' means that the resource
in question is the channel of which shortcut is 'meeting-cpp':
$ youtube-data --id=@meet --diff-recent
+vDmQlIeY4z0 4 hours 55 mins ago There is a new future - Felix Petrico...
+WVleZqzTw2k 1 day 4 hours ago Strong types for strong interfaces - ...
+DxP--1yEgKQ 2 days 3 hours ago The performance Addict's Toolbox - Pe...
3tIE6X5FjDE 3 days 1 hour ago Modern C++ testing with Catch2 - Phil...
4DvSxmIadb0 4 days 23 hours ago Improve your C++ with inspirations fr...
-UnIc_qJ0DRc 5 days 3 hours ago An inspiring introduction into Templa...
-fN7nVzbRiEk 5 days 20 hours ago Local (Arena) Allocators Part II - Jo...
-ko6uyw0C8r0 5 days 23 hours ago Local (Arena) Memory Allocators Part ...
Note that YouTube's Data API [2] arranges its published resources that provide
result sets on sets of pages, each page to be queried individually for content.
The command line option `--diff-recent[=NUM]' queries the named channel, or, by
case, playlist for the first page of its most recent video entries. (By default,
the number or entries retrieved per each page queried is 50.) Than produces the
'diff' with the previously cached first page selecting only the top NUM entries.
Using option `-p +NUM' on a `--diff-recent' command line will apply the 'diff'
to first 'NUM' pages instead of only the first page. Such use-case may come of
use when the resource in question gets updated very frequently and the change
set is not covered by the first page only.
Option `--no-relative-date' makes the previous command to display the video's
published date in the format received from the remote service:
$ youtube-data --id=@meet --diff-recent --no-relative-date
+vDmQlIeY4z0 2018-01-24T09:59:23.000Z There is a new future - Felix Petricon...
+WVleZqzTw2k 2018-01-23T10:42:10.000Z Strong types for strong interfaces - J...
+DxP--1yEgKQ 2018-01-22T10:55:10.000Z The performance Addict's Toolbox - Pet...
3tIE6X5FjDE 2018-01-21T13:31:19.000Z Modern C++ testing with Catch2 - Phil ...
4DvSxmIadb0 2018-01-19T15:31:07.000Z Improve your C++ with inspirations fro...
-UnIc_qJ0DRc 2018-01-19T11:28:54.000Z An inspiring introduction into Templat...
-fN7nVzbRiEk 2018-01-18T18:42:49.000Z Local (Arena) Allocators Part II - Joh...
-ko6uyw0C8r0 2018-01-18T15:00:18.000Z Local (Arena) Memory Allocators Part 1...
When one does not want to see a diff, but the most recent video entries composed
as table, he need using options `--show-recent[=NUM]' and `--table' (the latter
option is optional for the action option `--show-recent'):
$ youtube-data --id=@meet --show-recent
vDmQlIeY4z0 4 hours 55 mins ago There is a new future - Felix Petricon...
WVleZqzTw2k 1 day 4 hours ago Strong types for strong interfaces - J...
DxP--1yEgKQ 2 days 4 hours ago The performance Addict's Toolbox - Pet...
3tIE6X5FjDE 3 days 1 hour ago Modern C++ testing with Catch2 - Phil ...
4DvSxmIadb0 4 days 23 hours ago Improve your C++ with inspirations fro...
The output of action option `--show-recent' accompanied by option `--list' will
produce more information than that of option `--table'. Below can be seen that
video entries have a 'description' property attached:
$ youtube-data --id=@meet --show-recent=2 --list
videoId=vDmQlIeY4z0
publishedAt=4 hours 55 mins ago
title=There is a new future - Felix Petriconi - Meeting C++ 2017
description=Futures were added to C++11 as high level abstractions for
description=asynchronous operations and are planed to be enhanced with
description=the upcoming C++17 TS. Severals think today that std::future
description=is broken...
videoId=WVleZqzTw2k
publishedAt=1 day 4 hours ago
title=Strong types for strong interfaces - Jonathan Boccara - Meeting
title=C++ 2017
description=Strong types are such a powerful tool for bringing
description=expressiveness into code. By using them we can precisely
description=state our intentions about our C++ objects, both to
description=compilers and to human beings...
The 'description' text attached to channel video entries as provided by the API
may not suit user's terminal configuration. In such a case, option `--wrap=NUM'
is of help, wrapping the texts printed out at the specified number of columns.
In the case below, the added option extends a bit the wrapping column from 72nd
(the default) to 78th, allowing the title of the second item to fit in one line:
$ youtube-data --id=@meet --show-recent=2 --list --wrap=78
videoId=vDmQlIeY4z0
publishedAt=4 hours 55 mins ago
title=There is a new future - Felix Petriconi - Meeting C++ 2017
description=Futures were added to C++11 as high level abstractions for
description=asynchronous operations and are planed to be enhanced with the
description=upcoming C++17 TS. Severals think today that std::future is
description=broken...
videoId=WVleZqzTw2k
publishedAt=1 day 4 hours ago
title=Strong types for strong interfaces - Jonathan Boccara - Meeting C++ 2017
description=Strong types are such a powerful tool for bringing expressiveness
description=into code. By using them we can precisely state our intentions
description=about our C++ objects, both to compilers and to human beings...
If needed, there's the possibility to do no wrapping of text lines at all by
using option `--no-wrap':
$ youtube-data --id=@meet --show-recent=2 --list --no-wrap
videoId=vDmQlIeY4z0
publishedAt=4 hours 55 mins ago
title=There is a new future - Felix Petriconi - Meeting C++ 2017
description=Futures were added to C++11 as high level abstractions for asynchr...
videoId=WVleZqzTw2k
publishedAt=1 day 4 hours ago
title=Strong types for strong interfaces - Jonathan Boccara - Meeting C++ 2017
description=Strong types are such a powerful tool for bringing expressiveness ...
Let see now a few more use-cases that produce the information attached to given
video entries. Will pick the first video ID -- 'vDmQlIeY4z0' -- seen printed out
in the tables and lists above and will issue a few more 'youtube-data' commands.
Option `--table' becomes an action option -- i.e. it determines the output of
'youtube-data' -- when not used in tandem with action options `--[w]diff-recent'
or `--show-recent':
$ youtube-data --video=vDmQlIeY4z0 --table
UCJpMLydEYA08vusDkq3FmjQ 2018-01-24T09:59:23.000Z PT1H1M58S There is a new...
$ youtube-data --video=vDmQlIeY4z0 --table -b
UCJpMLydEYA08vusDkq3FmjQ 5 hours 8 mins ago PT1H1M58S There is a new...
The same is true for option `--list': it is also an action option if not paired
with action options `--*-recent'. The command below shows that `--list' does not
wrap text lines by default:
$ youtube-data --video=vDmQlIeY4z0 --list
channelId=UCJpMLydEYA08vusDkq3FmjQ
publishedAt=2018-01-24T09:59:23.000Z
duration=PT1H1M58S
title=There is a new future - Felix Petriconi - Meeting C++ 2017
description=Futures were added to C++11 as high level abstractions for asynchr...
description=
description=Therefore Sean Parent and I have developed an alternative, open so...
description=
description=In the first half of my talk I want to highlight those problems an...
description=
description=A graph of futures and continuations is for single execution only ...
description=
description=In the second half of my talk I want to give a brief introduction ...
But option `-w|--wrap' do wrap text lines when specified on a `--list' command
line. The difference with respect to options `--*--recent' is due to the fact
that wrapping is default behavior for the latter options, while it is not for
the former one.
$ youtube-data --video=vDmQlIeY4z0 --list -w
channelId=UCJpMLydEYA08vusDkq3FmjQ
publishedAt=2018-01-24T09:59:23.000Z
duration=PT1H1M58S
title=There is a new future - Felix Petriconi - Meeting C++ 2017
description=Futures were added to C++11 as high level abstractions for
description=asynchronous operations and are planed to be enhanced with
description=the upcoming C++17 TS. Severals think today that std::future
description=is broken in multiple ways.
description=
description=Therefore Sean Parent and I have developed an alternative,
description=open source, concurrency library that addresses these
...
When one needs a particular property of a video entry to be printed out, than he
has to use a corresponding action option in the invoking command line. E.g, for
'description' property, that action option would obviously be `--description':
$ youtube-data --video=vDmQlIeY4z0 --description
Futures were added to C++11 as high level abstractions for asynchronous operat...
Therefore Sean Parent and I have developed an alternative, open source, concur...
In the first half of my talk I want to highlight those problems and illustrate...
A graph of futures and continuations is for single execution only and would ha...
In the second half of my talk I want to give a brief introduction into channel...
By default, the text lines produced by action option `--description' are not
wrapped. Specifying though `-w|--wrap[=NUM]' will impose the wrapping behavior:
$ youtube-data --video=vDmQlIeY4z0 --description -w
Futures were added to C++11 as high level abstractions for
asynchronous operations and are planed to be enhanced with
the upcoming C++17 TS. Severals think today that std::future
is broken in multiple ways.
Therefore Sean Parent and I have developed an alternative,
open source, concurrency library that addresses these
...
4. Appendix: Building Json-Type, 'youtube-data.so' and 'youtube-data-litex.so'
==============================================================================
This section describes the procedures of obtaining Json-Type's binaries 'json',
'json.so' and 'json-litex.so' and Youtube-Data's type library 'youtube-data.so'
and path library 'youtube-data-litex.so'.
Note that 'youtube-data*.so' are not strictly necessary for the shell function
'youtube-data' in the shell script 'youtube-data.sh' to work OK. However, it's
recommended for one to actually build them, since using them will significantly
improve the responsiveness of 'youtube-data' at the 'bash' command line prompt.
Not having 'youtube-data.so' at hand, each time called by 'youtube-data', the
'json' program (to be accurate: the library 'json.so') has to parse the type
definitions in some '*.json' file for to construct internal data structures.
The library 'youtube-data.so' has all these data structures already built-up
-- thus making 'json' run faster.
Likewise, not having 'youtube-data-litex.so' present, makes the 'json' program
(i.e. the library 'json-litex.so') run slower, since it will have to construct
its internal structures from the litex definitions in some '*-litex.json' file.
The library 'youtube-data-litex.so' contains all these data structures as such.
For to obtain the Json-Type binaries 'json', 'json.so' and 'json-litex.so' one
needs to download the latest source release tarball from [4]. The releases of
Json-Type are signed with GnuPG, therefore upon downloading e.g. version 0.12.1
by means of the following two commands:
$ wget https://download.savannah.gnu.org/releases/json-type/json-type-0.12.1.tar.bz2
$ wget https://download.savannah.gnu.org/releases/json-type/json-type-0.12.1.tar.bz2.sig
one has to issue two instances of 'gpg' program to check the integrity of the
received tarball (GnuPG FAQ [5]):
$ gpg --recv-keys 3D805B7A
$ gpg --verify json-type-0.12.1.tar.bz2.sig
Of the two commands above, the former imports into the local keyring Json-Type
author's key, while the latter verifies the package signature -- thus validating
its integrity.
After unpacking the downloaded release tarball, for building the needed binaries
one must apply the procedure described in the README file to be found in the top
directory of the source tree.
Upon a successful build, for to make accessible to Youtube-Data the binaries
obtained -- '$JSON_TYPE_HOME/src/json', '$JSON_TYPE_HOME/lib/json.so' and
'$JSON_TYPE_HOME/lib/json-litex.so' --, one should define the following shell
function wrapper:
$ json() { local JSON_TYPE_HOME=...; env LD_LIBRARY_PATH="$JSON_TYPE_HOME"/lib "$JSON_TYPE_HOME"/src/json "$@"; }
where '$JSON_TYPE_HOME' points to the top directory of Json-Type's source tree.
Instead of defining the 'json' shell function each time using Youtube-Data, it
is more convenient to install the three binaries in the system directories, e.g.:
$ export JSON_TYPE_HOME=...
$ sudo install -t /usr/local/bin "$JSON_TYPE_HOME"/src/json
$ sudo install -t /usr/local/lib "$JSON_TYPE_HOME"/lib/json.so
$ sudo install -t /usr/local/lib "$JSON_TYPE_HOME"/lib/json-litex.so
Now that 'json', 'json.so' and 'json-litex.so' are ready to use, it's time to
show how to obtain the libraries 'youtube-data.so' and 'youtube-data-litex.so'.
For building 'youtube-data.so', will follow the steps documented by Json-Type's
README file, subsection 4.d, Make a compiled (shared object) type library. The
sole difference occurs with respect to the target name: 'youtube-data' instead
of 'jsonrpc'.
Step 1: obtain 'youtube-data.mk' and 'youtube-data.c' from Json-Type:
$ cp "$JSON_TYPE_HOME"/lib/json-type-module.c youtube-data.c
$ cp "$JSON_TYPE_HOME"/lib/json-type-module.mk youtube-data.mk
Step 2: generate 'youtube-data.def' out of '*.json' files using shell function
'youtube-data-json' sourced in the current 'bash' environment from shell script
'youtube-data.sh':
$ . youtube-data.sh
$ youtube-data-json|json -Td --sizes-lib-ptr-space-size=2M > youtube-data.def
Note that the option `--sizes-lib-ptr-space-size' is actually required, because
the type definitions produced by shell function 'youtube-data-json' have a size
that overruns the default size of one the memory pools of 'json.so':
$ youtube-data-json|json -Td > youtube-data.def
json.so: fatal error: ptr-space pool alloc failed
Step 3: make 'youtube-data.so' out of 'youtube-data.{c,def}':
$ make -f youtube-data.mk
Note that '$JSON_TYPE_HOME' needs to be defined prior the call to 'make'. If it
is not defined in the current environment, then pass it in explicitly:
$ make -f youtube-data.mk JSON_TYPE_HOME=...
If everything went OK -- one should expect to get neither warnings nor errors
out of 'make' --, the shared library 'youtube-data.so' is ready for its first
use:
$ ./youtube-data.so
youtube-data.so: library version: x.y.z
As seen, the library is executable by itself, producing on output the version
numbers 'x.y.z' of the 'json.so' library that produced it in the first place.
Another immediate use of 'youtube-data.so' would be that of asking 'json.so' to
verify that it accepts the freshly built library:
$ json -Tc ./youtube-data.so && echo OK
OK
For building 'youtube-data-litex.so', will follow the exact steps presented in
'doc/liter-expr.txt', section 4, Json-Litex's Compiled Path Libraries. There is
only one change applied compared to the original procedure: the target name used
below is 'youtube-data-litex' instead of 'jsonrpc-litex'.
Step 1: obtain 'youtube-data-litex.mk' and 'youtube-data-litex.c' from Json-Type:
$ cp "$JSON_TYPE_HOME"/lib/json-litex-module.mk youtube-data-litex.mk
$ cp "$JSON_TYPE_HOME"/lib/json-litex-module.c youtube-data-litex.c
Step 2: generate 'youtube-data-litex.def' out of '*-litex.json' files using shell
function 'youtube-data-litex' (already sourced in the current 'bash' environment
from shell script 'youtube-data.sh'):
$ youtube-data-litex|json -F -- json-litex.so -D > youtube-data-litex.def
Step 3: make 'youtube-data-litex.so' out of 'youtube-data-litex.{c,def}':
$ make -f youtube-data-litex.mk
As above, '$JSON_TYPE_HOME' needs to be defined in the current environment prior
the call to 'make'. If it is not, then pass it in explicitly:
$ make -f youtube-data-litex.mk JSON_TYPE_HOME=...
If everything went OK -- one should expect to get neither warnings nor errors
out of 'make' --, the shared library 'youtube-data-litex.so' is ready for its
first use:
$ ./youtube-data-litex.so
youtube-data-litex.so: litex library version: x.y.z
As seen, the library is executable by itself, producing on output the version
numbers 'x.y.z' of the 'json-litex.so' library that produced it.
The second immediate use of 'youtube-data-litex.so' is to ask 'json-litex.so' to
verify that it accepts the freshly built path library:
$ json -F -- json-litex.so -V ./youtube-data-litex.so && echo OK
OK
If the two validations above succeeded -- that of type library 'youtube-data.so'
against 'json.so' and, respectively, that of path library 'youtube-data-litex.so'
against 'json-litex.so' --, then these two newly built libraries are ready to be
used by the 'json' program within the shell function 'youtube-data'.
5. Appendix: Using Shell Function 'youtube-data'
================================================
The shell function 'youtube-data' is the main function defined by shell script
'youtube-data.sh'. (All other shell functions in 'youtube-data.sh' are auxiliary
tools for 'youtube-data' -- and are not meant to be used directly by the user.)
This function is a rich-featured tool, having quite a few command line options:
$ funchelp -f youtube-data.sh -c youtube-data --long-wrap-join=auto --wrap=84
actions:
-U|--url produce resource's API URL
-I|--info produce resource's cache age and hash file name
-H|--hash produce resource's hash file names (previous and
current versions) after eventually updating them
according to the specifier given by `--cache=SPEC'
-J|--json produce resource's JSON text
-S|--json2 produce resource's flattened JSON text
-D|--diff[=FILE] diff two resources (the other one is given by
`--file=FILE' or `--id=ID'); the default FILE is
'+', i.e. the one generated by action `-J'
-P:|--list[=KEYS] extract resource's content as list or table; KEYS
--table[=KEYS] are comma-separated lists of JSON key names;
--channel-id `-P?', `--list=?' or `--table=?' prints out the
--playlist-id valid names corresponding to the pair of resource
--video-id type and link type given and exits (default)
--published-at
--description
--duration
--title
-V:|--show-recent[=NUM] extract resource's content for the most recent NUM
--diff-recent[=NUM] video items; the output produced is specified by
--wdiff-recent[=NUM] the print options `-P', `--list', `--table', etc;
--show-uploads[=NUM] `--[w]diff-{recent,uploads}' produce a 'diff' or
--diff-uploads[=NUM] 'wdiff' between previous version of the
--wdiff-uploads[=NUM] resource's cached content and the one specified
by `--cache=SPEC'; the options `--relative-date',
`--wrap-text' and `--color=auto' are implied if
not overridden; NUM is at most the argument of
`--max-results' and it is 5 by default; the
`--*-recent' options are using Google's API
'search' endpoint, while the `--*-uploads'
options are using the 'playlist-items' endpoint
queried for channel's uploads playlist
options:
-a|--[user-]agent=STR user agent to be passed to 'wget' when action is
`-J' or when input type is 'id' (default: '+',
i.e. $YOUTUBE_DATA_AGENT)
-b|--relative-date[=NUM] produce relative dates on output or otherwise do
--no-relative-date not; when given NUM, compute dates relative to
NUM as number of seconds since the Epoch
(default: do for `-I' or else do not)
-c:|--cache=SPEC use resource's locally cached files according to
--no-cache the given specifier or, otherwise, do not use the
cache at all, but query the remote party for
content; SPEC is any of 'previous', 'local',
'current', 'auto' (default) or 'refresh';
'previous' means using resource's previous
version locally cached file; 'local' means using
resource's locally cached file -- in any case do
not call the remote party; 'current' means using
resource's current version locally cached file if
that file exists already -- if not, get it from
the remote party; 'auto' means using resource's
locally cached file if and only if the respective
file has not yet reached its aging threshold;
each aged cache file is refreshed the first time
its associated resource is requested; 'refresh'
makes each cached file to be refreshed prior to
using it further regardless of it not exceeding
its aging threshold yet; the aging threshold is
specified by $YOUTUBE_DATA_CACHE_THRESHOLD (which
is of form '[0-9]+[dhms]?'); the short option
`-c' accepts shortcut arguments too: '-', '~',
'#', '=', '+' and '!' for `--no-cache', `-c
previous', `-c local', `-c current', `-c auto'
and `-c refresh' respectively
-e|--output-mode=MODE output mode if output file already exists: 'error',
'append' or 'overwrite' with corresponding
shortcuts '-', '+' and '=' (default: '-' i.e.
'error')
-f|--[no-]check-certif pass `--no-check-certif' to 'wget' or otherwise do
not (default not)
-g|--debug pass `--debug' to 'wget'
-h|--home=DIR home dir (default: '+', i.e. $YOUTUBE_DATA_HOME)
-i:|--file[=FILE] input type -- 'file' or 'id' -- and input name (the
--id[=ID] default type is 'id')
-k|--[header-]cookie=STR header cookie to be passed to 'wget'; when '+' take
the cookie from $YOUTUBE_DATA_COOKIE
-l:|--itself linked resource's type: 'itself', 'playlists',
--playlists 'videos' or 'uploads' (default: 'itself')
--videos
--uploads
-m|--[max-]results=NUM max results per API query (value in [1..50],
default: 50)
-n:|--color=always|auto colorize the output produced by `--list' and
--no-color `--[w]diff-{recent,uploads}' as specified, or
otherwise do not; the short option `-n' accepts
shortcut arguments too: '-', '+' and '!' for
`--no-color', `-n auto' and `-n always'
respectively (default: do not for `-P' and 'auto'
for `-V' and `-D', i.e. color only if stdout is a
terminal)
-o|--output-file=STR output file name when action is `-J|--json'
(default: '+', i.e. generate a name; '+SUFFIX'
means appending '.SUFFIX' to the generated name)
-p|--page=SPEC resource's pages specifier: '-' means the first
page only, '+NUM' means first NUM >= 1 pages
only, '+' means all pages and '@TOKEN' means the
page identified by TOKEN (default: '+')
-q|--quiet be quiet: when action is `-J|--json' do not print
out the output file name
-r:|--channel[=INPUT] resource's type: 'channel', 'playlist' or 'video'
--playlist[=INPUT] (default: 'channel'); INPUT is either a file name
--video[=INPUT] or an id (default), determined by `--file' or
`--id'; when INPUT terminates with '.json',
consider it to be file name; when INPUT ends with
'{,-playlists,-videos,-uploads}.json', set the
linked type accordingly
-s|--shortcuts=FILE channel and playlist shortcuts file name (default:
'+', i.e. $YOUTUBE_DATA_SHORTCUTS)
-t|--[no-][json-]type do not type-check resource's JSON object, i.e. do
not pass either of the options `-t $h/*.json' or
`-t $h/youtube-data.so', and respectively,
neither `-f -- json-litex.so $h/*-litex.json' nor
`-f -- json-litex.so $h/youtube-data-litex.so' to
'json', or otherwise do (default do)
-u|--unified=NUM pass `-u NUM' to 'diff' when action is `-D|--diff'
or `-V|--diff-{recent,uploads}'
-v|--[no-]validate do not validate resource's parameter values or
otherwise do (default do)
-w|--wrap[-text][=NUM] wrap 'title' and 'description' texts when output is
--no-wrap[-text] of type 'list' or otherwise do not (default: do
not for `-P' and do for `-V'; NUM must be >= 13;
its default value is 72)
-y|--[app-]key=STR Google API application key (default: '+', i.e.
$YOUTUBE_DATA_APP_KEY)
-z:|--hyperlinks=always|auto produce hyperlinks on output as specified, or
--no-hyperlinks otherwise do not; the short option `-z' accepts
shortcut arguments too: '-', '+' and '!' for
`--no-hyperlinks', `-z auto' and `-z always'
respectively (default: 'auto', i.e. have
hyperlinks only if stdout is a terminal)
The help information seen above is pretty much self-explanatory. Have to add
that 'youtube-data' produces output on stdout or in named files, depending on
which mode of operation was chosen by the user -- equally: depending on which
action option was last specified in the invoking command line; 'youtube-data'
reads input from stdin or named files only when issued with options `--file'
or `--diff'.
Shell function 'youtube-data's most important helper is 'youtube-wget' -- also
a shell function defined in 'youtube-data.sh'. The user need not use this latter
function at all. Nevertheless, if one wants to hack around the behavior of shell
function 'youtube-data' (e.g. using option `-d'), then he inevitably will come
across function 'youtube-wget' too:
$ youtube-data --video=vDmQlIeY4z0 -p- -d
youtube-wget \
--no-check-certif \
--output-document=- \
--user-agent="$YOUTUBE_DATA_AGENT" \
--action=content \
--home="$YOUTUBE_DATA_HOME" \
--threshold="$YOUTUBE_DATA_THRESHOLD" \
--keep-prev \
--save-key \
--url='...'|
json --json2 -Vusyp -t "$YOUTUBE_DATA_HOME"/youtube-data.so:videos -f -- \
json-litex.so -p "$YOUTUBE_DATA_HOME"/youtube-data-litex.so:videos -l|
awk -F '\n' --re-interval '...'
The command line options of 'youtube-wget' are as shown below:
$ funchelp -f youtube-data.sh -c youtube-wget --long-wrap-join=auto
actions:
-H|--action=hash produce cache file name -- 'wget' may be implied
-I|--action=info produce cache age and file name -- no 'wget'
implied
-O|--action=content produce content -- 'wget' may be implied (default)
options:
-a:|--hash-age[=NUM] pass `--hash-age[=NUM]' to 'cache' when action is
--no-hash-age `-I' or otherwise do not (default not)
-c|--[no-]cache-only use only locally cached files when action is not
`-I' or otherwise do not (default not)
-e|--[no-]save-key pass `--save-key' to 'cache' when action is not
`-I' or otherwise do not (default not)
-h|--home=DIR home dir (default: '+', i.e. $YOUTUBE_DATA_HOME)
-g|--[no-]log-cache do not pass `-lvv' to 'cache' or otherwise do
(default do)
-k|--[no-]keep-prev pass `--keep-prev' to 'cache' when action is not
`-I' or otherwise do not (default not)
-l|--[no-]log-wget do not pass `--append-output=$h/.cache/wget.log'
to 'wget' or otherwise do (default do)
-t|--[cache-]threshold=VAL cache threshold value ('inf' or of form
'[0-9]+[dhms]?'; default: '16h')
-w:|--debug options to be passed as such to 'wget'
--quiet
--no-check-certif
--connect-timeout=NUM
--header=STR
--output-document=FILE
--user-agent=STR
-u|--url=STR input URL
The shell function 'youtube-wget' has three modes of operation -- corresponding
each to one of its action options. These three action options determines the
output provided: either a hash file name or a pair consisting of cache age and
hash file name or hash file content. As mentioned by the help information above,
depending on the invoking action, 'youtube-wget' may or may not call the remote
API on the given URL. The shell function is using the programs 'cache' and 'wget'
for to achieve its duties.
6. References
=============
[1] Google Developers Console
https://console.developers.google.com/
[2] YouTube Data API
https://developers.google.com/youtube/v3/
[3] Json-Type: JSON Push Parsing and Type Checking
http://nongnu.org/json-type/
[4] Json-Type's Release Tarballs
https://download.savannah.gnu.org/releases/json-type/
[5] GnuPG FAQ: How can I use GnuPG to verify a file I've downloaded?
https://www.gnupg.org/faq/gnupg-faq.html#how_do_i_verify_signed_packages