-
Notifications
You must be signed in to change notification settings - Fork 0
/
foreach.md
93 lines (59 loc) · 3.06 KB
/
foreach.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
jline-foreach
=============
Execute a function for each JSON line. This is roughly equivalent to gawk for JSON lines.
## Command line:
echo '{"this":24, "that":99}' | jline-foreach "console.log(record.this * record.that);"
echo '{"a":5}' | jline-foreach 'sum += record.a;' 'end::console.log(sum);' 'beginning::GLOBAL.sum=0;'
echo '{"a":"man"}' | jline-foreach 'console.log(require("fs").readFileSync(record.a));'
### Code
The flags are:
* `-q` - quiet - don't pass comments through.
* `-l` - log - start the output with a comment giving this command
Each argument is of the form: "event::code". If the event is omitted it is assumed to be `jline`. Others are:
* `beginning`/`beg`/`begin`
* `end`
* `error`
* `line` - any line, well formed or not.
The only way to share variables between separate chunks of code is to declare them global.
The preset variables are:
* `record` - the parsed JSON.
* `recordNumber` - the line number, excluding lines that are not well formed JSON.
* `line` - the raw string
* `lineNumber` - the line number.
* `error` - (when handling an error event)
All standard nodejs functions are available and in addition:
* `emit(data)` - emits a JSON line on stdout.
* `require(path)` - looks in the current working directory first for relative paths.
* `keyvals(dict, callback(key,val))` - iterates over a dictionary.
* `setpath(dict, [x1,x2,x3,...],val)` - sets `dict.x1.x2.x3=val`, creating any intermediate objects as necessary.
* `getpath(dict, [x1,x2,x3,...])` - returns `dict.x1.x2.x3` or `undefined`
* `find(dict,{options},callback(path,value){....})` Find all paths in a dictionary. Useful for converting trees into tabular form. Options are: `{maxdepth:3,prefix:["each","row","starts","with"]}`
* `tm` - the tree-math library.
### Require Modules
You can `require` any npm modules in your NODE_PATH. For example:
# First install a module; it doesn't matter where.
# I'll install one globally:
$ sudo npm install -g request-promise
...
# For me the global modules are here:
$ ls /usr/local/lib/node_modules/ | grep request-promise
request-promise
# Yours are probably in:
$ here "$(npm config get prefix)/lib/node_modules"
#
# Now use that module:
#
$ export NODE_PATH="/usr/local/lib/node_modules/"
$ echo '{"url":"http://winning.gold"}' | jline-foreach 'beg::req=require("request-promise")' 'req(record.url).then(console.log)'
<html>
<head>
...
Here is another example. MongoDB has no good way of upserting fields from the command line. But there is a node module for that:
echo '{"page":"index.html","hour":"2015-09-18T21:00:00Z","visitors":1001}' |\
jline-foreach \
'beg::dp=require("bluebird").promisifyAll(require("mongodb").MongoClient).connectAsync("mongodb://localhost:27017/nginx")' \
'dp.then(function(db){
updates = {}
updates["visitors.hour."+record.hour] = record.visitors;
db.collection("pagestats").update({_id:record.page},{$set:updates},{upsert:true});});' \
'end::dp.then(function(db){db.close()})'