/
README.txt
195 lines (137 loc) · 5.32 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
======
Beaver
======
python daemon that munches on logs and sends their contents to logstash
Requirements
============
* Python 2.7 (untested on other versions)
* Optional zeromq support: install libzmq (``brew install zmq`` or ``apt-get install libzmq-dev``) and pyzmq (``pip install pyzmq==2.1.11``)
Installation
============
Using PIP:
From Github::
pip install git+git://github.com/josegonzalez/beaver.git#egg=beaver
From PyPI::
pip install beaver==7
Usage
=====
usage::
beaver [-h] [-m {bind,connect}] [-p PATH] [-f FILES [FILES ...]]
[-t {rabbitmq,redis,stdout,zmq,udp}] [-c CONFIG] [-d DEBUG]
optional arguments::
-h, --help show this help message and exit
-m {bind,connect}, --mode {bind,connect}
bind or connect mode
-p PATH, --path PATH path to log files
-f FILES [FILES ...], --files FILES [FILES ...]
space-separated filelist to watch, can include globs
(*.log). Overrides --path argument
-t {rabbitmq,redis,stdout,zmq}, --transport {rabbitmq,redis,stdout,zmq}
log transport method
-c CONFIG, --configfile CONFIG
ini config file path
-d DEBUG, --debug DEBUG
enable debug mode
Background
==========
Beaver provides an lightweight method for shipping local log files to Logstash. It does this using either redis, stdin, zeromq as the transport. This means you'll need a redis, stdin, zeromq input somewhere down the road to get the events.
Events are sent in logstash's ``json_event`` format. Options can also be set as environment variables.
Examples
--------
Example 1: Listen to all files in the default path of /var/log on standard out as json::
beaver
Example 2: Listen to all files in the default path of /var/log on standard out with msgpack::
BEAVER_FORMAT='msgpack' beaver
Example 3: Listen to all files in the default path of /var/log on standard out as a string::
BEAVER_FORMAT='string' beaver
Example 4: Sending logs from /var/log files to a redis list::
REDIS_URL="redis://localhost:6379/0" beaver -t redis
Example 5: Use environment variables to send logs from /var/log files to a redis list::
REDIS_URL="redis://localhost:6379/0" BEAVER_PATH="/var/log" BEAVER_TRANSPORT=redis beaver
Example 6: Zeromq listening on port 5556 (all interfaces)::
ZEROMQ_ADDRESS="tcp://*:5556" beaver -m bind -t zmq
# logstash config:
input { zeromq {
type => 'shipper-input'
mode => 'client'
topology => 'pushpull'
address => 'tcp://shipperhost:5556'
} }
output { stdout { debug => true } }
Example 7: Zeromq connecting to remote port 5556 on indexer::
ZEROMQ_ADDRESS="tcp://indexer:5556" beaver -m connect -t zmq
# logstash config:
input { zeromq {
type => 'shipper-input'
mode => 'server'
topology => 'pushpull'
address => 'tcp://*:5556'
}}
output { stdout { debug => true } }
Example 8: Real-world usage of Redis as a transport::
# in /etc/hosts
192.168.0.10 redis-internal
# From the commandline
REDIS_NAMESPACE='app:unmappable' REDIS_URL='redis://redis-internal:6379/0' beaver -f /var/log/unmappable.log -t redis
# logstash indexer config:
input { redis {
host => 'redis-internal' # this is in dns for work
data_type => 'list'
key => 'app:unmappable'
type => 'app:unmappable'
}}
output { stdout { debug => "true" }}
As you can see, ``beaver`` is pretty flexible as to how you can use/abuse it in production.
Example 9: RabbitMQ connecting to defaults on remote broker::
# From the commandline
RABBITMQ_HOST="10.0.0.1" beaver -t rabbitmq
# logstash config:
input { amqp {
name => "logstash-queue"
type => "direct"
host => "10.0.0.1"
exchange => "logstash-exchange"
key => "logstash-key"
exclusive => false
durable => false
auto_delete => false
}}
output { stdout { debug => "true" }}
Example 10: Read config from config.ini and put to stdout::
# From the commandline
beaver -c config.ini -t stdout
# config.ini content:
[/tmp/somefile]
type: mytype
tags: tag1,tag2
add_field: fieldname1,fieldvalue1[,fieldname2,fieldvalue2, ...]
[/var/log/*log]
type: syslog
tags: sys
Example 11: UDP transport::
# From the commandline
UDP_HOST="127.0.0.1" UDP_PORT="9999" beaver -t udp
# logstash config:
input {
udp {
type => "shipper-input"
host => "127.0.0.1"
port => "9999"
}
}
output { stdout { debug => "true" }}
Todo
====
* Use python threading + subprocess in order to support usage of ``yield`` across all operating systems
* Fix usage on non-linux platforms - file.readline() does not work as expected on OS X. See above for potential solution
* More transports
* ~Create a python package~
* ~Ability to specify files, tags, and other metadata within a configuration file~
Credits
=======
Based on work from Giampaolo and Lusis::
Real time log files watcher supporting log rotation.
Original Author: Giampaolo Rodola' <g.rodola [AT] gmail [DOT] com>
http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
License: MIT
Other hacks (ZMQ, JSON, optparse, ...): lusis