forked from mozilla-services/heka
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES.txt
441 lines (293 loc) · 15.6 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
0.5.0 (2014-??-??)
==================
Backwards Incompatibilities
---------------------------
* The sandbox preservation data is now stored in the
{base_dir}/sandbox_preservation directory instead of with the plugin source.
On the initial restart no preserved data will be restored unless it is
manually moved to this directory first. (issue #626)
* The Heka utilities (flood, sbmgr, sbmgrload, inject) have been namespaced
with a `heka-` prefix for their respective binaries. I.e. Flood has been
renamed heka-flood., etc.
* MultiDecoder now gets its name from the TOML section name like the rest of
the plugins, instead of a separate 'name' config option.
* Major reorganization of the `pipeline` package, moving the implementation of
most plugins to sub-packages of a separate `plugins` package.
* Removed the wrapper 'table' element from the JSON serialization (issue #525)
i.e., {"table":{"value":1}} is now simply {"value":1}. The change also
removes the special '_name' metadata arribute; the top level _name element
should be created in the Lua structure if it is required.
* In the process of removing the core sandbox code from Heka (issue #464), the
sandbox was streamlined to only load the base library by default. All sandbox
plugins must now explictly load additional libraries with the require
function.
* Removed DecoderSet method from PluginHelper interface (and DecoderSet
abstraction entirely) and replaced it with Decoder and DecoderRunner methods
that return a Decoder or a DecoderRunner by name.
* Changed Decoder interface to support one input pack generating multiple
output packs.
Bug Handling
------------
* Set default StatAccumInput stat namespace prefix values even when
`legacy_namespaces` is set to false (issue #630).
* Fixed cpuprof file from being closed right after opening so no data was
being logged.
* Fixed LogfileInput so it will only override the default delimiter for a
RegexpParser if a delimiter is specified in the config.
* PluginWrapper will now check for the WantsName interface when creating a
plugin, and will set the plugin's name if appropriate.
* SandboxDecoder now explicitly logs a fatal error before triggering shutdown
to ensure error message is actually emitted.
* Message severity now defaults to the highest RFC 5424 value (i.e. 7) implies
low severity, rather than zero, which implies `emergency`, (issue #518).
* 'flood' command now outputs every send error (even 'connection refused'), and
always increments 'messages sent' count even when there is a sending error
(so setting 'num_messages' still works even if hekad stops responding,
etc.).
Features
--------
* heka-cat: A command-line utility for counting, viewing, filtering, and
extracting Heka protobuf logs.
* TcpOutput has been redesigned to handle unavailable endpoints and dropped
connections without data loss (issue #355)
* Added Logstreamer Input [LogstreamerInput]: An input that replaces the
Logfile and Logfile Directory Inputs and supports sequential reading of
logstreams that span sets of ordered logfiles. (issue #372).
* TcpInput, TcpOutput, and flood client now all support TLS encrypted TCP
connections using Go's crypto/tls package.
* Added Http Listen Input [HttpListenInput]: An input that listens for
HTTP requests on the specified address. If a decoder is not specified the
input generates a message with the HTML request body as the payload.
This input is especially useful for consuming and processing webhooks.
(Issue #431)
* Added support for local external packages (issue #393)
* Inject: A command-line utility for injecting arbitrary messages into
a Heka pipeline.
* Added Go ScribbleDecoder for efficient setting of purely static message
field values.
* Exposed `write_message` API function to Lua decoders to allow mutation of
existing decoded message (issue #577).
* HttpInput: Added urls (array) option.
* HttpInput: Failed and successful HTTP GET actions produce messages of Type
"heka.httpinput.data", Logger = Request URL, and severity appropriate to
HTTP status 200 or not. I.e. Connections responding with a status of 200
produce a message with Severity 6, non-200 statuses Severity 1. Failure to
connect produces a message with Severity 1 and Type "heka.httpinput.error".
* HttpInput: Fields[ResponseTime] populates with time duration in seconds for
HTTP GET of URL, Fields[Status] with HTTP Status, Fields[Protocol] with HTTP
protocol and version, Fields[StatusCode] with HTTP Response Status Code for
successful HTTP GETs. The Circular Buffer Graph Annotation (Alerts)
(http://hekad.readthedocs.org/en/latest/sandbox/graph_annotation.html) plugin
is compatible with the HttpInput plugin.
* HttpInput: Added success_severity and error_severity options for GET actions.
* HttpInput: Messages now set Logger, UUID, and Timestamp.
* Added log_errors option to PayloadregexDecoder to allow skipping invalid
payloads.
* Added "id" flag to elasticsearch output (issue #386).
* Added SmtpOutput (issue #472)
0.4.2 (2013-12-02)
==================
Bug Handling
------------
* Changed CPack configuration such that the 'make package' target no longer
creates deb packages, and added a new 'make deb' target that creates debs
with the filename expected by the deb package naming conventions (see
https://github.com/mozilla-services/heka/issues/545).
* Doc clarifications re: required use of ProtobufDecoder (see
https://github.com/mozilla-services/heka/issues/550).
* Explicitly exclude system level folders from those that CPack will include
as part of the Heka RPM (requires CMake >= 2.8.12) (see
https://github.com/mozilla-services/heka/issues/548).
0.4.1 (2013-11-05)
==================
Bug Handling
------------
* Updated Mozilla Sphinx theme submodule reference and configuration paths to
work around bug in ReadTheDocs rendering (see
https://github.com/rtfd/readthedocs.org/issues/529).
* Changed default DashboardOutput.StaticDirectory value to
`/usr/share/heka/dasher`, to match where the packages put the files.
Features
--------
* add require_all_fields flag to PayloadJsonDecoder
(issue #528)
0.4.0 (2013-10-30)
==================
Backwards Incompatibilities
---------------------------
* DashboardOutput now requires a `static_directory` setting to specify where
the Heka dashboard source code can be found.
* AMQPInput now only supports a single decoder to be specified (issue #414).
* UdpInput configuration now requires a parser_type and decoder to be specified
(issue #411).
* TcpInput configuration now requires a parser_type and decoder to be specified
(issue #165).
* Removed the LogfileInput decoders array configuration in favor of a single
decoder (issue #308).
* SandboxManagerFilter now auto-generates a default working directory based on
the plugin name, in the ${BASE_DIR}/sbxmgrs folder.
* LogfileInput seek journals now support a boolean `use_seek_journal` config
flag, and only the name of the journal file is now settable via
`seek_journal_name`, replacing the `seekjournal` option which specified the
full path.
* Build is now entirely cmake based and lives directly in the heka repository,
replacing the separate primarily gnu make driven separate heka-build repo.
* Core Heka plugins now default to writing data to /var/cache instead of
/var/run since /var/run is usually deleted on server restart.
* StatAccumInput now defaults to emitting stats data in the payload instead of
in the message fields.
* Renamed LoglineDecoder to PayloadRegexDecoder.
Bug Handling
------------
* Added check for nil RemoteAddr value since UDP connections won't have one
(issue #462).
* Statsd messages are now parsed manually (issue #44).
* Added mDRunner so MultiDecoder will work with sub-decoders that implement
the WantsDecoderRunner interface.
* Removed all code that attempted to catch and recover from panics in called
plugin code, since a) the recovery often left Heka in an undefined, broken
state b) the safety was a lie, any panic in a separate goroutine would not
be caught and c) panic traceback actually makes it easier to debug.
* WhisperOutput and CarbonOutput no longer require stats to use the `stats`
namespace prefix.
* LogfileInput now truncates and overwrites the seek journals instead of
appending.
* `protoc` now an optional dependency for the build, only required if the
protobuf definition is actually changed.
* MessageTemplate field name interpolation was only supporting alphabetic
characters for field names, now supports all "word" characters (i.e.
alphanumeric plus underscore).
* ElasticSearchOutput now always sends timestamps in expected UTC time zone
format.
* Fixed a CarbonOutput memory / socket leak.
* Fixed panic caused by an invalid config file path.
Features
--------
* Added a sandbox message field iterator (issue #460)
* Added ProcessInput. (issue #406)
* Replaced DashboardOutput UI with new Backbone.js-based `dasher` interface.
(issue #378)
* Add delta output to circular buffers. (issue #467)
* Added json decoding to the Lua sandbox (issue #463). If you have an old build
directory you will have to run make clean to pick up the new patch.
* Added a 'config' variable section to the Sandbox decoder/filter plugin.
(issue #444)
* Added StatsToFieldsDecoder that will convert a message with statsd formatted
string in the payload to one that also has the data available as message
fields, in the same format that would be generated by a StatAccumInput with
`emit_in_fields` set to true.
* MultiDecoder now supports `cascade_strategy` config options of `first-wins`
and `all`, defaults to `first-wins`.
* Added stream parsing to UdpInput. (issue #411)
* Added stream parsing to TcpInput. (issue #165)
* Added SandboxDecoder. (issue #370)
* Support Heka protobuf message data in the LogfileInput plugin. (issue #261)
* Added "payload" format option to ElasticSearchOutput to pass message payload
through to ES unchanged.
* Allow LogfileInput to use configurable delimiters including regexp (issue #263)
This includes a backwards incompatible change for the journal file, old
journals will not be honored and the new journal will overwrite it.
* Allow add_external_plugin to specify sub-packages. (issue #384)
* Added `base_dir` global (i.e. `[hekad]` section) config option to specify a
single location Heka plugins should use to persist data to the file system.
* The `hekad` commands `config` flag now supports use of a config directory in
addition to a single config file. If a directory path is specified, every
file in the directory will be parsed and they will all be merged to a single
running configuration.
* Added LogfileDirectoryManagerInput (issue #344).
* LPEG library added to Lua sandbox (issue #144).
* ElasticSearchOutput now supports `logstash_v0` output format that is modeled
after the original Logstash ElasticSearch message schema.
* Added PayloadJsonDecoder that can extract information from JSON text in a
message payload during decoding.
* Make index type_name option for ElasticSearchOutput definable from Field value (issue #356)
0.3.0 (2013-07-16)
==================
* Fixed default (and only supported, for now) sample rate in the stats
generated by the StatFilter.
* Fixes to the platform specific build constraints around signal handling.
* Added HttpInput that can make HTTP requests to a remote server, placing
the response body into a message payload.
* Filters and outputs now require a specified message_matcher or they will
raise an error applying the configuration.
* ticker_interval now exposed as a global config option for inputs, as well
as outputs and filters.
* Overhaul of StatAccumInput implementation to fix bugs and allow for better
handling of output format options.
* LoglineDecoder will now default to the current day, month, and year when
parsing text that contains bare timestamps w/ no date information.
* Added permission checks to FileOutput and WhisperOutput plugins to fail
gracefully when they've been asked to write to a location where they do not
have write permission.
* Fixed folder creation permission bugs and allow configurable folder
permission settings in FileOutput and WhisperOutput.
* `hekad` now emits help message if started w/ no command line options.
* Consistently use underscore_separated_words for plugin TOML configuration
options.
* Use strings instead of integers to specify octal permission values in the
config, since TOML doesn't support octal integer values.
* LoglineDecoder config now allows specification of a time zone in case the
parsed text contains non-UTC timestamps that don't include embedded time
zone information.
* Removed match capture group support from the router's message matching
functionality, since this is better done in the decoder layer where the
back-pressure won't slow down all message delivery. This gets rid of the
`PipelineCapture` object altogether, so now filter and output plugins deal
w/ PipelinePacks directly.
* Added ElasticSearch output.
* Lua filters can now emit tables, which will be serialized to JSON and
put in an injected message payload.
* Custom dashboard javascript now generated by Heka instead of hosted
remotely.
* Improvements to flood, a Heka protocol load test client.
* Config loading now fails if a config section specifies an unrecognized
configuration option.
* StatAccumulator now supports stat metric messages with stats data embedded
in message fields as well as message payload.
* Added support for plugin code to provide message_matcher and ticker_interval
default values.
* Reimplemented StatMonitor as StatAccumInput, providing a StatAccumulator
interface.
* SIGUSR1 signals now generate Heka report on stdout instead of sending
a message so it can be viewed even when message delivery is hosed.
* Added explicit Close() calls to whisperdb files in WhisperOutput.
* Redesigned message field metadata to use string specifiers instead of
an enum.
* Lua filters can now emit multiple named outputs to the Heka dashboard.
* LogfileInput now supports just one file per input.
* Removed race conditions exposed by Go's race detector.
* Improved self monitoring / dashboard output.
* Improved shutdown message flushing.
* Added Nagios output.
* Added support for LogfileInput to resume parsing where it left off.
* Added AMQP input and output plugins.
* Improved control over sandbox filter message injection restrictions.
* Added support for restartable plugins.
* Moved regular expression and capture group parsing out of the filter layer
(i.e. in the TransformFilter) and into the decoder layer (i.e.
LoglineDecoder) to prevent back pressure from impacting the entire router.
0.2.0 (2013-06-26)
==================
* Fix stat name regex to capture "." characters in the name.
0.2.0rc2 (2013-05-23)
=====================
* Fix Lua sandbox C code to work on 32 bit systems.
* Trivial release documentation ReST formatting fix.
0.2.0rc1 (2013-05-21)
=====================
* Use non-https links in docs to prevent bad rendering due to mixed http/https
content.
* A number of documentation tweaks and updates.
* Static linking of cgo wrapped Lua environment.
* Added LICENSE.txt and CHANGES.txt for better project hygiene.
* Changed default interval for log file reading from 1ms to 500ms so we don't
churn the machine in default configuration.
* Moved StatPacket channel setup into StatsdInput's Init method (from Run) to
avoid race conditions.
* Added support for mingw-based Windows build.
* Perform message injection in a goroutine to prevent blocking the router when
the plugin that is doing the injection has a full in channel.
* Added required pack recycling to TcpOutput.
0.2.0b1 (2013-04-30)
====================
* Initial public release