This file describes the interface between rsyslog and external output plugins.
Rsyslog uses stdin and stdout to communicate with the external plugin. This is an established mode of interprocess communication and well supported by all decent languages. Parameters, if any, will be passed in via command line arguments, which should also be easy to obtain in (almost) all languages. Where this is not the case, it is suggested to either use an external config file or hardcode the parameters inside the plugin code.
Rsyslog pushes messages via stdin. Each message is terminated by a LF. So a plugin can obtain a full message simply by reading a line.
This can cause problems with multi-line messages. There are some cures for this. The recommended one is to use JSON message format (more on message formats below). This will encode LF as "\n" (by JSON RFC requirements) and thus will ensure there are no embedded LFs even in multiline messages. An alternative is to use a message format which contains some other delimiter and program the plugin to watch for this delimiter. With the near-universal availability of JSON libraries in languages these days, we strongly think that the JSON approach is superior.
The message format is generated by standard rsyslog methods, that is via a template. This gives full flexibility over what the plugin is fed. Unfortunately this also means the necessary template definitions are needed. See the rsyslog doc for how to do that (in the future, this file will also contain some samples).
The plugin may convey error information to rsyslog. To do this, set the
confirmMessages flag to on in the omprog action configuration (this flag
is disabled by default). When this flag is enabled, rsyslog will wait for a
confirmation from the plugin after sending each log message to it.
The plugin must confirm the message by writing a line with the word OK to
its standard output. That is, the plugin must write the characters O, K and
LF (line feed) to stdout.
If the plugin writes a line to stdout containing anything else (for example,
the string Error: could not connect to the database followed by a LF), rsyslog
will consider that the plugin has not been able to process the message. The
message will be retried later, according to the retry settings configured for
the omprog action (in particular, the action.resumeInterval setting). If the
reportFailures flag is set to on, rsyslog will also log the returned error
line.
If the plugin terminates, the message is also considered as non-processed. The plugin will later be restarted, and the message retried, according to the configured retry settings.
When starting the plugin, if confirmMessages is on, rsyslog will also wait
for the plugin to confirm its initialization. The plugin must write an OK line
to stdout just after starting. If it writes anything else or terminates, rsyslog
will consider the plugin initialization has failed, and will try to restart it
later. It is recommended to ensure that the program is really ready to start
receiving messages before emitting this first confirmation (for example, by
checking that a destination database is accessible).
The following sequence illustrates the message exchanges between rsyslog and the
plugin. A right arrow (=>) indicates a message read by the plugin from its
stdin, and a left arrow (<=) indicates a message written by the plugin to its
stdout. Note that the arrows themselves are not read or written. Each line is
terminated by a LF (\n).
<= OK
=> log message 1
<= OK
=> log message 2
<= OK
...
=> log message N
<= OK
Note that the first OK does not confirm any message, but that the plugin has
correctly started and is ready to receive messages. When the plugin receives
an end-of-file (EOF), it must silently terminate.
The plugin must confirm each message before a timeout of 10 seconds. A different
timeout can be configured using the confirmTimeout setting. If the plugin does
not return anything to rsyslog before this timeout, rsyslog will restart the
program, and retry the message.
It is important that the confirmTimeout be adjusted to an appropriate value for
your plugin, since a too low timeout can cause duplicate processing of messages
(if rsyslog decides that the plugin has gone unresponsive when it really hasn't).
In case of doubt, consider either a) setting a rather high timeout (but note that
this can go against the fail fast pattern), b) enabling the signalOnClose
and/or killUnresponsive flags in the configuration, to ensure that the plugin
will stop its processing if rsyslog decides to restart it, or c) using keep-alive
feedback as explained below.
The plugin can also provide keep-alive feedback to rsyslog. This allows the
plugin to exceed the confirmTimeout for certain messages requiring a long-
running processing (for example, attempting to reconnect to a database after a
temporary loss of connection), without having to increase too much the configured
timeout in anticipation of those peak latencies.
To provide keep-alive feedback, the plugin must periodically write a dot (.)
to stdout, without ending the line. Each dot must be written before the timeout
is reached (rsyslog will restart the timeout after receiving each dot). Once the
plugin completes the processing of the message, it must write the OK word (or
an error message) as usual, followed by a line feed. rsyslog will ignore all
leading dots when processing the received line.
The following sequence illustrates the use of the keep-alive feedback:
<= OK
=> log message 1
<= OK
=> log message 2
<= .......OK
=> log message 3
<= ..OK
=> log message 4
<= ...OK
The plugin can also provide keep-alive feedback during its initialization, before
emitting the first OK. This can be useful to give enough time to the plugin to
complete its initialization checks.
Aside from confirming messages via stdout, at any moment the plugin may write
anything it wants to stderr. The output setting of the omprog action allows
capturing the plugin's stderr to a file. This provides an easy way for the
plugin to record its own logs in case something fails (for example, logging the
details of a database connection error).
To prevent the output file from growing too much over time, it is recommended to periodically rotate it using a tool like logrotate. After each rotation of the file, a HUP signal must be sent to rsyslog. This will cause rsyslog to reopen the file.
When multiple instances of the program are running concurrently (see Threading Model), rsyslog guarantees that the lines written to stderr by the various instances will not appear intermingled in the output file, as long as: 1) the lines are short enough (the actual limit depends on the platform: 4KB on Linux, and at least 512 bytes on other systems), and 2) the program writes each line at a time, without buffering multiple lines. (Commonly, this can be accomplished by either flushing the stream after each line, writing to the stream in line-buffered mode, or doing a single write in case the stream is unbuffered. Which of these alternatives is the easiest or most natural to use depends on the language and libraries the program is coded in.)
If the output setting is not specified, the plugin's stderr will be ignored
(it will be redirected to /dev/null).
When confirmMessages is set to off, the output setting will capture both
the stdout and stderr of the plugin to the specified file (and if omitted, both
stdout and stderr will be redirected to /dev/null). The same considerations
regarding the rotation of the file and the consistency of the lines apply in
this case.
See this Python plugin skeleton for a featured example on how a plugin can provide feedback to rsyslog.
You can write a plugin that processes the messages in batches (also called transactions), instead of individually. For a general explanation on how rsyslog handles the batching of messages, see dev-oplugins.
Warning: This feature is currently experimental. It could change in future releases without keeping backwards compatibility with existing configurations or the interface described below.
To enable transactions, set the useTransactions flag to on in the omprog
action configuration. When this flag is enabled, rsyslog will send a special
message line to the plugin's stdin to indicate that a batch of log messages is
going to be sent. This special message is BEGIN TRANSACTION by default, although
it can be customized using the beginTransactionMark setting of omprog.
After the BEGIN TRANSACTION line, rsyslog will send the log messages in the
batch, each one in its own line, and then another special message COMMIT TRANSACTION to indicate the batch has ended. (The later can be customized via
the commitTransactionMark setting.)
That is:
BEGIN TRANSACTION
log message 1
log message 2
...
log message N
COMMIT TRANSACTION
BEGIN TRANSACTION
...
COMMIT TRANSACTION
BEGIN TRANSACTION
...
COMMIT TRANSACTION
...
(with a LF at the end of each line)
When transactions are enabled, rsyslog will always send log messages within a transaction block, never outside it, even when the batch consists of a single message.
You can enable both the useTransactions and confirmMessages settings in
the omprog action, only one of them, or none of them. When both settings
are set to on, the plugin must confirm the BEGIN TRANSACTION and COMMIT TRANSACTION messages, and the log messages within the transaction. The log
messages within the transaction can be confirmed with any of the following
status codes (which the plugin must write to stdout):
OKDEFER_COMMITPREVIOUS_COMMITTED
Refer to https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html
for an explanation on the meaning of these status codes. You will typically
need to return the DEFER_COMMIT status code, since the other codes imply a
partial commit, and do not guarantee that the COMMIT TRANSACTION will be
received.
The following sequence illustrates the exchanges between rsyslog and the plugin
when transactions and message confirmations are enabled, and the plugin
confirms the log messages within each transaction with DEFER_COMMIT:
<= OK
=> BEGIN TRANSACTION
<= OK
=> log message 1
<= DEFER_COMMIT
=> log message 2
<= DEFER_COMMIT
...
=> log message 5
<= DEFER_COMMIT
=> COMMIT TRANSACTION
<= OK
=> BEGIN TRANSACTION
<= OK
=> log message 6
<= DEFER_COMMIT
=> log message 7
<= DEFER_COMMIT
...
=> log message 10
<= DEFER_COMMIT
=> COMMIT TRANSACTION
<= OK
Write your plugin as you would do in a single threaded environment. Rsyslog automatically detects when it is time to spawn additional threads. If it decides so, it will also spawn another instance of your script and feed it concurrently. Note that rsyslog will also terminate instances that it knows are no longer needed.
If your plugin for some reason cannot be run in multiple instances, there are ways to tell rsyslog to work with a single instance. But it is strongly suggested to not restrict rsyslog to do that. Multiple instances in almost all cases do NOT mean any burden to the plugin developer. Just think of two (or more) independent instances of your program running in different console windows. If that is no problem, rsyslog running multiple instances of it is also no problem.
Interfaces for external input, filter and message modification plugins are planned. Most probably, they will become available in the order mentioned in the last sentence.
The external plugin will use stdin to receive the message that it potentially can modify. The message will be LF-terminated, and no LF must be present within the message itself. By default, the MSG part of the message is provided as input. The "interface.input" parameter can be used to modify this. It may has the following values:
- "msg" (the default)
- "rawmsg", which is the complete message (including header) as received by rsyslog
- "json", which is the complete message object (with all properties broken out) as a JSON object string. This is the "jsonmesg" dynamic message property.
Note: if multi-line messages are to be processed, JSON representation must be used, otherwise errors will happen.
The ability to use non-JSON representations is primarily a performance enhancement. Building the JSON representation causes some overhead, and very often access to either msg or rawmsg is fully sufficient.
The plugin will emit a JSON representation of those properties that need to be modified and their new values to stdout. Again, this is delimited by LF, with no LF permitted inside the JSON representation. Only properties that are to be changed must be included in the response. Unchanged properties should NOT be included in the response, as this would increase processing cost. If no property is to be modified, an empty JSON representation is to be provided.
The plugin must emit one response line for each message (line) received, and must do so in the same order in which the messages were put in stdin. Note that like the output module interface, multiple instances of the plugin may be activated. See above for more information.
Most message properties can be modified. Modifiable are:
- rawmsg
- msg
- syslogtag
- syslogfacility
- syslogseverity
- msgid
- procid
- structured-data
- hostname (aliased "source")
- fromhost
- fromhost-ip
- all message variable ("$!" tree)
If the message variable tree is modified, new variables may also be added. Deletion of message variables is not directly supported. If this is desired, it is suggested to set the variable in question to the empty string ("").
The plugin interface is implemented via the "mmexternal" native plugin. See its documentation on how to tie your plugin into rsyslog's processing flow.