Senders

Batching

Many output can batch events for improved performance. A batching output will use the following arguments:

• batchSize, batch size
• workers, the number of concurrent batches can be flushed at the same time
• flushInterval, how often to flush a batch, even if not full.

Encoding

Some senders do their own encoding, the other needs an explicit encoder, whose setup is provided in the encoder attribute.

Outputs list

loghub.senders.ElasticSearch

It can batch and does not need an encoder.

• destinations, an array of Elastic servers.
• port, the port for Elasticsearch server, default to 9200.
• protocol, http or https, default to https.
• user, username for authentication.
• password, password for authentication.
• timeout, timeout for requests, in seconds, default to 2.
• index, an expression for the index name.
• dateformat, a date format pattern, that can be used to generate index name; used only if index expression is not defined.
• type, an expression for the _type value; default to _doc.
• templateName, the name for the index template; default to loghub.
• templatePath, the path to the index template file; if templateName is null, template handling is disabled.
• withTemplate, false disable any template handling from LogHub.
• typeHandling, can take the values USING, MIGRATING, DEPRECATED; default to USING.
  • USING means the old _type field is still used.
  • MIGRATING means the old _type is still used, but the parameter include_type_name=true is added when needed.
  • DEPRECATED means that no _type is used, in template handling or bulk send. So both type and typeX settings will be ignored.
• ilm, activate the index lifecycle management (ILM) feature; if true, index existence will be checked, and it’s missing, a new index with the name indexname-000001 will be created, and an alias to indexname will be added; default to false.
• pipeline, the name of the ingest pipeline to use; default to none.
• clusterName, the name of the cluster, used for display or JMX metrics; default to none.
• elasticArguments, a map of additional arguments to pass to the Elasticsearch bulk request. It can take expression, but they are not resolved against an event, as it’s used for the bulk request.
• statusCheck, check the status of the Elasticsearch cluster before sending; default to true.

If neither type or typeX is defined, the type of documents defaults to _doc, unless of course if typeHandling is defined to DEPRECATED, in which case it's ignored.

The settings elasticArguments and statusCheck can be used to send logs to VictoriaMetrics using an Elasticsearch data ingester, see Logstash Setup, but with minor differences. One should then add the following settings for compatibility.

    loghub.senders.ElasticSearch {
        ...
        statusCheck: false,
        elasticArguments: {
            "_msg_field": "message",
            "_time_field": "@timestamp",
            "_stream_fields": "host.name,process.name",
        }

loghub.senders.File

It can batch. If batching, the file on disk might be in an inconsistent state, with events missing and holes in the file, until a batch is totally committed to disk.

• fileName, an expression for the filename,
• cacheSize, if the filename is an expression, a cache should be used to avoid constant reopening of the same files, defines the open cache size; default to 10 entries.
• truncate, truncate before opening a file if it already exists.
• separator, the entry separator, as a String.

loghub.senders.Nsca

It does not need an encoder.

loghub.senders.Null

loghub.senders.Stdout

loghub.senders.Tcp

It can batch. When batching, each batch will use a dedicated socket, to the same destination.

• destination,
• port,
• separator, the entry separator, as a String

loghub.senders.Udp

loghub.senders.ZMQ

It can batch.

Attributes

• type, a ZMQ socket type.
• hwm
• destination, a ZMQ URL.
• method, can be BIND or CONNECT.
• serverKey
• security, can be NULL, PLAIN, CURVE.
• zapHandler, can be ALLOW, DENY, STRICT, METADATA.