Sinks Formats and Rollups

Sinks, formats, and rollups

Flags and YAML that are in the binary but were missing from the README / Advanced wiki. For OTLP specifically see Sending data with OTLP.

Formats (-format)

Live options (from cmd/ktranslate/main.go):

json | flat_json | avro | netflow | influx | carbon | prometheus | new_relic | new_relic_metric | splunk | elasticsearch | kflow | ddog | otel | snmp | parquet

-format_rollup and -format_metric take the same family (rollup/metric lists omit a few input-only formats).

Format	Notes	PR
otel	OTLP metrics + logs. Sending data with OTLP.	#653
parquet	Columnar files for S3/file sinks. Files cannot be appended — ktranslate flushes a complete object.	#728, #771
snmp	Re-emit inputs as SNMP traps (samplicator-style trap forwarder).	#704
elasticsearch	Optional -elastic.action (e.g. create).	#796

-format=snmp (trap forwarder)

ktranslate -snmp trap-v2.yaml \
  -format snmp \
  -snmp.format.conf trap-v2.yaml

trap:
  listen: 127.0.0.1:1620
  community: public
  version: v2c
  trap_only: true
  endpoint: 127.0.0.1
  endpoint_port: 1620
  drop_undefined: false

endpoint / endpoint_port are the upstream trap destination. version may be v1, v2c, or v3. Kentik uses enterprise OID .1.3.6.1.4.1.41263.6169 as the generic trap.

Trap event regex (#703)

In a profile trap event, conversion: "regexp:…" plus tag extracts named captures into attributes:

- trap_oid: 1.3.6.1.4.1.41263.999
  trap_name: ntxAlert
  events:
    - name: ntxAlertTitle
      OID: 1.3.6.1.4.1.41263.999.3
      conversion: "regexp:\\:\\s(?P<hostid>[0-9]+-[0-9]+-[0-9]+)\\s"
      tag: "Problem with"

Uses Go named captures.

Sinks (-sinks)

Live options: kafka | stdout | new_relic | kentik | net | http | splunk | prometheus | file | s3 | gcloud | gcppubsub | ddog | otel | null

Comma-separate to fan out.

Kafka (TLS / SASL)

README only documents -bootstrap.servers and -kafka_topic. Additional flags (#853, #865, #866):

Flag	Purpose
-kafka_security_protocol	Security protocol (PLAINTEXT / SSL / SASL_…).
-kafka_sasl_mechanism	SASL mechanism.
-kafka_sasl_username / -kafka_sasl_password	SASL user.
-kafka_ssl_ca_file / -kafka_ssl_cert_file / -kafka_ssl_key_file	TLS files.
-kafka_ssl_insecure	Skip TLS verify (also -kafka.tls.skip.verify on older builds).
-kafka_kerberos_*	Kerberos principal, keytab, realm, krb5.conf, disable PAFXFAST.
-kafka_required_acks	Producer acks.
-kafka_compression	Producer compression.
-kafka_max_message_bytes	Max message size.
-kafka_retry_max	Produce retries.
-kafka_flush_frequency / -kafka_flush_messages / -kafka_flush_bytes	Flush batching.

Older -kafka.tls.config values (basic, cert.pem,key.pem, cert.pem,key.pem,ca-cert.pem) were the first TLS API; prefer the kafka_ssl_* flags on current builds.

S3-compatible endpoints (#834)

Flag	Purpose
-s3_bucket	Bucket name.
-s3_prefix	Object prefix (default /kentik).
-s3_region	AWS region (default us-east-1).
-s3_endpoint	Override API URL for MinIO / on-prem S3.
-s3_signing_region	Signing region when the endpoint is not AWS.
-s3_flush_sec	Rotate objects this often (default 60).
-s3_assume_role_arn	Assume-role ARN.
-ec2_instance_profile	Use the instance profile.

HTTP and log URLs (#767)

Flag	Purpose
-http_url	Metrics / events POST URL.
-http_log_url	Separate URL for logs (syslog, traps, --tee_logs).
-http_insecure	Skip TLS verify.
-http_header	Repeatable Name: value header (e.g. Splunk token).
-http_timeout_sec	HTTP timeout.

Splunk Cloud example:

ktranslate -syslog.source 0.0.0.0:5143 \
  -sinks http \
  -http_insecure \
  -http_header "Authorization: Splunk xxxx" \
  -http_url https://xxx.splunkcloud.com:8088/services/collector/event \
  -http_log_url https://xxx.splunkcloud.com:8088/services/collector/event

Net sink (multi-destination) (#672)

-net_server accepts a comma-separated list:

ktranslate -nf.source=auto -format netflow -sinks net \
  -net_server 127.0.0.1:9090,127.0.0.1:9091 \
  -max_flows_per_message=1

Use -max_flows_per_message=1 for UDP so packets stay under MTU.

Redis format (#888)

Flag	Purpose
-redis.addr	Redis address.
-redis.password	Password.
-redis.db	DB index.
-redis.ttl.sec	Key TTL.
-redis.key_prefix	Key prefix.

Rollups and filters

Wiki Advanced documents only basic -filters on -nf.source. These flags are also supported:

Flag / syntax	Purpose	PR
-rollup_keep_undefined	Keep rollup dimensions that would have been dropped; emit the string undefined.	#749
-rollup_top_k	Keep only the top K series (default 10).	(older)
-rollup_key_join	Join token for dimension keys (default ^).	(older)
Named filter on a rollup	-filters "string,dim,==,val,my_rollup" then -rollups s_sum,my_rollup,in_bytes,…	#755
dimA$$---$$dimB	Merge two dimensions into one (aaa---bbb).	#769
sum,name_in;name_out,in_bytes;out_bytes,dim1,dim2	One rollup definition, two metrics (; separator).	#846
string,custom_str.foo,=~,	Key exists (any value). !~ = key absent.	#758

Flow extras

Flag	Purpose	PR
-nf.source	netflow5 | netflow9 | ipfix | sflow | nbar | asa | pan | auto	README still omits nbar, asa, pan, auto
-stitch.enable	Pair request/response flows; emit pair_* fields.	#902
-stitch.buffer.len	Ring-buffer size for pairing (default in code: set explicitly, e.g. 10000).	#902
-tee_flow	Tee decoded flow to another ktranslate instance.	#741
-km_value_map	Enrich flow from a Kentik NMS kmetrics map file.	#924

Unknown exporters still get a service_name (synthetic “missing device”) so OTLP resource labels are never empty (#784).

Debug / build

Flag or build-arg	Purpose	PR
-snmp_walk_file	Answer SNMP polls from a saved walk file (no live device).	#712
-generate-config	Print a default YAML config and exit.	(config path)
-config	Load flags from a YAML config file (merged over defaults).	(config path)
Docker --build-arg CONFIG-DIR=…	Bake a custom config directory into the image.	#640