Skip to content

Latest commit

 

History

History
246 lines (187 loc) · 7.64 KB

custom-tenant-support.adoc

File metadata and controls

246 lines (187 loc) · 7.64 KB
Raw

Configuring Custom Tenants

Most of the outputs supported have some concept of a tenant and can currently be configured with custom values. For example Kafka has topics, CloudWatch has log_groups, and ElasticSearch has indices. This document provides guidance on configuring the tenant for supported outputs.

CloudWatch

CloudWatch’s tenant can be configured by setting how logs are grouped together along with an optional prefix. Configure using the groupBy & groupPrefix fields.

  • Logs can be grouped by the following:

    1. namespaceName

    2. namespaceUUID

    3. logType

Example

cluster-log-forwarder.yaml
spec:
  outputs:
   - name: cw
     type: cloudwatch
     cloudwatch:
       groupBy: logType (1)
       groupPrefix: <group prefix> (2)
       region: us-east-2

  1. Specify the groupBy type here.

  2. Specify an optional prefix to add to the tenant.

References

  1. See the forwarding to CloudWatch section on the official documentation.

Elasticsearch

Elasticsearch’s default behavior is to send logs to indices defined by their log_type and appended by -write.

Example: `application` logs will be sent to the index `app-write`.

ElasticSearch’s index can be configured if parse: json is enabled.

  • Configure using the structuredTypeKey & structuredTypeName fields.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
    - name: external-es
      type: elasticsearch
      elasticsearch:
        structuredTypeKey: kubernetes.namespace_name (1)
        structuredTypeName: myParsedMessages (2)
  pipelines:
    - name: parsed-app-logs
      inputRefs:
        - application
      outputRefs:
        - external-es
      parse: json (3)

  1. Specify the structuredTypeKey here.

  2. Specify the structuredTypeName here.

  3. Parse JSON must be enabled.

Note
 Both or one of structuredTypeKey and/or structuredTypeName must be specified. If both are specified, the structureTypeName will be used as fallback if the field defined by the structuredTypeKey is not present in the log record.

References

  1. See the custom index configuration section of the internal Elasticsearch forwarding documentation.

  2. See also the configuring JSON log data for Elasticsearch on the offical documention.

Google Cloud Logging

Google Cloud Logging can be configured with a custom tenant by setting the logId field.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
    - name: gcp-1
      type: googleCloudLogging
      secret:
        name: gcp-secret
      googleCloudLogging:
        projectId : openshift-gce-devel
        logId : app-gcp (1)

  1. Specify the logId here.

References

  1. See the internal forwarding doc for Google Cloud Logging.

  2. See also the forwarding logs to GCL section on the official documentation.

HTTP

A header can be added to the HTTP output as a tenant label.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
    - name: httpout-app
      type: http
      url: <HTTP-URL>
      http:
        headers:
          h1: v1
          tenant: myAppLogs (1)
        method: POST

  1. Specify the header here.

References

  1. See the internal forwarding doc for Vector HTTP.

  2. See also the forwarding logs to HTTP section on the offical documentation.

Kafka

Kafka’s tenant can be configured by setting the topic either through the topic field or at the end of the URL.

Examples

Topic Field: cluster-log-forwarder.yaml
spec:
  outputs:
     - name: app-logs
       type: kafka
       kafka:
         topic: app-topic (1)

  1. Specify the topic here.

Topic in URL: cluster-log-forwarder.yaml
spec:
  outputs:
     - name: app-logs
       type: kafka
       url: tls://kafka.example.devlab.com:9093/app-topic (1)

  1. Specify the topic here.

References

  1. See the forwarding to Kafka section on the official documentation.

Loki

Loki’s tenant can be configured through the tenantKey field.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
     - name: loki-insecure
       type: "loki"
       url: http://loki.insecure.com:3100
       loki:
         tenantKey: kubernetes.namespace_name (1)

  1. Specify the tenantKey here.

References

  1. See the forwarding logs to Loki section on the official documentation.

Splunk

Splunk’s index can be configured by either setting an IndexKey or IndexName.

  1. IndexKey: Dynamic index extraction of logs.

    • If the field referenced by the IndexKey is not present, the log will be sent to Splunks default index

  2. IndexName: Static index values.

Note
 If IndexKey/IndexName is not defined, logs will be sent to Splunk’s default index.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
    - name: splunk-receiver
      type: splunk
      splunk:
        indexKey: "kubernetes.namespace_name" (1)
      url: 'http://example-splunk-hec-service:8088'

  1. Specify one of indexKey or indexName not both.

References

  1. See the customizing Splunk’s index section of the internal Splunk forwarding documentation.

  2. See also the Splunk forwarding section on the official documentation.

Syslog

Syslog uses a combination of facility & severity to group logs. A possible way to define a tenant is to configure the tag.

Example

cluster-log-forwarder.yaml
spec:
  outputs:
    - name: syslogout
      syslog:
        addLogSource: true
        tag: mytag (1)
      type: syslog
      url: tls://syslog-receiver.openshift-logging.svc:24224

  1. Specify the tag here.

References

  1. See adding log source information to message output for Syslog on the official documentation.

Outputs Without Customizable Tenants

Lokistack (non default)

Tenant customization not currently supported.

Default (Elasticsearch/LokiStack)

Tenant customization is not allowed.

Fluent Forward

FluentD only/ not applicable