Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[exporter/elasticsearch] Supports the logstash index format #29624

Closed
JaredTan95 opened this issue Dec 2, 2023 · 0 comments · Fixed by #29625
Closed

[exporter/elasticsearch] Supports the logstash index format #29624

JaredTan95 opened this issue Dec 2, 2023 · 0 comments · Fixed by #29625
Assignees
@JaredTan95
Labels
enhancement New feature or request exporter/elasticsearch

Comments

@JaredTan95
Copy link
Member

JaredTan95 commented Dec 2, 2023 •
edited

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

When users using otel col to collect logs like vector or fluentbit, they may have index management issues.

The common practice is creating and apply Index lifecycle management (ILM) policies to automatically manage those indices which accord with logstash's index format. and this helps users seamlessly switch from fluentbit or vector to otel col.

Describe the solution you'd like

add some configuration parameters:

  • logstash_format: true/false to enable Logstash format compatibility. default is false
  • logstash_prefix: When Logstash_Format is enabled, the Index name is composed using a prefix and the date, e.g: If logstash_prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated. default is logstash
  • logstash_prefix_separator: Set a separator between logstash_prefix and date. default is -
  • logstash_date_format: Time format (based on strftime) to generate the second part of the Index name. default is %Y.%m.%d

Describe alternatives you've considered

No response

Additional context

No response
@JaredTan95 JaredTan95 added enhancement New feature or request needs triage New item requiring triage labels Dec 2, 2023
@JaredTan95 JaredTan95 self-assigned this Dec 2, 2023
@JaredTan95 JaredTan95 added exporter/elasticsearch and removed needs triage New item requiring triage labels Dec 2, 2023
@JaredTan95 JaredTan95 mentioned this issue Dec 2, 2023
Merged
@github-actions github-actions bot mentioned this issue Dec 5, 2023
Closed
dmitryax pushed a commit that referenced this issue Dec 8, 2023
@JaredTan95
[exporter/elasticsearch] Logstash format compatibility (#29625)
a9d4196 
**Description:** Logstash format compatibility. Traces or Logs data can
be written into an index in logstash format.
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** <Issue number if applicable>
close
#29624

**Documentation:** added some descriptions for `logstash_format `
configurations.
1. otel-col.yaml
```yaml
receivers:
  otlp:
    protocols:
      grpc:
  filelog:
    include: [ ./examples/kubernetes/varlogpods/containerd_logs-0_000011112222333344445555666677778888/logs/0.log ]
    start_at: beginning
    operators:
      # Find out which format is used by kubernetes
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'
      # Parse CRI-O format
      - type: regex_parser
        id: parser-crio
        regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05.999999999Z07:00'
      # Parse CRI-Containerd format
      - type: regex_parser
        id: parser-containerd
        regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Parse Docker format
      - type: json_parser
        id: parser-docker
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Extract metadata from file path
      - type: regex_parser
        id: extract_metadata_from_filepath
        regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
        parse_from: attributes["log.file.path"]
        cache:
          size: 128  # default maximum amount of Pods per Node is 110
      # Update body field after finishing all parsing
      - type: move
        from: attributes.log
        to: body
      # Rename attributes
      - type: move
        from: attributes.stream
        to: attributes["log.iostream"]
      - type: move
        from: attributes.container_name
        to: resource["k8s.container.name"]
      - type: move
        from: attributes.namespace
        to: resource["k8s.namespace.name"]
      - type: move
        from: attributes.pod_name
        to: resource["k8s.pod.name"]
      - type: move
        from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
      - type: move
        from: attributes.uid
        to: resource["k8s.pod.uid"]
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  elasticsearch/log:
    tls:
      insecure: false
    endpoints: [http://localhost:9200]
    logs_index: otlp-logs
    logstash_format:
      enabled: true
    timeout: 2m
    flush:
      bytes: 10485760
    retry:
      max_requests: 5
    sending_queue:
      enabled: true
  elasticsearch/traces:
    tls:
      insecure: false
    endpoints: [http://localhost:9200]
    traces_index: otlp-traces
    logstash_format:
      enabled: true
    timeout: 2m
    flush:
      bytes: 10485760
    retry:
      max_requests: 5
    sending_queue:
      enabled: true

  debug:

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    logs:
      receivers: [otlp,filelog]
      processors: [batch]
      exporters: [debug, elasticsearch/log]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticsearch/traces]

```
3. es index created when `otel-col` write traces and logs:
<img width="913" alt="image"
src="https://github.com/open-telemetry/opentelemetry-collector-contrib/assets/12468337/0ede0fd7-ed85-4fd4-b843-093c13edc1e3">

4. query index data:
<img width="743" alt="image"
src="https://github.com/open-telemetry/opentelemetry-collector-contrib/assets/12468337/1e89a44c-cead-4aab-8b3a-284a8b573d3b">
<img width="817" alt="image"
src="https://github.com/open-telemetry/opentelemetry-collector-contrib/assets/12468337/429c25bc-336e-4850-9d83-ed7423f38e90">

---------

Signed-off-by: Jared Tan <jian.tan@daocloud.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JaredTan95 JaredTan95

Labels
enhancement New feature or request exporter/elasticsearch
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[exporter/elasticsearch] Logstash format compatibility JaredTan95/opentelemetry-collector-contrib
1 participant
@JaredTan95