How about add Ingest CRD for Elasticsearch.

[leptonyu]

Then we can define pipelines in k8s config files.

[barkbay]

Hi, can you share more about your use case or any specific example you have in mind ?

Note that there are some ongoing work about supporting beats.

[leptonyu]

I want to deploy elk on k8s, but use ingest feature of elasticsearch in stead of logstash.

The problem is I can use k8s yaml to configure elasticsearch and filebeat. But I must use es restful api to create Ingest pipeline manually.

If eck operator can define Ingest CRD, which can auto create ingests in elasticsearch, then I can deploy the whole stack automatically. Put ingest config in FileBeat CRD or Elasticsearch CRD are good too.

I use two ingest pipeline in the following config.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elastic-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover.providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        templates:
          - condition:
              equals: 
                kubernetes.container.name: nginx-ingress-controller
            config:
              - type: container
                paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log
                pipeline: nginx
          - condition:
              equals: 
                kubernetes.labels.lang: java
            config:
              - type: container
                paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log
                pipeline: my_java_app

[sebgl]

Relates other issues around declarative configuration management in Elasticsearch and Kibana: #2157, #2155, #3159.

[david-kow]

Currently we have no plans to introduce such CRD - it seems to me that it'd target too narrow category of ES APIs. We are thinking about your and similar use cases though - for now, best bet is to follow issues that were linked.

[mtparet]

To follow up, "CRDs" like for terraform have been released https://www.elastic.co/blog/streamline-configuration-processes-with-official-elastic-stack-terraform-provider