Enable beats stack monitoring configuration

[naemono]

Closes #5563

This change enables easy stack monitoring configuration for Beats, such as we already have for both Elasticsearch and Kibana, by adding the following configuration stanza in the Beats CRD

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
spec:
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: elasticsearch
    logs:
      elasticsearchRefs:
      - name: elasticsearch

[pebrc]

Had a quick first look, I haven't done any testing yet.

[pebrc]

Almost LGTM. I am having second thoughts though whether we should stick to the common

monitoring:
    metrics:
      elasticsearchRefs:
      - name: metrics-es
    logs:
      elasticsearchRefs:
      - name: logs-es

schema just to keep the door open for future enhancements that would for example export logs as well and to keep the CRDs somewhat uniform. Can't quite make my mind up. Maybe @thbkrkr has an opinion as well given that he wrote the original implementation.

[pebrc]

I have thought about it some more and asked for opinions from others and I think now we should indeed stick with the same API schema we have for Kibana and Elasticsearch. The reasons are as originally stated: to be consistent across CRDs and to be able to add log extraction through Filebeat. It would be ok to do the log bit in a separate PR though if you prefer. It might also be good to reach out to the Observability team to get their take on the preferred way to do Beats monitoring and log extraction given that this leads to a kind of Russian doll problem where we add more Beats eg. Filebeat to monitor another Beat, which then raises the question where does one stop (but we already have that problem with the monitoring of Kibana and ES so maybe that is ok)

[naemono]

@pebrc I've reached out to observability, and they also suggested we use sidecars to monitor these instead of internal collectors. My changes to allow sidecars is nearly complete, with some unit tests for the functionality being taken care of, and I'll note that this is ready for another look.

[naemono]

run/e2e-tests tags=beat

[thbkrkr]

This looks good! I'm testing with this manifest and I get a socket permission error for Filebeat:

{
    "log.level": "error",
    "@timestamp": "2022-09-09T13:40:34.362Z",
    "log.origin": {
        "file.name": "module/wrapper.go",
        "file.line": 256
    },
    "message": "Error fetching data for metricset beat.state: error making http request:
 Get \"http://unix/state\": dial 
 unix /var/shared/filebeat-default-filebeat.sock: connect: permission denied",
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

[naemono]

This looks good! I'm testing with this manifest and I get a socket permission error for Filebeat:

{
    "log.level": "error",
    "@timestamp": "2022-09-09T13:40:34.362Z",
    "log.origin": {
        "file.name": "module/wrapper.go",
        "file.line": 256
    },
    "message": "Error fetching data for metricset beat.state: error making http request:
 Get \"http://unix/state\": dial 
 unix /var/shared/filebeat-default-filebeat.sock: connect: permission denied",
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

I'm investigating these permission issues...

[naemono]

@thbkrkr meant to update yesterday. The permission errors were from attempting to use /usr/share/filebeat-sidecar as the path for the filbeat sidecar, which was unnecessary. I removed that from the config template, tested your manifest locally, and saw no issues. This should be ready for 👀 again.

[naemono]

@thbkrkr There were additional issues found surrounding sidecar permissions reading the unix socket. These were resolved, and tests were added around this feature. It's again ready for 👀

[pebrc]

I stepped into a trap when testing this yesterday by using one of our existing manifests with the -e option and only with @thbkrkr 's help was able to identify that this was the reason why my logs would not show up in the monitoring cluster.

I think we have a usability issue here with some of the existing recipes using the -e option. We can assume that some of our users also might have specified it when overriding the default beats command. I think we should at least document this requirement and maybe even consider, as @thbkrkr suggested, to automatically remove the -e option if log delivery is configured.

[naemono]

I think we have a usability issue here with some of the existing recipes using the -e option. We can assume that some of our users also might have specified it when overriding the default beats command. I think we should at least document this requirement and maybe even consider, as @thbkrkr suggested, to automatically remove the -e option if log delivery is configured.

I didn't think of the scenario where the customer/recipe already had the -e option. Resolved in 52cd829. I'll also get the documentation updated to note this.

[naemono]

@pebrc documentation added. Let me know how you feel about the wording.

[thbkrkr]

LGTM 🚀