[Filebeat] entity-analytics - no errors or failure metrics on bad client credentials

[andrewkroh]

The only way I could diagnose that the entity-analytics input was not operating normally was by enabling debug logging. I was expecting to be able to observe that errors were occurring by looking at metrics. Then I would go check the logs and find some warnings/errors.

The client credentials had expired and there was no indication of a problem.

For confirmed bugs, please report:

• Version: 8.7.0

Metrics

$ curl localhost:6060/inputs/?pretty
[
  {
    "id": "azure-ad-azure2elasticsearch",
    "input": "entity-analytics-azure-ad",
    "sync_error": 0,
    "sync_processing_time": {
      "histogram": {
        "count": 1,
        "max": 414570372,
        "mean": 414570372,
        "median": 414570372,
        "min": 414570372,
        "p75": 414570372,
        "p95": 414570372,
        "p99": 414570372,
        "p999": 414570372,
        "stddev": 0
      }
    },
    "sync_total": 1,
    "update_error": 0,
    "update_processing_time": {
      "histogram": {
        "count": 4,
        "max": 231179226,
        "mean": 214423684.25,
        "median": 215676942,
        "min": 195161627,
        "p75": 229935254.25,
        "p95": 231179226,
        "p99": 231179226,
        "p999": 231179226,
        "stddev": 14803094.325320626
      }
    },
    "update_total": 4
  }
]

Logs

{
  "log.level": "debug",
  "@timestamp": "2023-04-20T16:25:20.304Z",
  "log.logger": "input.entity-analytics-azure-ad",
  "log.origin": {
    "file.name": "azuread/azure.go",
    "file.line": 184
  },
  "message": "Running incremental update...",
  "service.name": "filebeat",
  "id": "azure-ad-azure2elasticsearch",
  "tenant_id": "aaaaaaaaa-417d-4664-bbbb-8f7640719adb",
  "provider": "azure-ad",
  "ecs.version": "1.6.0"
}
{
  "log.level": "debug",
  "@timestamp": "2023-04-20T16:25:20.304Z",
  "log.logger": "input.entity-analytics-azure-ad",
  "log.origin": {
    "file.name": "oauth2/oauth2.go",
    "file.line": 114
  },
  "message": "Existing token has expired or not set, renewing token",
  "service.name": "filebeat",
  "id": "azure-ad-azure2elasticsearch",
  "tenant_id": "aaaaaaaaa-417d-4664-bbbb-8f7640719adb",
  "provider": "azure-ad",
  "ecs.version": "1.6.0"
}
{
  "log.level": "debug",
  "@timestamp": "2023-04-20T16:25:20.535Z",
  "log.logger": "input.entity-analytics-azure-ad",
  "log.origin": {
    "file.name": "azuread/azure.go",
    "file.line": 123
  },
  "message": "Next update expected at: 2023-04-20 16:26:20.535453736 +0000 UTC m=+211.795356695",
  "service.name": "filebeat",
  "id": "azure-ad-azure2elasticsearch",
  "tenant_id": "aaaaaaaaa-417d-4664-bbbb-8f7640719adb",
  "provider": "azure-ad",
  "ecs.version": "1.6.0"
}

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)