[BUG] Unable to create monitor with custom analyzer

[paasi6666]

What is the bug?
When defining a new monitor (under alerting) and selecting the type 'Per document monitor', the monitor saves with following error:

Index settings:

{
  "rpz_0": {
    "settings": {
      "index": {
        "number_of_shards": "4",
        "provided_name": "rpz_0",
        "creation_date": "1649938793819",
        "analysis": {
          "analyzer": {
            "analyzer_keyword": {
              "filter": "lowercase",
              "tokenizer": "keyword"
            }
          }
        },
        "number_of_replicas": "0",
        "uuid": "e8NRlQCHQfau984C3QGMPQ",
        "version": {
          "created": "7100299",
          "upgraded": "136287827"
        }
      }
    }
  }
}

Index mapping:

{
  "rpz_0": {
    "mappings": {
      "dynamic_templates": [
        {
          "internal_fields": {
            "match": "gl2_*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "store_generic": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ],
      "properties": {
        "@metadata_beat": {
          "type": "keyword"
        },
        "@metadata_type": {
          "type": "keyword"
        },
        "@metadata_version": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date"
        },
        "agent_ephemeral_id": {
          "type": "keyword"
        },
        "agent_name": {
          "type": "keyword"
        },
        "beats_type": {
          "type": "keyword"
        },
        "client_id": {
          "type": "keyword"
        },
        "event_action": {
          "type": "keyword"
        },
        "full_message": {
          "type": "text",
          "analyzer": "standard"
        },
        "gl2_accounted_message_size": {
          "type": "long"
        },
        "gl2_message_id": {
          "type": "keyword"
        },
        "gl2_processing_error": {
          "type": "keyword"
        },
        "gl2_processing_timestamp": {
          "type": "date",
          "format": "uuuu-MM-dd HH:mm:ss.SSS"
        },
        "gl2_receive_timestamp": {
          "type": "date",
          "format": "uuuu-MM-dd HH:mm:ss.SSS"
        },
        "gl2_remote_ip": {
          "type": "keyword"
        },
        "gl2_remote_port": {
          "type": "long"
        },
        "gl2_source_input": {
          "type": "keyword"
        },
        "gl2_source_node": {
          "type": "keyword"
        },
        "host_name": {
          "type": "keyword"
        },
        "hostname": {
          "type": "keyword"
        },
        "log_file_path": {
          "type": "keyword"
        },
        "log_offset": {
          "type": "long"
        },
        "loglevel": {
          "type": "keyword"
        },
        "message": {
          "type": "text",
          "analyzer": "standard"
        },
        "query_action": {
          "type": "keyword"
        },
        "query_class": {
          "type": "keyword"
        },
        "query_name": {
          "type": "keyword"
        },
        "query_type": {
          "type": "keyword"
        },
        "rpz_category": {
          "type": "keyword"
        },
        "rpz_message": {
          "type": "keyword"
        },
        "rpz_zone": {
          "type": "keyword"
        },
        "source": {
          "type": "text",
          "analyzer": "analyzer_keyword",
          "fielddata": true
        },
        "source_ip": {
          "type": "keyword"
        },
        "source_port": {
          "type": "keyword"
        },
        "streams": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date",
          "format": "uuuu-MM-dd HH:mm:ss.SSS"
        },
        "url_domain": {
          "type": "keyword"
        },
        "url_short": {
          "type": "keyword"
        }
      }
    }
  }
}

Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: uncaught exception in thread [DefaultDispatcher-worker-5]
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: AlertingException[analyzer [analyzer_keyword] has not been configured in mappings]; nested: Exception[java.lang.IllegalArgumentException: analyzer [analyzer_keyword] has not been configured in mappings];
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:367)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:45)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: Suppressed: AlertingException[analyzer [analyzer_keyword] has not been configured in mappings]; nested: Exception[java.lang.IllegalArgumentException: analyzer [analyzer_keyword] has not been configured in mappings];
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: ... 9 more
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [analyzer_keyword] has not been configured in mappings
Sep  5 13:20:20 graylogsrv-001 systemd-entrypoint: ... 9 more

How can one reproduce the bug?

1. Go to Alerting>Monitors>Create monitor
2. Select 'Per document monitor', select any index and choose a query
3. Go to Preview query and performance and wait..
4. Try to save the monitor

What is the expected behavior?
I don't know, hence it never worked for me.

What is your host/environment?
OS: Centos7
Opensearch Version: 2.9.0
Opensearch-Dashboards Version: 2.9.0

NOTE

1. We are ingesting the logs using graylog.
2. I already opened an issue at the opensearch-project/alerting repository 3 months ago (when v2.7.0 was latest), but so far the issue hasn't been fixed.

[kavilla]

@lezzago would you be able to comment on this?

[paasi6666]

@lezzago any updates on this issue?

[wbeckler]

This needs to be taken up again by the alerting team as it is tied deeply to that part of the application. opensearch-project/alerting#961