[BUG] NullPointerException in multi_terms query

[z002Holpp]

Hi all,
we want to use OpenSearch for querying our systems to get aggregated system information like CPU and RAM usage.
Therefore we use date historgram aggregation. Since we want to have those measures for different hosts / containers / processes we need also a kind of grouping mechanism which we tried to realize with the multi_terms feature.

Our initial query was looking like this:

{
  "aggs": {
    "xLCAggPerTime": {
      "aggs": {
        "MyGroupingAggregate": {
          "aggs": {
            "CPU": {
              "avg": {
                "field": "system.cpu.total.norm.pct"
              }
            },
            "Memory": {
              "avg": {
                "field": "system.memory.actual.used.bytes"
              }
            }
          }, 
          "multi_terms": {
            "terms": [
                {"field": "host.name"},
                {"field": "agent.type"}
              ]
          }
        },
        "xLCAggBucketSort": {
          "bucket_sort": {
            "from": 0,
            "size": 1000
          }
        }
      },
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1s"
      }
    }
  },
  "from": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2023-02-09T12:15:47Z",
        "lt": "2023-02-09T12:16:47Z"
      }
    }
  },
  "size": 0,
  "sort": [
    {
      "@timestamp": {
        "unmapped_type": "date",
        "order": "asc"
      }
    }
  ]
}

However this query results in the following execption:
java.lang.NullPointerException: Cannot invoke \"org.opensearch.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null

If needed I can also provide the full error response, but for me it seems not to contain much more meaningful information.

The query runs fine if the multi_terms part is removed, leaving only

            "terms": 
                {"field": "agent.type.keyword"}
              

I also tried to minimize the query in order to isolate the issue and it came out that the following query will still produce the error:

{
  "aggs": {
    "xLCAggPerTime": {
      "aggs": {
        "MyGroupingAggregate": {
          "multi_terms": {
            "terms": [
                {"field": "host.name"},
                {"field": "agent.type"}
              ]
          }
        }
      },
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "60s"
      }
    }
  }
}

Important Note: The queries were tested with ElasticSearch 8.7.0 and they worked as expected. So the queries themsselves can be considered to be correct.

Thank you already in advance for your help! :slight_smile:

To Reproduce
Have an OpenSearch Instance with MetricBeat and run the above query.

Expected behavior
The query runs and returns zero or more results, depending on the data in OpenSearch. However, there shall not be a Nullpointer Exception.

Plugins
MetricBeat

Host/Environment (please complete the following information):
OpenSearch Version used: 2.2.0 & 2.6.0
MetricBeat: 2.7.12.1

[dblock]

Is this a dup of #5785?

It would be amazing if you could help us turn this into a failing unit test. Let's fix it.

[z002Holpp]

@dblock : Thanks for picking up this issue! :-) I would be delighted to help you in fixing this! Checking issue #5785, I would agree that it looks very much like a duplicate.
So, how can I support you here?

[dblock]

@z002Holpp Check out YAML tests and try to contribute one that reproduces this problem? The reduce the query to a bare minimum that causes the issue to happen. Make a pull request with that. https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-the-rest-layer is a good place to start.

[dblock]

I am going to close this one as a dup of #5785. It does have something closer to a repro, so I'm glad you reported it, thank you!

[z002Holpp]

I unfortunately have little time to familiarize myself with the technology stack and the domain knowledge. I was hoping that I could support here more from a consumer perspective.

[austintlee]

@z002Holpp MetricBeat 7.12.1 uses docType to bulk ingest which is not supported on OpenSearch 2.2 or 2.6. I am guessing you have not discovered this because you are using ElasticSearch 8.7.0? If you are using OpenSearch, how are you getting around #3484 ?

[z002Holpp]

@austintlee Hi, thanks for mentioning that! We do not use ElasticSearch, we use only OpenSearch. At least from my side, I did not see the bug / exception mentioned in #3484.
Just now reproduced the issue with multi_terms on a colleagues machine with OpenSearch 2.6.0 and MetricBeat 7.12.1. (Docker-Compose from Repo with 2 Node Setup + MetricBeat as standalone exe on local Windows machine) and did not see any Exceptions mentioned in #3484.

[austintlee]

@penghuo when you get a chance, can you review these changes?

[austintlee]

I had a lot of trouble writing the unit test for this because constructing a MultiTermsAggregator object was quite tricky if you have to mock some of the members of the object. But I didn't think I could do much better than what you see in that PR. Please let me know what you think.