In the Alias of data streams filter on only one data_stream ends up applying to all others

[mpakoupete]

Elasticsearch Version

8.5

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

The use of alias of data streams behaves differently from the alias of indices.
When trying to apply filter to only one data_stream in the Alias, the result is that It ends up applying to both.

Steps to Reproduce

POST _aliases
{
    "actions": [
        {
            "add": {
                "index": "data_stream_1",
                "alias": "alias_1",
                "filter": {
                    "bool": {
                        "filter": [
                            {
                                "term": {
                                    "reaction": "report"
                                }
                            }
                        ]
                    }
                }
            }
        },
        {
            "add": {
                "index": "data_stream_2",
                "alias": "alias_1"
            }
        }
    ]
}

Per the above request, I’m trying to apply filter to only one data stream in the Alias but the result is that It ends up applying to both.

result

GET _alias/alias_1
{
  "data_stream_2": {
    "aliases": {
      "alias_1": {
        "filter": {
          "bool": {
            "filter": [
              {
                "term": {
                  "reaction": "report"
                }
              }
            ]
          }
        }
      }
    }
  },
  "data_stream_1": {
    "aliases": {
      "alias_1": {
        "filter": {
          "bool": {
            "filter": [
              {
                "term": {
                  "reaction": "report"
                }
              }
            ]
          }
        }
      }
    }
  }
}

Logs (if relevant)

No response

[dakrone]

I looked into this a little bit. Here's a quick reproduction that includes creating the data streams from ./gradlew run:

POST /logs-foo-bar/_doc
{
  "@timestamp": "2022-01-01"
}

POST /logs-foo-baz/_doc
{
  "@timestamp": "2022-01-01"
}

POST _aliases
{
    "actions": [
        {
            "add": {
                "index": "logs-foo-bar",
                "alias": "alias_2",
                "filter": {
                    "bool": {
                        "filter": [
                            {
                                "term": {
                                    "reaction": "report"
                                }
                            }
                        ]
                    }
                }
            }
        },
        {
            "add": {
                "index": "logs-foo-baz",
                "alias": "alias_2"
            }
        }
    ]
}

GET /_alias/alias_2

The output includes the filter duplicated for both data streams, instead of only for the logs-foo-bar data stream.

---

Looking into where this is happening in the code, this is due to a couple of blocks of code:

This first is:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamAlias.java

Lines 160 to 172 in c43571b

	boolean filterUpdated;
	CompressedXContent filter;
	if (filterAsMap != null) {
	filter = compress(filterAsMap);
	if (this.filter == null) {
	filterUpdated = true;
	} else {
	filterUpdated = filterAsMap.equals(decompress(this.filter)) == false;
	}
	} else {
	filter = this.filter;
	filterUpdated = false;
	}

Where the issue is the filterUpdated ends up not being updated if the newly passed in filter is null but the previous filter was non-null.

The second part is this function:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/DataStreamMetadata.java

Lines 102 to 126 in c43571b

	public DataStreamMetadata withAlias(String aliasName, String dataStream, Boolean isWriteDataStream, String filter) {
	if (dataStreams.containsKey(dataStream) == false) {
	throw new IllegalArgumentException("alias [" + aliasName + "] refers to a non existing data stream [" + dataStream + "]");
	}
	Map<String, Object> filterAsMap;
	if (filter != null) {
	filterAsMap = XContentHelper.convertToMap(XContentFactory.xContent(filter), filter, true);
	} else {
	filterAsMap = null;
	}
	DataStreamAlias alias = dataStreamAliases.get(aliasName);
	if (alias == null) {
	String writeDataStream = isWriteDataStream != null && isWriteDataStream ? dataStream : null;
	alias = new DataStreamAlias(aliasName, List.of(dataStream), writeDataStream, filterAsMap);
	} else {
	DataStreamAlias copy = alias.update(dataStream, isWriteDataStream, filterAsMap);
	if (copy == alias) {
	return this;
	}
	alias = copy;
	}
	return new DataStreamMetadata(dataStreams, ImmutableOpenMap.builder(dataStreamAliases).fPut(aliasName, alias).build());
	}

Where the data stream aliases are being stored in an ImmutableOpenMap<String, DataStreamAlias>, where the key is the alias name. Unfortunately this means that one alias (alias_2 in the example above) can only refer to a single DataStreamAlias instance, and they end up clobbering each other. We'll need to figure out a different way to store this so that we can have multiple DataStreamAlias instances (with separate filters) for a single alias name.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)