Alias [test-schema-active-logs] has more than one indices associated with it [[......]], can't execute a single index op

[ankitachow]

Elasticsearch 5.4.1
Rollover API problem

We are using Rollover to create new index upon a document count condition is reached.

But while ingestion is happening, if we run the rollover API, getting below error:
"Alias [test-schema-active-logs] has more than one indices associated with it [[test-schema-000004, test-schema-000005]], can't execute a single index op"

From Rollover API understanding write alias should automatically switch to new index (test-schema-000005) created and move the alias from the old index (test-schema-000004). How can this error be handled?

[dnhatn]

@ankitachow, Please ask questions on https://discuss.elastic.co where we can give you a better support. We use Github for bug reports and feature requests. Thank you.

[dadoonet]

TBH it sounds like a bug to me. But would be great if @ankitachow shares a full script to reproduce all the steps done.

@ankitachow could you do that?

[ankitachow]

@dadoonet Sure see below steps.

We are ingesting data through ES-Hadoop connector continuously. Below are the steps conducted in production:

1. Data ingested with a template having below information
   a. write & search alias
   b. no. of shards = no. of nodes
   c. best_compression
2. Rollover based on certain doc count
3. Shrink the index. The template of the compressed index has
   a. No. of shards = 1
   b. best_compression
4. Remove the search-logs alias from the old index and add it to the compressed index
5. Forcemerge
6. Delete the old index

During Rollover, sometimes(80%) we are getting above error in Spark job and its stopping ingestion. New rolled over index getting created properly. Once we start ingesting again, data gets written to new index created from rollover.

Below is our rollover API command.

RESPONSE=$(curl -s -XPOST ''$ip':9200/'$active_writealias'/_rollover?pretty=true' -d'
{
"conditions": {
"max_docs": "'"$rollovercond"'"
}
}')

But if we run the script after ingestion, there's no error.

[dadoonet]

The error you are getting seems to indicate that 2 indices are defined behind the write alias.
This should not happen.

Do you call _rollover API only from one single machine? Or is it executed from different nodes?
Can you share the elasticsearch logs when the problem appears? I mean some lines before the problem and some lines after if any.

[ankitachow]

The template for the test-schema is as follows:
{
"template": "test-schema-*",
"settings": {
"number_of_shards": 13,
"number_of_replicas": 0,
"refresh_interval" : "30s",
"codec":"best_compression"
},
"aliases": {
"test-schema-active-logs": {},
"test-schema-search-logs": {}
},
"mappings":{
"test-log":{
"_all":{"enabled": false},
"properties":{ .....

So, rollover is creating the new index and also creating the write alias point to the new index which shouldn't happen.
The rollover API is called from only 1 machine. There's no problem with the Elasticsearch front. So, elasticsearch rollover runs fine. The ES hadoop spark job faies giving below error.

"Alias [test-schema-active-logs] has more than one indices associated with it [[test-schema-000004, test-schema-000005]], can't execute a single index op"

[nonpool]

I also met this bug,especially in multi-thread writing is very easy to happen.
_aliases API wrote 'Renaming an alias is a simple remove then add operation within the same API. This operation is atomic, no need to worry about a short period of time where the alias does not point to an index' in document,so this bug is because you did not use this api or _aliases API has this bug?

[dnhatn]

I can reproduce this with the below test snippet.

public void testIndexingAndRolloverConcurrently() throws Exception {
    client().admin().indices().preparePutTemplate("logs")
        .setPatterns(Collections.singletonList("logs-*"))
        .addAlias(new Alias("logs-write"))
        .get();
    assertAcked(client().admin().indices().prepareCreate("logs-000001").get());
    ensureYellow("logs-write");

    final AtomicBoolean done = new AtomicBoolean();
    final Thread rolloverThread = new Thread(() -> {
        while (done.get() == false) {
            client().admin().indices()
                .prepareRolloverIndex("logs-write")
                .addMaxIndexSizeCondition(new ByteSizeValue(1))
                .get();
        }
    });
    rolloverThread.start();
    try {
        int numDocs = 10_000;
        for (int i = 0; i < numDocs; i++) {
            logger.info("--> add doc [{}]", i);
            IndexResponse resp = index("logs-write", "doc", Integer.toString(i), "{}");
            assertThat(resp.status(), equalTo(RestStatus.CREATED));
        }
    } finally {
        done.set(true);
        rolloverThread.join();
    }
}

We create an index with alias (via template) and update index alias in two separate cluster tasks. This can be a root cause of this issue.

https://github.com/dnhatn/elasticsearch/blob/c7ce5a07f26f09ec4e5e92d07aa08f338fbb41b8/core/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java#L133-L135