Alias [test-schema-active-logs] has more than one indices associated with it [[......]], can't execute a single index op

[ankitachow]

Elasticsearch 5.4.1
Rollover API problem

We are using Rollover to create new index upon a document count condition is reached.

But while ingestion is happening, if we run the rollover API, getting below error:
"Alias [test-schema-active-logs] has more than one indices associated with it [[test-schema-000004, test-schema-000005]], can't execute a single index op"

From Rollover API understanding write alias should automatically switch to new index (test-schema-000005) created and move the alias from the old index (test-schema-000004). How can this error be handled?

[dnhatn]

@ankitachow, Please ask questions on https://discuss.elastic.co where we can give you a better support. We use Github for bug reports and feature requests. Thank you.

[dadoonet]

TBH it sounds like a bug to me. But would be great if @ankitachow shares a full script to reproduce all the steps done.

@ankitachow could you do that?

[ankitachow]

@dadoonet Sure see below steps.

We are ingesting data through ES-Hadoop connector continuously. Below are the steps conducted in production:

1. Data ingested with a template having below information
   a. write & search alias
   b. no. of shards = no. of nodes
   c. best_compression
2. Rollover based on certain doc count
3. Shrink the index. The template of the compressed index has
   a. No. of shards = 1
   b. best_compression
4. Remove the search-logs alias from the old index and add it to the compressed index
5. Forcemerge
6. Delete the old index

During Rollover, sometimes(80%) we are getting above error in Spark job and its stopping ingestion. New rolled over index getting created properly. Once we start ingesting again, data gets written to new index created from rollover.

Below is our rollover API command.

RESPONSE=$(curl -s -XPOST ''$ip':9200/'$active_writealias'/_rollover?pretty=true' -d'
{
"conditions": {
"max_docs": "'"$rollovercond"'"
}
}')

But if we run the script after ingestion, there's no error.

[dadoonet]

The error you are getting seems to indicate that 2 indices are defined behind the write alias.
This should not happen.

Do you call _rollover API only from one single machine? Or is it executed from different nodes?
Can you share the elasticsearch logs when the problem appears? I mean some lines before the problem and some lines after if any.

[ankitachow]

The template for the test-schema is as follows:
{
"template": "test-schema-*",
"settings": {
"number_of_shards": 13,
"number_of_replicas": 0,
"refresh_interval" : "30s",
"codec":"best_compression"
},
"aliases": {
"test-schema-active-logs": {},
"test-schema-search-logs": {}
},
"mappings":{
"test-log":{
"_all":{"enabled": false},
"properties":{ .....

So, rollover is creating the new index and also creating the write alias point to the new index which shouldn't happen.
The rollover API is called from only 1 machine. There's no problem with the Elasticsearch front. So, elasticsearch rollover runs fine. The ES hadoop spark job faies giving below error.

"Alias [test-schema-active-logs] has more than one indices associated with it [[test-schema-000004, test-schema-000005]], can't execute a single index op"

[nonpool]

I also met this bug,especially in multi-thread writing is very easy to happen.
_aliases API wrote 'Renaming an alias is a simple remove then add operation within the same API. This operation is atomic, no need to worry about a short period of time where the alias does not point to an index' in document,so this bug is because you did not use this api or _aliases API has this bug?

[dnhatn]

I can reproduce this with the below test snippet.

public void testIndexingAndRolloverConcurrently() throws Exception {
    client().admin().indices().preparePutTemplate("logs")
        .setPatterns(Collections.singletonList("logs-*"))
        .addAlias(new Alias("logs-write"))
        .get();
    assertAcked(client().admin().indices().prepareCreate("logs-000001").get());
    ensureYellow("logs-write");

    final AtomicBoolean done = new AtomicBoolean();
    final Thread rolloverThread = new Thread(() -> {
        while (done.get() == false) {
            client().admin().indices()
                .prepareRolloverIndex("logs-write")
                .addMaxIndexSizeCondition(new ByteSizeValue(1))
                .get();
        }
    });
    rolloverThread.start();
    try {
        int numDocs = 10_000;
        for (int i = 0; i < numDocs; i++) {
            logger.info("--> add doc [{}]", i);
            IndexResponse resp = index("logs-write", "doc", Integer.toString(i), "{}");
            assertThat(resp.status(), equalTo(RestStatus.CREATED));
        }
    } finally {
        done.set(true);
        rolloverThread.join();
    }
}

We create an index with alias (via template) and update index alias in two separate cluster tasks. This can be a root cause of this issue.

https://github.com/dnhatn/elasticsearch/blob/c7ce5a07f26f09ec4e5e92d07aa08f338fbb41b8/core/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java#L133-L135

[Somnath-Guthula]

Hi guys, I am having a similar issue with a newer version

So, We were trying a rollover indice with our newly setup cluster with Elasticsearch 6.2

When we are trying to rollover the indice, It gives the following error.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Rollover alias [active-fusion-logs] can point to multiple indices, found duplicated alias [[search-fusion-logs, active-fusion-logs]] in index template [fusion-logs]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Rollover alias [active-fusion-logs] can point to multiple indices, found duplicated alias [[search-fusion-logs, active-fusion-logs]] in index template [fusion-logs]"
  },
  "status": 400
}

Please find details below of the template that we are having and steps that I used. This can be fairly used to reproduce the issue.

Template name : fusion-logs

PUT _template/fusion-logs
{
  "template": "fusion-logs-*",
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "routing.allocation.include.box_type": "hot"
  },
  "aliases": {
    "active-fusion-logs": {},
    "search-fusion-logs": {}
  },
  "mappings": {
    "logs": {
      "properties": {
        "host": {
          "type": "keyword"
        },
        "job_id": {
          "type": "keyword"
        },
        "job_result": {
          "type": "keyword"
        }
      }
    }
  }
}

We inserted a 1000 documents in the above active-fusion-logs index and then used the following to roll over the index

POST active-fusion-logs/_rollover
{
  "conditions": {
    "max_docs":   1000
  }
}

The above API gives us an error when we are trying to rollover

Some other info about the cluster.

1. There is no other index other than the above index.
2. active-fusion-logs is aliased to just one write index
3. search-fusion-logs is aliased to multiple indexes

Also, I had tried the same thing with Elasticsearch 5.3.2 and it worked as expected without the error.

[dnhatn]

"reason": "Rollover alias [active-fusion-logs] can point to multiple indices, found duplicated alias [[search-fusion-logs, active-fusion-logs]] in index template [fusion-logs]"

You should remove alias [active-fusion-logs] from the index template [fusion-logs].

PUT _template/fusion-logs
{
  "template": "fusion-logs-*",
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "routing.allocation.include.box_type": "hot"
  },

  "aliases": {
    "active-fusion-logs": {}, // Remove this line
    "search-fusion-logs": {}
  },
  "mappings": {
    "logs": {
      "properties": {
        "host": {
          "type": "keyword"
        },
        "job_id": {
          "type": "keyword"
        },
        "job_result": {
          "type": "keyword"
        }
      }
    }
  }
}
```

[Somnath-Guthula]

Oh. Great. That worked. I am not sure how I missed that out. Thanks! @dnhatn

[kkr78]

I having this problem with 6.4

PUT _template/application-logs
{
"template": "xx-*",
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"routing.allocation.include.box_type": "hot",
"index": {
"codec": "best_compression",
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s"
}
},
"mappings": {
"_doc": {
"properties": {
"date": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
"logData": {"type": "text"},
"message": {"type": "text"},
"logger_name": {"type": "keyword"},
"thread_name": {"type": "keyword"},
"level": {"type": "keyword"},
"levelvalue": {"type": "long"},
"stack_trace": {"type": "text"}
}
}
},
"aliases": {
"search-application-logs": {}
}
}

POST /search-application-logs/_rollover?dry_run
{
"conditions": {
"max_age": "1d",
"max_docs": 5,
"max_size": "5gb"
}
}
"reason": "Rollover alias [search-application-logs] can point to multiple indices, found duplicated alias [[search-application-logs]] in index template [application-logs]"

I would like to setup rollover policy on alias so it would take effect on all the indexes that follow pattern setup in template.

[kkr78]

My application will follow the date format similar to mentioned in this article https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html

Multiple indexes logs-2018.09.09-1 and logs-2018.09.10-1 would be pointing to same alias "logs_write". how to best setup rollover in this type of situation?

PUT logs-2018.09.09-1
{
"aliases": {
"logs_write": {}
}
}

PUT logs-2018.09.10-1
{
"aliases": {
"logs_write": {}
}
}

PUT logs-2018.09.10-1/_doc/1
{
"message": "a dummy log"
}

POST logs_write/_refresh

POST /logs_write/_rollover
{
"conditions": {
"max_docs": "1"
}
}

    "type": "illegal_argument_exception",
    "reason": "source alias maps to multiple indices"

[titan1978]

I am getting the same problem as @kkr78. I just have ONE index though. This is occurring on 6.3.0.

"reason": "Rollover alias [my-index] can point to multiple indices, found duplicated alias [[my-index]] in index template [mytemplate]"

Index : my-index-2018.09.01-1
Alias : my-index

{
    "mytemplate": {
        "order": 0,
        "index_patterns": [
            "my-index-*"
        ],
        "settings": {},
        "mappings": {
            "_doc": {
                "properties": {
                    "@timestamp": {
                        "type": "date"
                    }
                }
            }
        },
        "aliases": {
            "my-index": {}
        }
    }
}