Java API: IndicesOptions.lenientExpandOpen() required index names to contain "*"

[jjbursik]

Elasticsearch version: 5.1.1

Description of the problem including expected versus actual behavior:
Using aliases to manage indices used for search, I'm unable to ignore closed indices without specifying a wildcard search char ("*") in the index name. Here are my sample SearchRequestBuilders that works and doesn't work:

Works:

SearchRequestBuilder srb=elasticsearch.getClient().prepareSearch("reportsource*")
                .setIndicesOptions(IndicesOptions.lenientExpandOpen())
                .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                .setQuery(functionScoreQueryBuilder)
                .addSort("_score", SortOrder.DESC)
                .setFrom(page * size).setSize(size);

SearchResponse searchResponse = srb.execute().actionGet();

Doesn't work:

SearchRequestBuilder srb=elasticsearch.getClient().prepareSearch("reportsource")
                .setIndicesOptions(IndicesOptions.lenientExpandOpen())
                .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                .setQuery(functionScoreQueryBuilder)
                .addSort("_score", SortOrder.DESC)
                .setFrom(page * size).setSize(size);

SearchResponse searchResponse = srb.execute().actionGet();

I would have expected that I don't need the "*" in the index name to ignore closed indices having specified IndicesOptions.lenientExpandOpen(). Using Sense, I don't see the same requirement:

GET reportsource/_search?ignore_unavailable=true,allow_no_indices=true,expand_wildcards_open=true,expand_wildcards_closed=false,allow_alisases_to_multiple_indices=true,forbid_closed_indices=false&pretty&search_type=dfs_query_then_fetch
{
  "from" : 0,
  "size" : 10,
  "query" : {
    "function_score" : {
      "query" : {
        "bool" : {
          "should" : [
            {
              "match" : {
                "name.raw" : {
                  "query" : "Sub Agreement Id",
                  "operator" : "OR",
                  "prefix_length" : 0,
                  "max_expansions" : 50,
                  "fuzzy_transpositions" : true,
                  "lenient" : false,
                  "zero_terms_query" : "NONE",
                  "boost" : 1.0
                }
              }
            }
          ],
          "disable_coord" : false,
          "adjust_pure_negative" : true,
          "minimum_should_match" : "2",
          "boost" : 1.0
        }
      },
      "functions" : [
        {
          "filter" : {
            "match_all" : {
              "boost" : 1.0
            }
          },
          "field_value_factor" : {
            "field" : "rating",
            "factor" : 1.0,
            "missing" : -0.3499999940395355,
            "modifier" : "log2p"
          }
        }
      ],
      "score_mode" : "multiply",
      "boost_mode" : "multiply",
      "max_boost" : 2.0,
      "boost" : 1.0
    }
  },
  "ext" : { }
}

Steps to reproduce:

1. Create 2 indexes, both with aliases "reportsource"
2. Close 1 index
3. Use the SearchRequestBuilder code above, you can remove the query if needed, and you'll either get an error or not. Error for index closed pasted below.

Provide logs (if relevant):
Error log:

2017-03-12 16:05:44.608 ERROR 72028 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ClusterBlockException[blocked by: [FORBIDDEN/4/index closed];]] with root cause

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/4/index closed];
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:178) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.buildPerIndexAliasFilter(TransportSearchAction.java:80) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1343) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1242) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-client-5.1.1.jar:5.1.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

[javanna]

Hi @jjbursik ,
the indices options api can definitely be improved, especially how it is exposed in our Java API.

That said, your REST call doesn't doesn't provide any indices options after all. The right separator between the different arguments is & and not ,. You end up passing in a huge string (ignore_unavailable=true,allow_no_indices=true,expand_wildcards_open=true,expand_wildcards_closed=false,allow_alisases_to_multiple_indices=true,forbid_closed_indices=false) for the ignore_unavailable option, which gets ignored as it is not a boolean value (this will cause an error in 6.0, see #22200).

The only valid arguments in your REST call are pretty&search_type=dfs_query_then_fetch. That has the same effect as not passing in any indices options, hence using the default indices options for SearchRequest : IndicesOptions#strictExpandOpenAndForbidClosed. The forbid closed is the important part here, which is an internal flag that defines whether closed indices are to be treated as unavailable or not (varies api by api). This bit is implicitly set to false in IndicesOptions#lenientExpandOpen which causes the error you are seeing. You could create lenient indices options with forbid closed set to true this way: IndicesOption.fromOptions(true, true, true, false, IndicesOption.strictExpandOpenAndForbidClosed).

Your REST call though should return an error when performed against a closed index, or an alias that points to at least one closed index, something like this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_closed_exception",
        "reason" : "closed",
        "index_uuid" : "hcUaFRjURxO6Yu6RvqY63g",
        "index" : "test2"
      }
    ],
    "type" : "index_closed_exception",
    "reason" : "closed",
    "index_uuid" : "hcUaFRjURxO6Yu6RvqY63g",
    "index" : "test2"
  },
  "status" : 403
}

Also, when you say that you didn't expect to have to use the wildcard in your index name, that is correct. Do keep in mind though that one thing is expanding wildcard expressions, controlled via expand_wildcards option, and another thing is ignoring unavailable indices when provided explicitly, controlled via ignore_unavailable.

Conclusion:

1. check your REST call, it should return an error as a closed index is being hit and indices options are not lenient.
2. Provide ignore_unavailable=true to the REST call with explicit index names to make it lenient, meaning that closed indices will not be hit.
3. Update your Java api code to do the same by initializing indices options this way: IndicesOption.fromOptions(true, true, true, false, IndicesOption.strictExpandOpenAndForbidClosed)