Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk request against multiple indices fails on missing index instead of failing individual actions #6410

Closed
ppf2 opened this Issue Jun 4, 2014 · 5 comments

Comments

Projects
None yet
7 participants
@ppf2
Copy link
Member

commented Jun 4, 2014

Consider the following repro use case:

PUT /bulkindex1
PUT /bulkindex2
POST /_bulk
{"index":{"_id":"1","_type":"index1_type","_index":"bulkindex1"}}
{"text": "hallo1" }
{"index":{"_id":"1","_type":"index2_type","_index":"bulkindex2"}}
{"text": "hallo2" }
GET /bulkindex*/_search
POST /bulkindex2/_close
POST /_bulk
{"index":{"_id":"1","_type":"index1_type","_index":"bulkindex1"}}
{"text": "hallo1-update" }
{"index":{"_id":"1","_type":"index2_type","_index":"bulkindex2"}}
{"text": "hallo2" }

The 2nd bulk action will certainly fail since bulkindex2 is closed. However, when the _bulk request is submitted, ES fails the entire _bulk request:

{
   "error": "IndexMissingException[[bulkindex2] missing]",
   "status": 404
}

Expected behavior in this case is for ES to still process the request against the other index that is available and report the 404 as part of the response for the action against bulkindex2 like the following:

{
   "took": 14,
   "errors": true,
   "items": [
      {
         "index": {
            "_index": "bulkindex1",
            "_type": "index1_type",
            "_id": "1",
            "_version": 3,
            "status": 200
         }
      },
      {
         "index": {
            "_index": "bulkindex2",
            "_type": "index2_type",
            "_id": "1",
            "status": 404,
   "error": "IndexMissingException[[bulkindex2] missing]",
         }
      }
   ]
}

@ppf2 ppf2 changed the title Bulk request fails on missing index instead of failing individual actions Bulk request against multiple indices fails on missing index instead of failing individual actions Jun 4, 2014

@tmkujala

This comment has been minimized.

Copy link

commented Jun 12, 2014

This issue was giving me some trouble as well, saw it was fixed in the 1.1 and 1.2 branches, see #4987.

@TheLudd

This comment has been minimized.

Copy link

commented Jul 8, 2014

@tmkujala This bug is not fixed. It is not the same as #4987. That one tackles invalid index names. This one talks about performing bulk operations against indices that does not exits and where the response would be 404. This of course only occurs if you have the option action.auto_create_index set to false. But I experience this error as well.

@clintongormley

This comment has been minimized.

Copy link
Member

commented Jul 8, 2014

@spinscale I've confirmed that this bug still exists in master.

@TheLudd

This comment has been minimized.

Copy link

commented Jul 9, 2014

@clintongormley @spinscale
I have seen a few similar issues like #4987 which had the same effect but was caused by a different error. Is there any reason why there are checks for specific errors in the bulk items and the errors that do not match those cause the full request to fail?

Right now I index my documents in elasticsearch one by one but I am planing on queuing up our incoming data in redis and then bulk it in to optimize performance. The errors that could occur when those bulks are indexed cannot be known but I was hoping that if one index fails the others will not. Can I trust this to be the case?

@s1monw s1monw removed the v1.2.2 label Jul 9, 2014

spinscale added a commit to spinscale/elasticsearch that referenced this issue Jul 11, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Closes elastic#6410

@s1monw s1monw added v1.4.0 and removed v1.3.0 labels Jul 14, 2014

@bytenik

This comment has been minimized.

Copy link

commented Aug 4, 2014

Has any progress been made on this? Its a huge performance show-stopper for my company.

@s1monw s1monw assigned s1monw and unassigned spinscale Sep 10, 2014

GaelTadh added a commit to GaelTadh/elasticsearch that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes elastic#6410

@GaelTadh GaelTadh closed this in 405e581 Sep 19, 2014

GaelTadh added a commit that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410

GaelTadh added a commit that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410

GaelTadh added a commit to GaelTadh/elasticsearch that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes elastic#6410

GaelTadh added a commit that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410

GaelTadh added a commit that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410

GaelTadh added a commit that referenced this issue Sep 19, 2014

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes elastic#6410

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes elastic#6410
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.