Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_reindex in 5.4 fails against 1.7.6 #24520

Closed
lemkepf opened this issue May 5, 2017 · 26 comments

Comments

Projects
None yet
@lemkepf
Copy link

commented May 5, 2017

Elasticsearch version: 1.7.6 and 5.4

Plugins installed: None

JVM version: 1.8.0_111, 1.8.0_131

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:
We are migrating from a 1.7.6 cluster to 5.4 using the reindex from remote in 5.4. I'm getting a weird ElasticsearchParseException error on the response after executing the request to reindex.

Here is my _reindex command I ran on my 5.4 cluster:

POST - http://search02:9200/_reindex
{
  "source": {
    "remote": {
      "host": "http://search01:9200"
    },
    "index": "labor"
  },
  "dest": {
    "index": "labor"
  }
}

Here is the response I got:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://search01:9200/_search/scroll?scroll=300000000000nanos: HTTP/1.1 400 Bad Request\n{\"error\":\"ElasticsearchParseException[Failed to parse [300000000000nanos]]; nested: NumberFormatException[For input string: \\\"300000000000nano\\\"]; \",\"status\":400}"
    }
  },
  "status": 400
}

I can execute a request with a "size" parameter in place and it works, but only the first 200 records.

{
  "size": 200,
  "source": {
    "remote": {
      "host": "http://search01:9200"
    },
    "index": "labor",
    "size": 2000,
    "sort": { "_id": "desc" }
  },
  "dest": {
    "index": "labor"
  }
}

If that size parameter goes up to 300 I get the same error as before but with a different scroll number:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://search01:9200/_search/scroll?scroll=2620000083934nanos: HTTP/1.1 400 Bad Request\n{\"error\":\"ElasticsearchParseException[Failed to parse [2620000083934nanos]]; nested: NumberFormatException[For input string: \\\"2620000083934nano\\\"]; \",\"status\":400}"
    }
  },
  "status": 400
}

I've played around with ?requests_per_second but that didn't seem to make any difference. I am seriously stumped... any ideas what I'm doing wrong?
Thanks

@nik9000 nik9000 added the :Reindex API label May 5, 2017

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 5, 2017

any ideas what I'm doing wrong?

Nothing. We don't have tests that run reindex-from-remote against 1.7 because 1.7 doesn't fit into our testing infrastructure. I broke this.

@nik9000 nik9000 added the v5.4.0 label May 5, 2017

@nik9000 nik9000 self-assigned this May 5, 2017

@lemkepf

This comment has been minimized.

Copy link
Author

commented May 5, 2017

@nik9000 - Ha. Is this something on the radar or should I be looking at other solutions? elasticdump is my next option.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 5, 2017

It is on my radar now, yeah. I've started working on something to run the old Elasticsearch now. But no matter what I do it isn't going to be released super fast. New versions of 5.3 ought to work but whatever works for you works for you.

@apierleoni

This comment has been minimized.

Copy link

commented May 8, 2017

Same thing happening here, when reindexing from 2.4.4 to 5.4

Query

{
  "source": {
    "remote": {
      "host": "http://myhost:9200"
    },
    "index": "myindex",
    "size": 100,
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "myindex"
  }
}

Output

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://myhost:9200/_search/scroll?scroll=300000000000nanos: HTTP/1.1 400 Bad Request\n{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\"}],\"type\":\"parse_exception\",\"reason\":\"Failed to parse [300000000000nanos]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"300000000000nano\\\"\"}},\"status\":400}"
    }
  },
  "status": 400
}
@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 8, 2017

Right. I'm working on #23828 which will give us real tests that we don't break against older versions at the same time as I fix this. So I can be sure I'm not breaking anything else....

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 9, 2017

nik9000 added a commit to nik9000/elasticsearch that referenced this issue May 9, 2017

Tests for reindex-from-remote against old versions
Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes elastic#23828
Closes elastic#24520
@ejsmith

This comment has been minimized.

Copy link

commented May 10, 2017

I'm running into this issue as well.

nik9000 added a commit that referenced this issue May 11, 2017

Add qa module that tests reindex-from-remote against pre-5.0 versions…
… of Elasticsearch (#24561)

Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes #23828
Closes #24520

nik9000 added a commit that referenced this issue May 11, 2017

Add qa module that tests reindex-from-remote against pre-5.0 versions…
… of Elasticsearch (#24561)

Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes #23828
Closes #24520

nik9000 added a commit that referenced this issue May 11, 2017

Add qa module that tests reindex-from-remote against pre-5.0 versions…
… of Elasticsearch (#24561)

Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes #23828
Closes #24520
@MichaelPalmer-Orange

This comment has been minimized.

Copy link

commented May 11, 2017

Likewise getting this error as well. Running 5.4 _reindex against 2.4 - if I DON'T have a query or _source section in my remote section of the reindex, I don't get the issue but I get other issues (that I am trying to circumvent by using the _source).

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 11, 2017

I merged the fix for this this morning. It included much better testing for reindex-from-remote against older versions of Elasticsearch so this shouldn't happen again. At least not in so obvious a way. It should ship with 5.4.1.

@MichaelPalmer-Orange

This comment has been minimized.

Copy link

commented May 11, 2017

Any chance we have an ETA on that?

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 11, 2017

Any chance we have an ETA on that?

We have a policy of not predicting our future release dates publicly because we don't want to disappoint.

@tommymonk

This comment has been minimized.

Copy link

commented May 12, 2017

Also impacting us in production.

@vsiv

This comment has been minimized.

Copy link

commented May 15, 2017

We are also impacted by this. 5.3.2 does not have this problem. This might be an option for some who cant wait for 5.4.1. We can't however because we depend on word_delimiter_graph.

@DennisDyallo

This comment has been minimized.

Copy link

commented May 18, 2017

Just chiming in here to add that the source.size is the property that sets the TOTAL number of documents that get reindexed.

For example:
POST _reindex?wait_for_completion=false
{
"size": 10000,
"conflicts": "proceed",
"source": {
"size":10,
"remote": {
"host": "https://example.com:9200",
"username": "xxx",
"password": "yyy"
},
"index": ["sourceIndex"]
},
"dest": {
"index": "destinationIndex",
"op_type": "create"
}
}

Will put 10 documents in destinationIndex and then return, even though the reindex.size (query size) is set to 10000.

This is reindexing from 2.4.4 to 5.4 on Elastic Cloud.

PS. You should also change the names of these properties, it's confusing

@vsiv

This comment has been minimized.

Copy link

commented May 19, 2017

@DennisDyallo does that mean 5.4 would still work with the proper size set?

@DennisDyallo

This comment has been minimized.

Copy link

commented May 20, 2017

No, is you omit the total size (10000) it will just take the amount for the second size (10)

@dragonjohn

This comment has been minimized.

Copy link

commented May 25, 2017

Hi @nik9000 , doest it works from elasticsearch 1.4.5 to 5.4? I faced the same error output after did the reindex. Thanks.

@vsiv

This comment has been minimized.

Copy link

commented May 26, 2017

@nik9000 I think it might help if its possible to add a note to the docs of 5.4.0 reindex API as people might be unaware of this issue and get confused why reindex is not working .. It might save people some time..

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 26, 2017

@nik9000 I think it might help if its possible to add a note to the docs of 5.4.0 reindex API as people might be unaware of this issue and get confused why reindex is not working .. It might save people some time..

Yeah. I'll push something.

nik9000 added a commit that referenced this issue May 26, 2017

Docs: Link to reindex-from-remote issue
Reindex from remote is broken in 5.4.0 and will be fixed in 5.4.1.
Lots of folks come to the github issue after searching. Adding this
link should save people time searching for the github issue. Hopefully.

Relates to #24520
@monikamaheshwari

This comment has been minimized.

Copy link

commented May 30, 2017

@nik9000 I am getting same error.How to download 5.4.1 package.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented May 30, 2017

@vovolie

This comment has been minimized.

Copy link

commented Jun 1, 2017

same issue. reindex 1.6.0 to 5.4
dates release 5.4.1?
Thanks.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Jun 1, 2017

5.4.1 is just now live.

@tinder-xhao

This comment has been minimized.

Copy link

commented Jul 25, 2017

Is 5.4.1 available in docker?

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Jul 25, 2017

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Jul 25, 2017

5.4.1 should also be available though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.