Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Transform] transform jobs on nodes < 7.4.0 mixed with nodes > 7.4 might fail #48019

Closed
hendrikmuhs opened this issue Oct 14, 2019 · 1 comment
Closed

[Transform] transform jobs on nodes < 7.4.0 mixed with nodes > 7.4 might fail #48019

hendrikmuhs opened this issue Oct 14, 2019 · 1 comment
Labels
>bug :ml/Transform Transform v7.4.1 v7.5.0 v8.0.0-alpha1

Comments

@hendrikmuhs
Copy link
Contributor

hendrikmuhs commented Oct 14, 2019 •
edited
Loading

Nodes in all versions of 7.2 and 7.3 might not see transform documents created in 7.4, 7.5 ...

Only after the switch to a versioned index in #45375 (7.4.0) retrieval uses a search pattern to read back documents (config, checkpoint, state). This allows to retrieve jobs even if the cluster is partially upgraded. But if nodes are on version 7.3 and 7.4 the hardcoded index name those documents might not be retrievable.

Long story short: Mixed version clusters with nodes <7.4 together with nodes >=7.4 do not work reliable.

Solution:

  • disable node allocation if any node < 7.4. This will prevent placing jobs on mixed clusters until the upgrade has completed
  • document as known limitation: transforms do not execute during a rolling upgrade from 7.2/7.3 to 7.x
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

This was referenced Oct 15, 2019
Closed
Merged
hendrikmuhs pushed a commit that referenced this issue Oct 15, 2019
[7.5][Transform] prevent assignment if any node is older than 7.4 (#4…
b2ce728 
…8055)

disable task assignment of transforms if any node uses version 7.2 or 7.2 (mixed cluster).

fixes #48019
hendrikmuhs pushed a commit to hendrikmuhs/elasticsearch that referenced this issue Oct 16, 2019
prevent assignment if any node is older than 7.4 (elastic#48055)
00686a1 
disable task assignment of transforms if any node uses version 7.2 or 7.3 (mixed cluster).

fixes elastic#48019
@hendrikmuhs hendrikmuhs mentioned this issue Oct 16, 2019
Merged
@hendrikmuhs hendrikmuhs mentioned this issue Oct 16, 2019
Merged
hendrikmuhs pushed a commit that referenced this issue Oct 17, 2019
prevent assignment if any node is older than 7.4 (#48055) (#48116)
1da3cc1 
disable task assignment of transforms if any node uses version 7.2 or 7.3 (mixed cluster).

fixes #48019
@romseygeek romseygeek mentioned this issue Oct 18, 2019
Closed
This was referenced Oct 18, 2019
Merged
Merged
hendrikmuhs pushed a commit that referenced this issue Oct 19, 2019
[7.4][Transform] Remove transform mixed cluster tests for 7.4 upgrade…
778a9ce 
… testing (#48239)

remove mixed cluster tests that test transform BWC due to #48019: upgrade of 7.2/3 to 7.4 suspends re-assignment of transforms and retrieval of transforms and transform stats is not reliable in mixed clusters.

fixes #48236
hendrikmuhs pushed a commit that referenced this issue Oct 21, 2019
blacklist transform tests in mixed cluster prior 7.4 (#48247)
4b3a672 
blacklist transform tests in mixed cluster prior 7.4, see #48019

fixes #47943
hendrikmuhs pushed a commit that referenced this issue Oct 21, 2019
blacklist transform tests in mixed cluster prior 7.4 (#48247)
2e30147 
blacklist transform tests in mixed cluster prior 7.4, see #48019

fixes #47943
@hendrikmuhs hendrikmuhs mentioned this issue Oct 21, 2019
Merged
@codebrain codebrain mentioned this issue Oct 25, 2019
Closed
39 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :ml/Transform Transform v7.4.1 v7.5.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Transform] prevent assignment to nodes older than 7.4 hendrikmuhs/elasticsearch
3 participants
@jakelandis @hendrikmuhs @elasticmachine