Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forced awareness fails to balance shards #3580

Closed
twinforces opened this Issue Aug 27, 2013 · 14 comments

Comments

Projects
None yet
2 participants
@twinforces
Copy link

twinforces commented Aug 27, 2013

I built a cluster for amazon zone awareness as follows:

Background: I've setup the number of shards to 3, and I have allocation awareness set to match the awszone property which I'm setting in a config.

There's 6 nodes total:

2 in us-west-2a
2 in us-west-2b
1 in us-east-2a
1 in us-east-2b

So in theory, I should be able to lose either an entire side of the country. I would also expect the nodes in us-west to balance the shards, not just allocate a single shard to the machine.

However, the western zone shards aren't balancing except to move 1 shard onto each of the other hosts in the zone.

S1monw is on the case.

Discussion here: https://groups.google.com/forum/m/#!topic/elasticsearch/9yZw7sryFb4

@ghost ghost assigned s1monw Aug 27, 2013

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 27, 2013

thanks for opening this!

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Aug 27, 2013

Hey, no problem! Thanks for digging in to fix this!

Sent from my iPad

On Aug 27, 2013, at 3:57 AM, Simon Willnauer notifications@github.com wrote:

thanks for opening this!


Reply to this email directly or view it on GitHub.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 27, 2013

I added a test for this to master which awaits a fix... I think I know the problem here but I need to verfiy it. Seems like a problem in the shards allocator algorithm.

see: e7ff8ea

s1monw added a commit to s1monw/elasticsearch that referenced this issue Aug 28, 2013

Prevent allocation algorithm from prematurely exiting balance operations
In a special case if allocations from the "heaviest" to the "lighter" nodes is not possible
due to some allocation decider restrictions like zone awareness. if one zone has for instance
less nodes than another zone so one zone is horribly overloaded from a balanced perspective but we
can't move to the "lighter" shards since otherwise the zone would go over capacity.

Closes elastic#3580

@s1monw s1monw closed this in b9558ed Aug 28, 2013

s1monw added a commit that referenced this issue Aug 28, 2013

Prevent allocation algorithm from prematurely exiting balance operations
In a special case if allocations from the "heaviest" to the "lighter" nodes is not possible
due to some allocation decider restrictions like zone awareness. if one zone has for instance
less nodes than another zone so one zone is horribly overloaded from a balanced perspective but we
can't move to the "lighter" shards since otherwise the zone would go over capacity.

Closes #3580
@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 28, 2013

pushed to 0.90 & master. @twinforces would you be able to build a snapshot from 0.90 branch and try the fix in your env?

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Aug 28, 2013

Yeah, let me try that.

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Aug 28, 2013

Dumb question. Is there an easy way for me to do this by updating one node and designating it the primary?

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 30, 2013

not really, you would need to do a full upgrade I guess. 0.90.4 is not too far away you might wanna just wait? I mean from my perspective it would be great if you could verify the problem is solved!

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Aug 30, 2013

Maybe I'll just do it manually on my laptop.

Sent from my iPad

On Aug 30, 2013, at 12:23 PM, Simon Willnauer notifications@github.com wrote:

not really, you would need to do a full upgrade I guess. 0.90.4 is not too far away you might wanna just wait? I mean from my perspective it would be great if you could verify the problem is solved!


Reply to this email directly or view it on GitHub.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Oct 6, 2013

@twinforces this has been released can you confirm that this bug is fixed in your scenario?

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Oct 11, 2013

Sorry, out with the flu.

I tried to confirm it immediately after release, but it didn't quite behave like I expected.

Basically, the problem is that since I initially found this, I set the number of replicas set such that every node has a copy of each shard to work around the problem. (i.e. 5, because 4 zones, 2 nodes in West1, 2 nodes in West2, 1 node in east1, 1 node in east2)

After I upgraded to 0.90.5, I thought, ok, and set the number of replicas to 4.

It removed all shards from one of the nodes, which was not what I expected. And it didn't rebalance after that.

So I set it back to 5 and then had to wait while the shards got recopied back to that node.

Additionally, this didn't seem to affect primary allocation, because I have one node that's primary for all of the shards.

screen shot 2013-10-11 at 11 23 09 am

Would that be what you expected to happen?

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Oct 11, 2013

hmm this seems pretty much balanced no? Sorry I am confused a bit - what is wrong with the shard allocation in that picture? maybe you can describe what you would expect?

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Oct 11, 2013

Shouldn't the shard masters be distributed? (Thick lines)

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Oct 13, 2013

Well a primary might not be relocated at all if you upgrade since the cost of a relocation might outrule the fact that it is a primary. It doesn't matter that much to be honest. if you delete your index and recreate it does it still look like this?

@twinforces

This comment has been minimized.

Copy link
Author

twinforces commented Oct 16, 2013

What's the cost of a relocation? Isn't it just kind of just a "blessing"?

Ok, we experimented some more today.
screen shot 2013-10-16 at 3 45 53 pm

Here's what I did:

ES had run out of memory on the primary. That caused the primary to switch to the 10.100 network, which only has two nodes. (They're basically live backups). I wanted to try to see if the nodes would come back immediately after a restart, plus I want to check the balancing.

I locked allocation based on a thread on the mailing list of someone telling me that they would restart faster if I locked allocation before restarting nodes.
I killed the two eastern nodes, forcing it to elect a new primary in the west.
I reduced the number of replicas, which caused it to completely remove a node from elasticsearch.
I then started up the two eastern nodes, expecting them to come back immediately. They didn't.
Then I gave up and turned allocation back on, and bumped the number of replicas back up.
Now I have the above, the primary is in the west, its bootstrapping the node in the west that got kicked out, and its bootstrapping the nodes in the east.

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Prevent allocation algorithm from prematurely exiting balance operations
In a special case if allocations from the "heaviest" to the "lighter" nodes is not possible
due to some allocation decider restrictions like zone awareness. if one zone has for instance
less nodes than another zone so one zone is horribly overloaded from a balanced perspective but we
can't move to the "lighter" shards since otherwise the zone would go over capacity.

Closes elastic#3580
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.