Cassandra clusters specify initial token for large startup speedup #844
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rukai
force-pushed
the
initial_tokens
branch
4 times, most recently
from
aedddd5
to
1c8a6fc
Compare
conorbros
approved these changes
Oct 10, 2022
XA21X
approved these changes
Oct 10, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The shotover-int-tests/cassandra:4.0.6 image no longer sets a CMD, it now falls back to
CMD ["cassandra", "-f"], which is the safest default.None of our
docker-compose.ymls actually use that fallback though and instead overwrite it in various ways as detailed below:Cluster tests
We make use of
shotover-int-tests/cassandra:4.0.6to set some config we previously relied on bitnami for and then use the custom command to set initial_token, which even bitnami didn't offer.In order to keep the token list more readable we do some weird things with environment variables.
The token list is set to the docker-compose env var CASSANDRA_INITIAL_TOKENS.
Then the container uses that environment variable at runtime.
We have to use
$$to escape the$otherwise docker-compose will substitute the variable with an empty string, rather than letting the container do the subsitution.Manually specifying the tokens allows cassandra cluster init to skip a lot of waiting around and takes the time for initialization to complete on my machine from 70s to 15s
It will also allow us to improve the quality of the integration tests as we will be able to make assertions on the tokens returned in system.local.
This will be done in a follow up though.
The tokens were chosen by extracting them from the naturally randomly generated tokens of a real cluster.
Possibly 128 tokens per node is excessive and it would be easier to deal with if we reduced it to say 16per node, but I'll leave that for a potential follow up.
The
dc1->Marschange intest_cassandra_peers_rewrite_cassandra4was needed because its docker-compose.yml was settingCASSANDRA_DC: Marswhich was previously unused by the bitnami image, but now that we are using the official image as a base theCASSANDRA_DCfield is being properly picked up.Single instance tests
The only change here was to set the command:
cassandra -f -Dcassandra.skip_wait_for_gossip_to_settle=0 -Dcassandra.initial_token=0I think this is an improvement as setting the command explicitly shows the mild hack these tests use to speed up initialization.