Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dtest fail with sorted_tokens is empty in first_token_index #3382

Closed
asias opened this issue Apr 24, 2018 · 11 comments

Comments

@asias
Copy link
Contributor

commented Apr 24, 2018

Scylla version (or git commit hash): 11940ca

update_cluster_layout_tests.py:TestUpdateClusterLayout.increment_decrement_counters_in_threads_nodes_restarted_test

ERROR 2018-04-24 19:03:17,414 [shard 1] token_metadata - sorted_tokens is empty in first_token_index!
@duarten

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2018

This happens in many tests. I think it may be related to #3121.

@duarten duarten added the bug label Apr 24, 2018

@juliayakovlev

This comment has been minimized.

Copy link

commented Apr 24, 2018

I received same problem despite the #3121 is closed.
I ran dtest (materialized_views_test.py) on scylla master includes the commit that the #3121 is closed with:

[juliay@juliay scylla]$ git branch

  • master
    [juliay@juliay scylla]$ git log | grep 3c8ed25
    commit 3c8ed25
@slivne

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

@juliayakovlev - is increment_decrement_counters_in_threads_nodes_restarted_test supposed to be enabled ? I am not seeing it being run

@juliayakovlev

This comment has been minimized.

Copy link

commented Jul 16, 2018

@slivne I am not familiar with increment_decrement_counters_in_threads_nodes_restarted_test test. I seems the question not for me

@roydahan

This comment has been minimized.

Copy link

commented Aug 1, 2018

It's weird, I see it's part of the dtest master run till July 2017 (mails from Jenkins), but I can't find any trace for it being part of scylla-tests file.

@tzach

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2018

@roydahan @slivne what is the way forward with this issue?

@slivne

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2018

@roydahan - maybe we ran all the tests in update_cluster_layout_tests.py at some point and did a change to run a subset and forgot to add these.

@slivne slivne assigned roydahan and unassigned asias Sep 2, 2018

@slivne

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2018

I ran the test manually 2 times and the issue does not occur - I have not seen this reported for a while - following the conclusion about the tests and why they are not run lets close the issue

tx @roydahan

@slivne slivne added the fixed label Sep 2, 2018

@slivne slivne modified the milestones: 2.3, 2.4 Sep 2, 2018

@roydahan

This comment has been minimized.

Copy link

commented Sep 3, 2018

Since my comment I added back the test to scylla-tests.
I think the reason the test had run before is that at some point someone changed master to run with a private repo and after that we changed it back.

@slivne slivne removed the fixed label Sep 4, 2018

@slivne slivne assigned asias and unassigned roydahan Sep 4, 2018

@slivne slivne modified the milestones: 2.4, 2.3 Sep 4, 2018

@asias

This comment has been minimized.

Copy link
Contributor Author

commented Sep 5, 2018

Patch on list: Subject: [PATCH scylla] storage_service: Wait for range setup before announcing join status

avikivity added a commit that referenced this issue Sep 5, 2018

storage_service: Wait for range setup before announcing join status
When a joining node announcing join status through gossip, other
existing nodes will send writes to the joining node. At this time, it
is possible the joining node hasn't learnt the tokens of other nodes
that causes the error like below:

   token_metadata - sorted_tokens is empty in first_token_index!
   storage_proxy - Failed to apply mutation from 127.0.4.1#0:
   std::runtime_error (sorted_tokens is empty in first_token_index!)

To fix, wait for the token range setup before announcing the join
status.

Fixes: #3382
Tests: 60 run of materialized_views_test.py:TestMaterializedViews.add_dc_during_mv_update_test

Message-Id: <01abb21ae3315ae275297e507c5956e5774557ef.1536128531.git.asias@scylladb.com>
(cherry picked from commit 89b769a)

syuu1228 added a commit to syuu1228/scylla that referenced this issue Sep 22, 2018

storage_service: Wait for range setup before announcing join status
When a joining node announcing join status through gossip, other
existing nodes will send writes to the joining node. At this time, it
is possible the joining node hasn't learnt the tokens of other nodes
that causes the error like below:

   token_metadata - sorted_tokens is empty in first_token_index!
   storage_proxy - Failed to apply mutation from 127.0.4.1#0:
   std::runtime_error (sorted_tokens is empty in first_token_index!)

To fix, wait for the token range setup before announcing the join
status.

Fixes: scylladb#3382
Tests: 60 run of materialized_views_test.py:TestMaterializedViews.add_dc_during_mv_update_test

Message-Id: <01abb21ae3315ae275297e507c5956e5774557ef.1536128531.git.asias@scylladb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.