Revert "Maintain dc/rack by topology" ++

[xemul]

This set reverts the large recent change that moved topology (dc and rack) information onto token_metadata::topology and some more changes made on top of it. The motivation is that change revealed some nasty hidden land-mines not all of which had been fixed after it.

@asias , it turned out that #11936 stood on the way. It turned out to be simpler to reverted first, then reverted this revert fixing few trivial conflicts afterwards. Please, check

@bhalevy , I had to revert #12001 , and getting it back was not trivial, sorry :(

@kbr-, I tried to keep #11998 (all the more so it didn't cause direct conflicts) but not 100% sure it stays correct, please check

https://jenkins.scylladb.com/job/releng/job/Scylla-CI/3138/ is on its way

[kbr-]

But... why?

@kbr-, I tried to keep #11998 (all the more so it didn't cause direct conflicts) but not 100% sure it stays correct, please check

It seems that #11998 is kept, but I don't know about #11942:

-        && _token_metadata->get()->get_topology().has_endpoint(addr, locator::topology::pending::yes);
+        && _token_metadata->get()->get_topology().has_endpoint(addr);

but I see that after these reverts, there's no such thing as _pending_locations.
Are pending endpoints now kept in _current_locations?

[xemul]

Are pending endpoints now kept in _current_locations?

That's the problem -- pending endpoints are not kept on topology

[xemul]

@benipeled , need your help here. The CI fails on sanity-tests stage without staring any test

[benipeled]

@benipeled , need your help here. The CI fails on sanity-tests stage without staring any test

Caused by https://github.com/scylladb/scylla-pkg/pull/3159 - I dequeued it - you can retrigger the CI

[xemul]

https://jenkins.scylladb.com/job/releng/job/Scylla-CI/3156/

[xemul]

☝️ fails:

[2022-11-24T15:15:22.398Z] [2580/2581] topology  debug  [ FAIL ] test_topology.2571 469.52s

[xemul]

The test caught bug #11780

[avikivity]

The test caught bug #11780

Do you mean, we had a unit test that caught the bug but we ignored it due to "flakiness"?

[xemul]

Do you mean, we had a unit test that caught the bug but we ignored it due to "flakiness"?

First the test was disabled. Next messaging service was taught (1bd2471) to keep connections to endpoints even if they were joining/leaving and this test was enabled.

After the revert topology doesn't maintain locations for joining/leaving endpoints, respectively the messaging service drops those connections at unwanted moments. It probably can be fixed by reverting the fix that made messaging-service topology-aware (a396c27 and 13ace7a at least)

[kbr-]

Do you mean, we had a unit test that caught the bug but we ignored it due to "flakiness"?

If I understand correctly, the sequence of events was as follows:

• Decommission can fail due to concurrent fiber dropping RPC connections #11780 was introduced at some point
• later we added test_topology.test_decommission_node_add_column, which reproduced that bug, but rarely, in a very flaky way. The test managed to get merged because it didn't hit the bug in the original CI run and next promotion.
• sometime later I debugged the flaky failures and opened Decommission can fail due to concurrent fiber dropping RPC connections #11780.
• Then I fixed Decommission can fail due to concurrent fiber dropping RPC connections #11780 and placed error injections with sleeps that significantly increased the probability of reproducing the bug.
• Now this revert re-introduces Decommission can fail due to concurrent fiber dropping RPC connections #11780, but apparently the injections work. (Or we're just (un)lucky :P)

[xemul]

but apparently the injections work. (Or we're just (un)lucky :P)

... or both 😛

[bhalevy]

Can this PR be closed?

[bhalevy]

@xemul can this PR draft be closed?

[xemul]

@xemul can this PR draft be closed?

Definitely