DC names with underscore in name not correctly supported

[0x6373]

I'm running version 1.4.1 and just added a new node in a new datacenter to cassandra.

Before i had "dc1" and "dc2" and added the new node in "dc1_arch".
"nodetool status" correctly reports the new node in the new datacenter, but Cassandra Reaper has added the node to "dc1".

┆Issue is synchronized with this Jira Task by Unito
┆Issue Number: K8SSAND-389

[0x6373]

Any ideas where this could originate from?

From the communication between browser and reaper i see that the browser already receives only "dc1" and "dc2" and the node in "dc1_arch" as belonging to "dc1".

I tried to search through the code and found nothing obvious, but i'm not a java developer.
But i guess this happens when the data is queried via JMX.

[baileygm]

I noticed the same issue. I haven't used reaper before this version but in the Clusters tab it shows all nodes in DC's containing an underscore as in the same datacenter. I'm also unable to view any jmx metrics from those nodes.

[baileygm]

Looks like a duplicate of #587

[0x6373]

I just verified that JMX provides the correct value for "Datacenter", so the mangling appears to come from the reaper server code.

How is the dc_suffix being reported by JMX/nodetool?

Until now i did not find any code wich would split the datacenter name on some characters, but i'm mostly grep'ing the code so probably not the best way.

[a-nldisr]

Checked and still present in reaper version 2.3.1. As you can see the nodetool status reports 2 Datacenters:

Datacenter: dc1_eu1a_test
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  node-1    218.64 GB  256          58.7%             id1  rac1
UN  node-2    191.4 GB   256          59.6%             id2  rac1
UN  node-3    280.03 GB  256          60.7%             id3  rac1
UN  node-4     191.3 GB   256          59.1%             id4  rac1
UN  node-5      226.6 GB   256          62.0%             id5  rac1
Datacenter: dc1_eu1b_test
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  node-1 468 GB     256          100.0%            id1  rack1
UN  node-2  312.31 GB  256          100.0%            id2  rack1
UN  node-3   309.46 GB  256          100.0%            id3  rack1

The UI lists them as 1 single datacenter:

We have 3x replication per datacenter, we run 1 reaper in dc1_eu1b_test (other DC cant support remote JMX due to restrictions in the framework running Cassandra). So we have REAPER_CASS_LOCAL_DC configured and DATACENTER_AWARE=LOCAL. Will Reaper ignore this now? Or is this purely an UI bug?

In the advanced repair settings you cannot select datacenters or override with your own values:

When trying to repair with parallel config and incremental repair this causes issues:

Nodetool gossipinfo lists all the racks accordingly:

  DC:8:dc1_eu1a_test
  DC:8:dc1_eu1b_test

@michaelsembwever Is there any other information you would need?

[adejanovski]

Hi @a-nldisr, this is purely a UI bug and the repairs will not be affected (although you can't select individual datacenters for your repair indeed).
I'll see if it can be fixed easily or not.

[weisdd]

@adejanovski I've just opened a PR with the bugfix: #1129

Test build image is published here: quay.io/weisdd/cassandra-reaper:b1d3b39

By the way, if rack names can contain underscores as well, then we can also apply the same fix to their patterns.

[adejanovski]

Fixed by #1129