piping carbon-list to carbon-sieve produces nothing

[gerases]

Hi,

When I pipe carbon-list to carbon-sieve on the same carbon cache host, I'm expecting to see all the metrics that are piped to sieve, but I see nothing. What could be the reason? The config contains all the members of the cluster correctly.

Thanks,
Sergei

edit:

Oh, I see:

Finally, you can provide HASHING_TYPE of your cluster. Default is carbon_ch, also fnv1a_ch is supported. Please note that for using fnv1a_ch hashing you need carbon 1.0.2 or newer installed (or you need to use carbon-c-relay relay instead).

edit2:

my carbonate is hopelessly old (Build Date : Wed 08 Jul 2015 02:16:26 PM UTC).

I have 0.9.16 and I'm using fnv1a_ch. Shouldn't I get an error to the effect that my hashing algorithm is not recognized?

[gerases]

Today I created a python environment with the right python module versions. I have carbon 1.1.6 and carbonate 1.1.6. However, the results I'm getting are different from what carbon c relay is doing. The algorithm is fnv1a_ch in both cases. The same replication factor is used as well.

Basically, I just want to confirm that the hashing that carbon-c-relay and carbonate tools do match.

Any ideas?

[deniszh]

@gerases

I have 0.9.16 and I'm using fnv1a_ch. Shouldn't I get an error to the effect that my hashing algorithm is not recognized?

Well, it's quite hard to predict a future. You probably used old carbonate version, when fnv1a_ch hashing in carbon simple didn't exist.

Basically, I just want to confirm that the hashing that carbon-c-relay and carbonate tools do match.
Any ideas?

I implemented tests to test fnv1a_ch hashing and test that it's matching with carbon-c-relay (at least with version 3.1).
So, more context needed. How ring looks like in carbon-c-relay and how it looks like in carbonate config.

[gerases]

So, carbon-c-relay has:

cluster replication
  fnv1a_ch replication 2
      graphite101:2003
      graphite102:2003
      graphite103:2003
      graphite104:2003
    ;

match *
  send to
    replication
  ;

carbonate.conf:

[main]
DESTINATIONS = graphite101:2003,graphite102:2003,graphite103:2003,graphite104:2003
REPLICATION_FACTOR = 2
HASHING_TYPE = fnv1a_ch

Running carbonate-lookup -c carbonate.conf METRIC_NAME returns machines on which the metric doesn't actually exist.

For example:

carbon-lookup -c carbonate.conf test.test

prints 1 and 2, whereas in reality the metric is on 2 and 3.

I sanitized the hostnames so you may not get the same results.

P.S.: carbon-c-relay version is 2.6

[deniszh]

I think it's a issue similar to jjneely/buckytools#11
You're not using instances, so, Carbonate interpret graphite101:2003 as graphite101:2003:None
and carbon-c-relay is using '' instead of None. Instances in fnv1a_ch are not optional, unfortunately.
You need to use proper instances both in carsbon-c-relay and carbonate.
Or, if you don't want to change your ring you can try buckytools to rebalance your cluster. You can try original one or go-graphite fork (which is more maintained).

[gerases]

Are you saying the carbon-c-relay config is invalid?

Can i hack carbonate to use '' instead of None? Or have it as command-line option?

[deniszh]

Are you saying the carbon-c-relay config is invalid?

Well. Whole idea behind fnv1a_ch iirc is using instances instead hostnames/ports for ring manipulation, which will make ring more resilient for e.g. hostname change. Having empty instances are kinda useless for that purpose.

Can i hack carbonate to use '' instead of None? Or have it as command-line option?

Sure, feel free to do so. Ofc I'll accept PR for command line option to fix this, just make this optional and do not change default behavior.

[gerases]

Ok, I think carbon-c-relay has special syntax to guard against hostname changes? The = syntax?

Anyway, I think it makes sense to make that change. But I thought this behavior is not under carbonate's control since you're using the hashing library from python-carbon?

[deniszh]

Ok, I think carbon-c-relay has special syntax to guard against hostname changes? The = syntax?

Yes, you'll need to use : as instance separator for carbon, and = - for carbon-c-relay.
e.g.

DESTINATIONS = 192.168.240.186:2003:a, 192.168.240.184:2003:b
HASHING_TYPE = fnv1a_ch

should looks like thos in carbon-c-relay

cluster graphite
    fnv1a_ch
        192.168.240.186:2003=a proto tcp
        192.168.240.184:2003=b proto tcp
    ;

Anyway, I think it makes sense to make that change. But I thought this behavior is not under carbonate's control since you're using the hashing library from python-carbon?

Exactly. So, I'm afraid, cli option will be hard. You can try ta make quick hack in carbon hashing, just for your usecase, though.

[gerases]

Cool. think I will add the instances and rebalance.

Unrelated issue: when running carbon-sync to sync data from an old cluster, I occasionally get an error like this:

WARNING:root:Failed to copy SOME_PATH/used.wsp! [Errno 2] No such file or directory: 'TMP_DIR/graphite_sync/SOME_PATH/used.wsp

... where SOME_PATH is the actual path to the metric on disk and TMP is where rsync is looking for the downloaded whisper file. Have you seen that message before?

[gerases]

I found the reason: I'm overloading the sshd on the target system with my parallel rsyncs :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.