No Duplicate Polling

[dishmael]

Not sure if this is a feature request or worthy of an RFC, but I think it is worthy of consideration/discussion.

Problem and Proposed Solution
Often, we have a need to gather metrics from a remote system from a centralized collector or, ideally, cluster (tribe) of snap collectors. In the latter architecture, I would want to define a single task to collect one or more metrics from a remote node; however, I do not want all the collectors to gather the same metric. Instead, I would want only one of the collectors in the cluster to gather that metric. This would be accomplished using a decentralized queue model leveraging a consensus algorithm (e.g. Raft - https://raft.github.io).

Use Case
At a configurable rate, the collectors would vote on which collector would be used to gather the metric(s) defined in a task shared amongst members in a tribe. Busy collectors would be naturally slower to respond and so faster, under/less utilized collectors would be selected for gathering those metrics.

[andrzej-k]

@dishmael Thanks for submitting this feature proposal. It seems to resemble #773 a bit.
Would you be interested in preparing design/RFC for this feature or would you rather just use it when available?

[dishmael]

I would be happy to contribute any way I can. I have never written an RFC before so would not know where to start.

[andrzej-k]

Your problem statement looks like a good RFC, you can also check how we wrote RFCs in the past, for example #1522
What I would be more interested in is some description of design and implementation alternatives so we can gather opinions on how to implement this finally. Thanks for your involvement so far.

[dishmael]

Sounds good! I'm traveling right now so will knock this out as soon as I get back.

[dishmael]

Should probably change the title of this Issue to "No Duplicate Polling" so it is not confused with the RFC I created

[mbbroberg]

I changed the title for you @dishmael 👍