Add tags to replicas to support many replicas

[jberkus]

Replicas should have the following attributes available. This would be added as tags within the replica description, both in the API and in the SIS. If the tags are absent, default behavior remains the same; in the config, the tags below are set to True or False:

nofailover: this replica should not be a candidate for automatic failover. Used for ephemeral replicas, and multi-replica setups where there are a few designated failover targets. If the tag is missing, the replica is assumed to be a failover target.

noloadbalance: this replica should not be part of any load-balancing, if HAProxy autoconfig is being used.

clonefrom: take basebackups from this replica instead of from the master. If multiple CloneFrom replicals are available, one is randomly chosen for new basebackups. If cloning using WALE or other archival backup, this tag is ignored.

replicatefrom: not true or false, but the IP address/hostname of another replica. Used to support cascading replication.

[kmoppel]

Also an "environment" tag should be there to distinguishing between release/live environments etc.
Even better would be the possibility to add "custom" user defined tags, with no meaning to patroni. Those tags should be persisted and easily inspectable via (future) patroni-cli for example.

[valgog]

@kmoppel you are now introducing the idea of tags for the whole cluster. Until now the tags suggested by Josh are tagging instances in the cluster.

[jberkus]

The only global tag I'm proposing to add will be:

disabled: if this tag is set for the cluster, no auto-failover will take place. This tag is meant for disabling auto-failover during manual failover and similar expected downtime. It should always be set with a TTL.

[jberkus]

I don't think I agree with the idea of an "environment" tag. In what way is that not simply part of the "scope"?

As for user-defined custom tags, nothing keeps patroni-cli from adding any tags it wants. I haven't conceived of any I need yet though.

[kmoppel]

@valgog yes, exactly for the whole cluster. @jberkus ok yes, the "environment" concept could be an user defined concept (tag) also - keeping mandatory parts from operational requirements only might be a good line to keep as patroni grows in features.
But custom tags in general would be very useful in my opinion - marking environments, applications using some cluster, criticality level, being part of a sharded cluster etc, will definitely help large setups.

[jberkus]

@kmoppel Certianly you could easily add a self.custom_tags dictionary to init.py, and then an optional config setting called "custom_tags" which takes a list to the patroni config file. However, what good would this do you? Pretty much the only thing you could do with the custom tags is pass them along to create_replica scripts; there's no place else in patroni we would read them. So what's the point?

If you want custom tags just for the CLI, that's really a completely different feature.

[kmoppel]

Exactly, just for the CLI (or any reader), no point for the slaves or init scripts or any other operational stuff. Idea would be to just persist some values in etcd/zookeeper, so that for example our "database discovery service" could give users more hints about the nature and purpose of found database clusters. These tags could be changed/added at any moment, not on cluster creation, so maybe it deserves a separate issue indeed.

[jberkus]

@kmoppel I welcome your contributions to patroni-cli, once I have the initial prototype up.

[alexeyklyukin]

The nofailover tag, along with some general code to support tags in the DCS and the API, is implemented in #80

[jberkus]

Great! Will test ...

[feikesteenbergen]

It seems we still need to implement:

noloadbalance

Are there any negative (or positive) test results?

[CyberDem0n]

noloadbalance is implemented in #185