Handle ip changes

[ivan-kripakov-m10]

Pull request is associated with issue #1442

I made several things here:

1. add server_lookup and pass it to raft transport layer to provide possibility for ip changing in cluster (id -> ip mapping is updated on serf events)
2. add serf member update handling
3. resolve todo Query each of the servers and make sure they report no Raft peers. (I'm not sure if it is correct approach)
4. Avoid self removing from raft cluster in leader loop because it leads to raft shutdown (on both add [don't remove if server id matches, just update configuration] and remove peer [skip if server id matches with current])

I built image and tested it in k8s using this helm - distribworks/dkron-helm#7
Seems like everything works fine now

[vcastellm]

Thanks your this contribution, really appreciated, can you add a simple test file to cover server_lookup?

Also, can you switch base branch to https://github.com/distribworks/dkron/tree/4.x branch instead? we'll focus on that at this point.

[ivan-kripakov-m10]

yes, sure
done

[fieldju-harness]

@vcastellm how is this PR looking?

I'm trying to get a sense of when this might get merged and also a rough idea of when 4.x is coming.

[vcastellm]

I'm trying to get a sense of when this might get merged and also a rough idea of when 4.x is coming.

First v4 beta in the following hours

[vcastellm]

LGTM great job!