Enable automatic leader election by default for new installs

[drewmiranda-gl]

What?

Builds off of the work from #11373 .

Enable automatic leader election by default for new installs

leader_election_mode=automatic

Currently in https://github.com/Graylog2/graylog2-server/blob/master/misc/graylog.conf we have is_leader = true.

It may also be useful to allow nodes participating in leader_election_mode=automatic to not be eligible (although it is possible that omitting leader_election_mode and setting is_leader = false accomplishes this.

Why?

Currently, and in line with current documentation, the leader role is explicitly configured for graylog clusters. This creates a single point of failure for graylog clusters. Allowing automatic leader election will simplify the graylog cluster creation process and can allow clusters to, by default, have better resiliency if the master node is offline or has a failure.

[thll]

Currently, leader_election_mode=automatic only works correctly when also run_migrations=false is configured. Migrations then need to be executed manually by running the graylog migrate command. This could be documented but is not very convenient and we should strive to find a better solution for this before considering to make automatic leader election the default.

Also see #16234

[drewmiranda-gl]

Understand. More was just recently shared in slack about this so me opening this may have been premature :(

[thll]

Understand. More was just recently shared in slack about this so me opening this may have been premature :(

To the contrary. I think it's good to get the conversation going!