Applier fiber — ER_READONLY: Can't modify data because this instance is in read-only mode.

[filonenko-mikhail]

After some admin actions on new cluster, I cant reboot the whole cluster.

Seems that somehow it's possible to make several masters, while one of them still not bootstraped.

The results

repro.tar.gz

tar xzf repro.tar.gz

cd repro 
tarantoolctl rocks install cartridge 2.3.0
cartridge start --name makeshort

# storage-02 cant start er_readonly

[rosik]

It seems you're missing stateboard.

If you restart a cluster without a stateboard, not a single instance could fetch a leader information and everyone will be read-only.
If you try to join an instance, it'll fail.

[rosik]

But there's another real problem: the same error arise when a leader isn't first in failover priority list.

Steps to reproduce:

1. Start 3 unconfigured instances and a stateboard
2. Compose a replicaset of two instances (3301 and 3302) with 3301 in top of failover priority list
3. Enable stateful failover
4. Promote the second instance (3302)
5. Try to join the third instance (3303) to the same replicaset

Result:

I> bootstrapping replica from 4c4c9e4a-bdf9-4250-bb59-200155272620 at 127.0.0.1:3301
E> ER_READONLY: Can't modify data because this instance is in read-only mode. 

[rosik]

Here is a problem:

cartridge/cartridge/confapplier.lua

Line 314 in fb4b3a3

local function boot_instance(clusterwide_config)

local leader_uuid = leaders_order[1] -- !!!!! with stateful failover it's not always a leader
local leader = topology_cfg.servers[leader_uuid]
box_opts.replication = {pool.format_uri(leader.uri)}
box.cfg(box_opts)