database is in readonly mode

[barnzdan]

% heketi-cli --version
heketi-cli v3.1.0-2-g900f849-release-3

Im seeing the following error during volume create in heketi. I cant find any other issues related and do not see a function to change the mode:

% heketi-cli volume create \

--size=10
--persistent-volume
--persistent-volume-endpoint=grafana-storage-endpoints > grafana-pv.json
Error: Error calling v.allocBricksInCluster: database is in read-only mode

==Logs from the heketi pod==

[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 257.173µs
[asynchttp] INFO 2017/02/14 16:08:26 Started job a3beb648ffc81b11f230e2fb3d5ce03a
[heketi] INFO 2017/02/14 16:08:26 Creating volume 6cad2dc48f6569ce1d114c75a96b4132
[heketi] INFO 2017/02/14 16:08:26 brick_num: 0
[asynchttp] INFO 2017/02/14 16:08:26 Completed job a3beb648ffc81b11f230e2fb3d5ce03a in 5.641725ms
[heketi] ERROR 2017/02/14 16:08:26 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:57: database is in read-only mode
[heketi] ERROR 2017/02/14 16:08:26 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/02/14 16:08:26 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:154: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started GET /queue/a3beb648ffc81b11f230e2fb3d5ce03a
[negroni] Completed 500 Internal Server Error in 51.943µs

[lpabon]

Is the database on a GlusterFS volume that has gone into read only mode? See: https://github.com/heketi/heketi/blob/master/apps/glusterfs/app.go#L98 . It could also be that there are two Heketi processes running and one has the lock to the db and the other (the one you communicated with) does not.

[barnzdan]

Great suggestion! While I didnt find evidence of 2 heketi processes running, I deleted the heketi pod, after rebuild I can create a volume again. Thanks!!

[lpabon]

Awesome! 👍