Skip to content
This repository has been archived by the owner on Jul 7, 2020. It is now read-only.

Deleting PV in kubernetes is not deleting volume on gluster w/ heketi #257

Closed
debianmaster opened this issue Apr 23, 2017 · 10 comments
Closed

Deleting PV in kubernetes is not deleting volume on gluster w/ heketi #257

debianmaster opened this issue Apr 23, 2017 · 10 comments

Comments

@debianmaster
Copy link

i have a setup a storage class backed by gluster w/ heketi.
I have provisioned few volumes with this dynamic storage volume but when i delete PV its not deleting underlying volume automatically
@humblec
Copy link
Contributor

humblec commented Apr 23, 2017

@debianmaster can you provide me more details and logs ? also which version of kube is in use here ?

@debianmaster
Copy link
Author

debianmaster commented Apr 24, 2017 •
edited

OpenShift Master:
v1.5.0+031cbe4
Kubernetes Master:
v1.5.2+43a9be4

My disks were 50GB previously and i got no storage error, (which seems right) , but now i increased it to 100GB. and restarted heketi pod, i'm still getting following error.

Failed to provision volume with StorageClass "default": glusterfs: create volume err: error creating volume Error calling v.allocBricksInCluster: database is in read-only mode.

@debianmaster
Copy link
Author

i have restarted gluster DaemonSets (by killing) and also heketi still see that database ready only mode error

@jarrpa
Copy link
Contributor

jarrpa commented Apr 24, 2017

@debianmaster Can you restart the heketi pod and paste the logs from the point where it starts coming back up?

@debianmaster
Copy link
Author 

Heketi v4.0.0-57-g27e7ea4
[heketi] INFO 2017/04/24 11:32:18 Loaded kubernetes executor
[heketi] WARNING 2017/04/24 11:32:18 Unable to open database.  Retrying using read only mode
[heketi] INFO 2017/04/24 11:32:18 Loaded simple allocator
[heketi] INFO 2017/04/24 11:32:18 GlusterFS Application Loaded
Authorization loaded
Listening on port 8080
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 735.351µs
[asynchttp] INFO 2017/04/24 11:32:24 Started job 5ac838d25ba64ecfe7e08315674713c4
[heketi] INFO 2017/04/24 11:32:24 Creating volume 21dc2b01f95b207ccd224ab09a07954a
[heketi] INFO 2017/04/24 11:32:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:24 Completed job 5ac838d25ba64ecfe7e08315674713c4 in 2.078337ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 240.783µs
[asynchttp] INFO 2017/04/24 11:32:24 Started job 33692bcea29f867d165ccdeef20f6fbd
[heketi] INFO 2017/04/24 11:32:24 Creating volume 918a11c044201679511ead21dbfb182f
[heketi] INFO 2017/04/24 11:32:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:24 Completed job 33692bcea29f867d165ccdeef20f6fbd in 698.725µs
[negroni] Started GET /queue/5ac838d25ba64ecfe7e08315674713c4
[negroni] Completed 500 Internal Server Error in 88.756µs
[negroni] Started GET /queue/33692bcea29f867d165ccdeef20f6fbd
[negroni] Completed 500 Internal Server Error in 43.042µs
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 169.386µs
[asynchttp] INFO 2017/04/24 11:32:39 Started job cb43ed7de7bc3bcb9cd34365d7038a41
[heketi] INFO 2017/04/24 11:32:39 Creating volume 86a96c805ef47e35e7cf514ee49e73fd
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:32:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:39 Completed job cb43ed7de7bc3bcb9cd34365d7038a41 in 1.01767ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 104.023µs
[asynchttp] INFO 2017/04/24 11:32:39 Started job 84b1a9d3d5eaa0c7f5e3195ea6a272bd
[heketi] INFO 2017/04/24 11:32:39 Creating volume 9ba7cd1ae936acfa8e211faf9c7a833e
[heketi] INFO 2017/04/24 11:32:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:39 Completed job 84b1a9d3d5eaa0c7f5e3195ea6a272bd in 607.641µs
[negroni] Started GET /queue/cb43ed7de7bc3bcb9cd34365d7038a41
[negroni] Completed 500 Internal Server Error in 75.599µs
[negroni] Started GET /queue/84b1a9d3d5eaa0c7f5e3195ea6a272bd
[negroni] Completed 500 Internal Server Error in 67.341µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 181.19µs
[asynchttp] INFO 2017/04/24 11:32:54 Started job f42076002f8dde52b68463b5885b35ba
[heketi] INFO 2017/04/24 11:32:54 Creating volume 54f38956790fce0b5981f9defcfcfdbc
[heketi] INFO 2017/04/24 11:32:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:54 Completed job f42076002f8dde52b68463b5885b35ba in 4.385998ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 163.506µs
[asynchttp] INFO 2017/04/24 11:32:54 Started job c9658d6f9e9a5fb6fa954f39b794b5b5
[heketi] INFO 2017/04/24 11:32:54 Creating volume 69b3a566f6b013af7aa749d43b466f68
[heketi] INFO 2017/04/24 11:32:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:54 Completed job c9658d6f9e9a5fb6fa954f39b794b5b5 in 610.964µs
[negroni] Started GET /queue/f42076002f8dde52b68463b5885b35ba
[negroni] Completed 500 Internal Server Error in 83.412µs
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started GET /queue/c9658d6f9e9a5fb6fa954f39b794b5b5
[negroni] Completed 500 Internal Server Error in 64.039µs
[negroni] Started POST /volumes
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Completed 202 Accepted in 549.541µs
[asynchttp] INFO 2017/04/24 11:33:09 Started job 8b6b253f80de8afe06d741ab88d322e8
[heketi] INFO 2017/04/24 11:33:09 Creating volume c18795c0a62d0378d204fa9b0a676d23
[heketi] INFO 2017/04/24 11:33:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:09 Completed job 8b6b253f80de8afe06d741ab88d322e8 in 737.056µs
[negroni] Started GET /queue/8b6b253f80de8afe06d741ab88d322e8
[negroni] Completed 500 Internal Server Error in 92.453µs
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 184.729µs
[asynchttp] INFO 2017/04/24 11:33:09 Started job fd5bf24afa83f977f448208b1a5d8507
[heketi] INFO 2017/04/24 11:33:09 Creating volume 58ebd326b87ff38f535ddf3a92dca362
[heketi] INFO 2017/04/24 11:33:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:09 Completed job fd5bf24afa83f977f448208b1a5d8507 in 727.503µs
[negroni] Started GET /queue/fd5bf24afa83f977f448208b1a5d8507
[negroni] Completed 500 Internal Server Error in 86.055µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 176.067µs
[asynchttp] INFO 2017/04/24 11:33:24 Started job 5b73e827dc21fac22ec342633bb4165c
[heketi] INFO 2017/04/24 11:33:24 Creating volume ce1d02813672a7aa433492012bf71ebb
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:33:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:24 Completed job 5b73e827dc21fac22ec342633bb4165c in 1.003952ms
[negroni] Started GET /queue/5b73e827dc21fac22ec342633bb4165c
[negroni] Completed 500 Internal Server Error in 80.524µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 98.002µs
[asynchttp] INFO 2017/04/24 11:33:24 Started job 7639172d29537fc11390f723c82e2641
[heketi] INFO 2017/04/24 11:33:24 Creating volume 06bd6964d8a99856b240dd90fc544ea1
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:33:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:24 Completed job 7639172d29537fc11390f723c82e2641 in 717.965µs
[negroni] Started GET /queue/7639172d29537fc11390f723c82e2641
[negroni] Completed 500 Internal Server Error in 59.848µs
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 245.159µs
[asynchttp] INFO 2017/04/24 11:33:39 Started job 798c2ae3ba8b6480d43b6c11022b446f
[heketi] INFO 2017/04/24 11:33:39 Creating volume 5fb706cfdf55c5ef0bfd6c464d42455d
[heketi] INFO 2017/04/24 11:33:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:39 Completed job 798c2ae3ba8b6480d43b6c11022b446f in 844.851µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 147.371µs
[asynchttp] INFO 2017/04/24 11:33:39 Started job 44247a00d6b0ed7a93d59e1d26cf2abf
[heketi] INFO 2017/04/24 11:33:39 Creating volume c495a3009cb8761b0575d859b4a374bf
[heketi] INFO 2017/04/24 11:33:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:39 Completed job 44247a00d6b0ed7a93d59e1d26cf2abf in 632.587µs
[negroni] Started GET /queue/798c2ae3ba8b6480d43b6c11022b446f
[negroni] Completed 500 Internal Server Error in 85.886µs
[negroni] Started GET /queue/44247a00d6b0ed7a93d59e1d26cf2abf
[negroni] Completed 500 Internal Server Error in 75.35µs
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 219.09µs
[asynchttp] INFO 2017/04/24 11:33:54 Started job ab96a60c47a7592fa8c8693c78ddba38
[heketi] INFO 2017/04/24 11:33:54 Creating volume 6785440ed64a6ee617d360f377ce37a7
[heketi] INFO 2017/04/24 11:33:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:54 Completed job ab96a60c47a7592fa8c8693c78ddba38 in 884.413µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 130.356µs
[asynchttp] INFO 2017/04/24 11:33:54 Started job 2e5e94b16e36ce3588910f0eea551acc
[heketi] INFO 2017/04/24 11:33:54 Creating volume 03e1ad5f5c02d99730608e7540dcfb26
[heketi] INFO 2017/04/24 11:33:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:54 Completed job 2e5e94b16e36ce3588910f0eea551acc in 705.094µs
[negroni] Started GET /queue/2e5e94b16e36ce3588910f0eea551acc
[negroni] Completed 500 Internal Server Error in 78.332µs
[negroni] Started GET /queue/ab96a60c47a7592fa8c8693c78ddba38
[negroni] Completed 500 Internal Server Error in 45.787µs
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 178.808µs
[asynchttp] INFO 2017/04/24 11:34:09 Started job f2bf7860f572b0bc796e88333886a687
[heketi] INFO 2017/04/24 11:34:09 Creating volume eb2db6279f0e02236954550e827315c4
[heketi] INFO 2017/04/24 11:34:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:09 Completed job f2bf7860f572b0bc796e88333886a687 in 924.276µs
[negroni] Started GET /queue/f2bf7860f572b0bc796e88333886a687
[negroni] Completed 500 Internal Server Error in 75.65µs
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 133.054µs
[asynchttp] INFO 2017/04/24 11:34:09 Started job d48f5fb0ffd46da615f3bc28bc4754b3
[heketi] INFO 2017/04/24 11:34:09 Creating volume 74978a5cfc4aa369b46839b70d88c142
[heketi] INFO 2017/04/24 11:34:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:09 Completed job d48f5fb0ffd46da615f3bc28bc4754b3 in 914.321µs
[negroni] Started GET /queue/d48f5fb0ffd46da615f3bc28bc4754b3
[negroni] Completed 500 Internal Server Error in 65.367µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 509.384µs
[asynchttp] INFO 2017/04/24 11:34:24 Started job 9d1a4333db1f9f67b0dac8d1566c1f4d
[heketi] INFO 2017/04/24 11:34:24 Creating volume 6d4c6f5a89cc92b2d40957ae6b76ac25
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:34:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:24 Completed job 9d1a4333db1f9f67b0dac8d1566c1f4d in 1.169951ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 101.554µs
[asynchttp] INFO 2017/04/24 11:34:24 Started job 231ce63a4a9d985640d8848514cb3694
[heketi] INFO 2017/04/24 11:34:24 Creating volume 943e5fb8ff4fab389afbff6e4dcaeeae
[heketi] INFO 2017/04/24 11:34:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:24 Completed job 231ce63a4a9d985640d8848514cb3694 in 654.238µs
[negroni] Started GET /queue/9d1a4333db1f9f67b0dac8d1566c1f4d
[negroni] Completed 500 Internal Server Error in 75.882µs
[negroni] Started GET /queue/231ce63a4a9d985640d8848514cb3694
[negroni] Completed 500 Internal Server Error in 77.351µs
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 176.093µs
[asynchttp] INFO 2017/04/24 11:34:39 Started job adbd582658367c9656f485e25698a4ed
[heketi] INFO 2017/04/24 11:34:39 Creating volume 57c7e3392c205a6eb5d8e58c52f5d119
[heketi] INFO 2017/04/24 11:34:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:39 Completed job adbd582658367c9656f485e25698a4ed in 843.757µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 116.165µs
[asynchttp] INFO 2017/04/24 11:34:39 Started job 4121196d91a554fb533092afa1def5d7
[heketi] INFO 2017/04/24 11:34:39 Creating volume 477d6f0a1023ec1a2c46fbd90a17401e
[heketi] INFO 2017/04/24 11:34:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:39 Completed job 4121196d91a554fb533092afa1def5d7 in 635.208µs
[negroni] Started GET /queue/adbd582658367c9656f485e25698a4ed
[negroni] Completed 500 Internal Server Error in 94.771µs
[negroni] Started GET /queue/4121196d91a554fb533092afa1def5d7
[negroni] Completed 500 Internal Server Error in 41.669µs
[negroni] Started POST /volumes
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Completed 202 Accepted in 67.095634ms
[asynchttp] INFO 2017/04/24 11:34:54 Started job f9c1150695b709cccf49e53de579f211
[heketi] INFO 2017/04/24 11:34:54 Creating volume d7e7177c45b2bf90c9e6ab40a076eca5
[heketi] INFO 2017/04/24 11:34:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:54 Completed job f9c1150695b709cccf49e53de579f211 in 836.735µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 202.707µs
[asynchttp] INFO 2017/04/24 11:34:54 Started job f692c9a476109fee6fc9436ccd696e85
[heketi] INFO 2017/04/24 11:34:54 Creating volume 23c04ae3d36580b92fb2e9bf7da097fd
[heketi] INFO 2017/04/24 11:34:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:54 Completed job f692c9a476109fee6fc9436ccd696e85 in 650.293µs
[negroni] Started GET /queue/f9c1150695b709cccf49e53de579f211
[negroni] Completed 500 Internal Server Error in 88.178µs
[negroni] Started GET /queue/f692c9a476109fee6fc9436ccd696e85
[negroni] Completed 500 Internal Server Error in 40.837µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 613.505µs
[asynchttp] INFO 2017/04/24 11:35:09 Started job 21c7c7e96a5bfc62e054be7c7e934689
[heketi] INFO 2017/04/24 11:35:09 Creating volume 481456f74c49510ad7bbd0e32eb2a563
[heketi] INFO 2017/04/24 11:35:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:09 Completed job 21c7c7e96a5bfc62e054be7c7e934689 in 1.216958ms
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 173.166µs
[asynchttp] INFO 2017/04/24 11:35:09 Started job 4be651549ddbc8cc657f47638c5ee3e6
[heketi] INFO 2017/04/24 11:35:09 Creating volume e72e2e1856d4989f7c65de6e1a332721
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:09 Completed job 4be651549ddbc8cc657f47638c5ee3e6 in 767.981µs
[negroni] Started GET /queue/21c7c7e96a5bfc62e054be7c7e934689
[negroni] Completed 500 Internal Server Error in 78.358µs
[negroni] Started GET /queue/4be651549ddbc8cc657f47638c5ee3e6
[negroni] Completed 500 Internal Server Error in 42.179µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 168.881µs
[asynchttp] INFO 2017/04/24 11:35:24 Started job 1a3e19fbc3a1ba32c1e290bb1b712374
[heketi] INFO 2017/04/24 11:35:24 Creating volume c0f1803e5359e9430533986abd1afcd8
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:24 Completed job 1a3e19fbc3a1ba32c1e290bb1b712374 in 1.109376ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 759.801µs
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:24 Started job 01fcdd0fb72c0943f059638e95c96126
[heketi] INFO 2017/04/24 11:35:24 Creating volume 0061d2b32cf18d12e7cc882c156e5568
[heketi] INFO 2017/04/24 11:35:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:24 Completed job 01fcdd0fb72c0943f059638e95c96126 in 865.098µs
[negroni] Started GET /queue/01fcdd0fb72c0943f059638e95c96126
[negroni] Completed 500 Internal Server Error in 99.422µs
[negroni] Started GET /queue/1a3e19fbc3a1ba32c1e290bb1b712374
[negroni] Completed 500 Internal Server Error in 74.521µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 352.344µs
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:39 Started job 139d045eff0289ded3251988b25eb0f9
[heketi] INFO 2017/04/24 11:35:39 Creating volume 4639880c01b30f616ad9eca2aeabc66c
[heketi] INFO 2017/04/24 11:35:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:39 Completed job 139d045eff0289ded3251988b25eb0f9 in 819.578µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 774.621µs
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:39 Started job 5401cad00308a3bfc8f1b301ede47631
[heketi] INFO 2017/04/24 11:35:39 Creating volume 0e8fb6bc6245f52a63f1b4bd4f922979
[heketi] INFO 2017/04/24 11:35:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:39 Completed job 5401cad00308a3bfc8f1b301ede47631 in 708.601µs
[negroni] Started GET /queue/139d045eff0289ded3251988b25eb0f9
[negroni] Completed 500 Internal Server Error in 106.851µs
[negroni] Started GET /queue/5401cad00308a3bfc8f1b301ede47631
[negroni] Completed 500 Internal Server Error in 43.412µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 172.159µs
[asynchttp] INFO 2017/04/24 11:35:54 Started job 4736e63d1795d994aae1d2a5f2c4303c
[heketi] INFO 2017/04/24 11:35:54 Creating volume 89425a5eb21019ae0675c225e313434f
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:54 Completed job 4736e63d1795d994aae1d2a5f2c4303c in 1.003371ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 101.368µs
[asynchttp] INFO 2017/04/24 11:35:54 Started job 4930e85fd29def20576c3c46a77b4a6b
[heketi] INFO 2017/04/24 11:35:54 Creating volume cc3c9b565807c8f4e7c656bae2982cc4
[heketi] INFO 2017/04/24 11:35:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:54 Completed job 4930e85fd29def20576c3c46a77b4a6b in 716.502µs
[negroni] Started GET /queue/4930e85fd29def20576c3c46a77b4a6b
[negroni] Completed 500 Internal Server Error in 80.269µs
[negroni] Started GET /queue/4736e63d1795d994aae1d2a5f2c4303c
[negroni] Completed 500 Internal Server Error in 63.702µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 515.754µs
[asynchttp] INFO 2017/04/24 11:36:09 Started job 3d8820343f5ff2fa8b5b60e29f4a416b
[heketi] INFO 2017/04/24 11:36:09 Creating volume 99edcd2ee905ee9130a5009fb96cfe70
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:09 Completed job 3d8820343f5ff2fa8b5b60e29f4a416b in 1.380268ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 181.663µs
[asynchttp] INFO 2017/04/24 11:36:09 Started job 95219ae40cd191c3a17ea271afb82b11
[heketi] INFO 2017/04/24 11:36:09 Creating volume 668106e46d644a828e6434acddbb0f55
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:09 Completed job 95219ae40cd191c3a17ea271afb82b11 in 885.904µs
[negroni] Started GET /queue/3d8820343f5ff2fa8b5b60e29f4a416b
[negroni] Completed 500 Internal Server Error in 78.19µs
[negroni] Started GET /queue/95219ae40cd191c3a17ea271afb82b11
[negroni] Completed 500 Internal Server Error in 70.674µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 188.918µs
[asynchttp] INFO 2017/04/24 11:36:24 Started job 8161accbd0ebbe904fa04a5d5c6d92dc
[heketi] INFO 2017/04/24 11:36:24 Creating volume 2045bf9eef11d28aa32632fd7446521c
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:24 Completed job 8161accbd0ebbe904fa04a5d5c6d92dc in 1.099057ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 647.185µs
[asynchttp] INFO 2017/04/24 11:36:24 Started job 1ee2370cd54f9fcd1dbc3f00650d6c7e
[heketi] INFO 2017/04/24 11:36:24 Creating volume 4e076ec13cc5052bd210bae74d5d2311
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:24 Completed job 1ee2370cd54f9fcd1dbc3f00650d6c7e in 936.396µs
[negroni] Started GET /queue/8161accbd0ebbe904fa04a5d5c6d92dc
[negroni] Completed 500 Internal Server Error in 102.638µs
[negroni] Started GET /queue/1ee2370cd54f9fcd1dbc3f00650d6c7e
[negroni] Completed 500 Internal Server Error in 322.754µs
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 169.805µs
[asynchttp] INFO 2017/04/24 11:36:39 Started job 93d88b7e03555b2999b152efcb01f667
[heketi] INFO 2017/04/24 11:36:39 Creating volume a771bdc21e7b66dd578c4383c0fa7780
[heketi] INFO 2017/04/24 11:36:39 brick_num: 0
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:36:39 Completed job 93d88b7e03555b2999b152efcb01f667 in 781.398µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 159.636µs
[asynchttp] INFO 2017/04/24 11:36:39 Started job 4854e7d420f42e5cb38550c92b61b586
[heketi] INFO 2017/04/24 11:36:39 Creating volume ddd53f2269b530be9c6105d0c8f35c64
[heketi] INFO 2017/04/24 11:36:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:39 Completed job 4854e7d420f42e5cb38550c92b61b586 in 625.261µs
[negroni] Started GET /queue/93d88b7e03555b2999b152efcb01f667
[negroni] Completed 500 Internal Server Error in 77.727µs
[negroni] Started GET /queue/4854e7d420f42e5cb38550c92b61b586
[negroni] Completed 500 Internal Server Error in 71.383µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 529.85µs
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:36:54 Started job 3cde3b0cc9a8a4f2d241345176a8ea28
[heketi] INFO 2017/04/24 11:36:54 Creating volume 5983d8c2e019b20946e7212ac29d4224
[heketi] INFO 2017/04/24 11:36:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:54 Completed job 3cde3b0cc9a8a4f2d241345176a8ea28 in 765.055µs
[negroni] Started GET /queue/3cde3b0cc9a8a4f2d241345176a8ea28
[negroni] Completed 500 Internal Server Error in 87.44µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 127.96µs
[asynchttp] INFO 2017/04/24 11:36:54 Started job 3a6080a63ec05714473a25466d14e48b
[heketi] INFO 2017/04/24 11:36:54 Creating volume b9b5ead78eaa876d23ee2d9676563770
[heketi] INFO 2017/04/24 11:36:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:54 Completed job 3a6080a63ec05714473a25466d14e48b in 632.469µs
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started GET /queue/3a6080a63ec05714473a25466d14e48b
[negroni] Completed 500 Internal Server Error in 71.521µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 275.707µs
[asynchttp] INFO 2017/04/24 11:37:09 Started job f1d79f1503fb8c186469cb3ccb1aecb4
[heketi] INFO 2017/04/24 11:37:09 Creating volume 401642815509420268fc1d7d698a39ee
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:09 Completed job f1d79f1503fb8c186469cb3ccb1aecb4 in 1.100964ms
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 125.011µs
[asynchttp] INFO 2017/04/24 11:37:09 Started job 9bbeac8e22895b9aac49127f9864e51a
[heketi] INFO 2017/04/24 11:37:09 Creating volume 872f9d8c85328568a9683d859a7bd6b6
[heketi] INFO 2017/04/24 11:37:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:09 Completed job 9bbeac8e22895b9aac49127f9864e51a in 1.03045ms
[negroni] Started GET /queue/f1d79f1503fb8c186469cb3ccb1aecb4
[negroni] Completed 500 Internal Server Error in 115.502µs
[negroni] Started GET /queue/9bbeac8e22895b9aac49127f9864e51a
[negroni] Completed 500 Internal Server Error in 1.92336ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 251.474µs
[asynchttp] INFO 2017/04/24 11:37:24 Started job a88de7ab81a2807eb1be8ec25c9d598d
[heketi] INFO 2017/04/24 11:37:24 Creating volume cfe19ba109cf871b2e3814c144a80e7b
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:24 Completed job a88de7ab81a2807eb1be8ec25c9d598d in 896.621µs
[negroni] Started GET /queue/a88de7ab81a2807eb1be8ec25c9d598d
[negroni] Completed 500 Internal Server Error in 82.529µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 130.861µs
[asynchttp] INFO 2017/04/24 11:37:24 Started job 1160009e6699c103ae2f74f6953b1f59
[heketi] INFO 2017/04/24 11:37:24 Creating volume 697cd65b8f94b0ec1bf0153920d849af
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:24 Completed job 1160009e6699c103ae2f74f6953b1f59 in 750.409µs
[negroni] Started GET /queue/1160009e6699c103ae2f74f6953b1f59
[negroni] Completed 500 Internal Server Error in 69.751µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 192.712µs
[asynchttp] INFO 2017/04/24 11:37:39 Started job c641cac7b2f257b668ed26836aa09efe
[heketi] INFO 2017/04/24 11:37:39 Creating volume b40185487ab5b1719816dadfd41d6dd3
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:39 Completed job c641cac7b2f257b668ed26836aa09efe in 1.138811ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 224.507µs
[asynchttp] INFO 2017/04/24 11:37:39 Started job e7e6472b5376453372da25d2d8c533b6
[heketi] INFO 2017/04/24 11:37:39 Creating volume 36f33d0bcd0ba8335db19780534230ef
[heketi] INFO 2017/04/24 11:37:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:39 Completed job e7e6472b5376453372da25d2d8c533b6 in 662.948µs
[negroni] Started GET /queue/c641cac7b2f257b668ed26836aa09efe
[negroni] Completed 500 Internal Server Error in 117.395µs
[negroni] Started GET /queue/e7e6472b5376453372da25d2d8c533b6
[negroni] Completed 500 Internal Server Error in 51.534µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 423.765µs
[asynchttp] INFO 2017/04/24 11:37:54 Started job ae9450ecfe2387fa8d4e0205b3ce648d
[heketi] INFO 2017/04/24 11:37:54 Creating volume 9e7c9b03acfe05d31e1ab3f0b687da63
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode

i did restart couple of times

@debianmaster
Copy link
Author

any pointers where i can look?

@debianmaster
Copy link
Author

❓

@saidimu
Copy link

saidimu commented Jun 1, 2017

@debianmaster This comment might help you: heketi/heketi#677 (comment)

Restarting the heketi pod didn't help in our case but scaling down the heketi deployment, waiting for a few minutes, then scaling it up again fixed the underlying issue with BoltdDB file locks heketi/heketi#731 (comment)

kubectl scale --replicas=0 deploy/heketi

... wait a few minutes for the write file locks to be released, then

kubectl scale --replicas=1 deploy/heketi

@debianmaster
Copy link
Author

@saidimu thanks for your help, i will give a try. closing for now.

@john-deng john-deng mentioned this issue Jul 11, 2017
Closed
@davidkarlsen
Copy link

heketi/heketi#1215 looks related

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@davidkarlsen @saidimu @jarrpa @debianmaster @humblec