Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Got "Permission denied" in successful attached volume #1713

Closed
manhhailua opened this issue Aug 24, 2020 · 8 comments
Closed

[Question] Got "Permission denied" in successful attached volume #1713

manhhailua opened this issue Aug 24, 2020 · 8 comments
Labels
kind/question Please use `discussion` to ask questions instead

Comments

@manhhailua
Copy link

manhhailua commented Aug 24, 2020

I got error "permission denied" in all pods which is using longhorn volumes on a k3s cluster with 1 master and 2 nodes (all those threes are ubuntu 18.04 on AWS and having Security Groups rules follow k3s network config requirements).

Redis pod log:

redis 03:48:12.03 
redis 03:48:12.03 Welcome to the Bitnami redis container
redis 03:48:12.03 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
redis 03:48:12.04 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
redis 03:48:12.04 
redis 03:48:12.04 INFO  ==> ** Starting Redis setup **
redis 03:48:12.05 INFO  ==> Initializing Redis
redis 03:48:12.07 INFO  ==> Setting Redis config file
redis 03:48:12.09 INFO  ==> ** Redis setup finished! **

redis 03:48:12.12 INFO  ==> ** Starting Redis **
1:C 24 Aug 2020 03:48:12.143 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 24 Aug 2020 03:48:12.143 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 24 Aug 2020 03:48:12.143 # Configuration loaded
1:M 24 Aug 2020 03:48:12.146 # Can't open the append-only file: Permission denied

Postgresql pod log:

postgresql 03:48:06.01 
postgresql 03:48:06.01 Welcome to the Bitnami postgresql container
postgresql 03:48:06.01 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 03:48:06.01 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 03:48:06.02 
postgresql 03:48:06.03 INFO  ==> ** Starting PostgreSQL setup **
postgresql 03:48:06.05 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 03:48:06.06 INFO  ==> Loading custom pre-init scripts...
postgresql 03:48:06.07 INFO  ==> Initializing PostgreSQL database...
mkdir: cannot create directory ‘/bitnami/postgresql/data’: Permission denied
postgresql 03:48:06.08 INFO  ==> Stopping PostgreSQL...

I installed the k3s server with:

curl -sfL https://get.k3s.io | sh -s - --disable traefik --disable local-storage

And installed longhorn by helm:

sudo helm install longhorn --set persistence.defaultClassReplicaCount=2 ./longhorn/chart/ --namespace longhorn-system

I did try using securityContext but no luck:

securityContext:
  runAsUser: 1001
  runAsGroup: 0
@manhhailua manhhailua added the kind/question Please use `discussion` to ask questions instead label Aug 24, 2020
@joshimoo
Copy link
Contributor

  • Are you running SELinux?
  • Are you able to create a single shell pod, with a longhorn volume mounted?
  • Are you able to access the mount in that pod? (ls, touch)
  • Anything in the longhorn-manager logs? kubetail longhorn-manager -n longhorn-system

@manhhailua
Copy link
Author

manhhailua commented Aug 24, 2020

@joshimoo I updated this reply after successfully access the mount /data in a single nginx:stable-alpine pod. (followed this https://longhorn.io/docs/1.0.2/volumes-and-nodes/create-volumes/)
- Are you running SELinux?
It is disabled.
- Are you able to create a single shell pod, with a longhorn volume mounted?
Yes, after added one more node (now have 1 master and 3 nodes) and reinstalled longhorn with all default values.
- Are you able to access the mount in that pod? (ls, touch)
Yes, ls and touch work both fine.
- Anything in the longhorn-manager logs? kubetail longhorn-manager -n longhorn-system
Log by: sudo kubectl -n longhorn-system logs -f -l app=longhorn-manager after helm install my main chart.

time="2020-08-24T17:04:37Z" level=info msg="Longhorn volume longhorn-system/pvc-d9f522f0-1233-44ea-b46d-0f15e39f5318 has been deleted"
time="2020-08-24T17:04:38Z" level=info msg="Longhorn volume longhorn-system/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e has been deleted"
10.42.0.1 - - [24/Aug/2020:17:04:38 +0000] "GET /v1/volumes/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:04:38 +0000] "GET /v1/volumes/pvc-d9f522f0-1233-44ea-b46d-0f15e39f5318 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn engine longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-bb711852 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-b74cda20 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-9460d403 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn volume longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d has been deleted"
10.42.0.1 - - [24/Aug/2020:17:05:04 +0000] "GET /v1/volumes/pvc-14b14d17-525e-4d44-ab93-223342d7679d HTTP/1.1" 404 0 "" "Go-http-client/1.1"
time="2020-08-24T17:04:37Z" level=info msg="Longhorn replica longhorn-system/pvc-d9f522f0-1233-44ea-b46d-0f15e39f5318-r-7628bba6 has been deleted"
time="2020-08-24T17:04:37Z" level=info msg="Longhorn replica longhorn-system/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e-r-f3c1b15f has been deleted"
time="2020-08-24T17:04:37Z" level=info msg="Longhorn volume longhorn-system/pvc-d9f522f0-1233-44ea-b46d-0f15e39f5318 has been deleted"
time="2020-08-24T17:04:38Z" level=info msg="Longhorn volume longhorn-system/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e has been deleted"
time="2020-08-24T17:04:56Z" level=debug msg="Cleanup replica pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-bb711852 at ip-172-31-23-91:/var/lib/longhorn/replicas/pvc-14b14d17-525e-4d44-ab93-223342d7679d-117259a9 completed"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn engine longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-bb711852 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-b74cda20 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-9460d403 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn volume longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d has been deleted"
time="2020-08-24T17:04:38Z" level=info msg="Longhorn volume longhorn-system/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-14b14d17-525e-4d44-ab93-223342d7679d\", UID:\"d193a404-88a3-4be7-9560-34ae1c46a5e9\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206091\", FieldPath:\"\"}): type: 'Normal' reason: 'Delete' Deleting volume pvc-14b14d17-525e-4d44-ab93-223342d7679d"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn engine longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-bb711852 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-b74cda20 has been deleted"
time="2020-08-24T17:04:56Z" level=debug msg="Cleanup replica pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-9460d403 at ip-172-31-2-6:/var/lib/longhorn/replicas/pvc-14b14d17-525e-4d44-ab93-223342d7679d-a5da4d8a completed"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-9460d403 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn volume longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d has been deleted"
time="2020-08-24T17:04:57Z" level=info msg="stop engine pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7 monitoring because the engine no longer exists"
time="2020-08-24T17:04:57Z" level=debug msg="Stop monitoring pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7"
time="2020-08-24T17:04:38Z" level=info msg="Longhorn volume longhorn-system/pvc-f41692d4-1c59-4b65-b0b9-b98b00e0ba6e has been deleted"
10.42.2.1 - - [24/Aug/2020:17:04:56 +0000] "GET /v1/volumes/pvc-14b14d17-525e-4d44-ab93-223342d7679d HTTP/1.1" 200 6271 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:04:56 +0000] "DELETE /v1/volumes/pvc-14b14d17-525e-4d44-ab93-223342d7679d HTTP/1.1" 200 0 "" "Go-http-client/1.1"
time="2020-08-24T17:04:56Z" level=debug msg="Deleted volume pvc-14b14d17-525e-4d44-ab93-223342d7679d"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn engine longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-e-af0461f7 has been deleted"
time="2020-08-24T17:04:56Z" level=debug msg="Cleanup replica pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-b74cda20 at ip-172-31-6-234:/var/lib/longhorn/replicas/pvc-14b14d17-525e-4d44-ab93-223342d7679d-c081ddbb completed"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-bb711852 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-b74cda20 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn replica longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d-r-9460d403 has been deleted"
time="2020-08-24T17:04:56Z" level=info msg="Longhorn volume longhorn-system/pvc-14b14d17-525e-4d44-ab93-223342d7679d has been deleted"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 404 0 "" "Go-http-client/1.1"
time="2020-08-24T17:05:47Z" level=debug msg="Volume controller ip-172-31-23-91 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020"
time="2020-08-24T17:05:47Z" level=debug msg="Volume controller ip-172-31-2-6 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4"
time="2020-08-24T17:05:48Z" level=debug msg="Volume controller ip-172-31-23-91 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8"
time="2020-08-24T17:05:48Z" level=debug msg="Volume controller ip-172-31-23-91 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d"
time="2020-08-24T17:05:48Z" level=debug msg="Volume controller ip-172-31-23-91 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb"
time="2020-08-24T17:05:48Z" level=debug msg="Volume controller ip-172-31-2-6 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e"
time="2020-08-24T17:05:48Z" level=debug msg="Engine controller ip-172-31-23-91 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6"
time="2020-08-24T17:05:48Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 state, old state , new state stopped"
time="2020-08-24T17:05:48Z" level=debug msg="Engine controller ip-172-31-23-91 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0"
time="2020-08-24T17:05:48Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 state, old state , new state stopped"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4: {Size:10737418240 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 973 "" "Go-http-client/1.1"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb: {Size:2147483648 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020: {Size:5368709120 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8: {Size:2147483648 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d: {Size:2147483648 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e: {Size:2147483648 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 972 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 972 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:48 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 972 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 972 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:47 +0000] "POST /v1/volumes HTTP/1.1" 200 972 "" "Go-http-client/1.1"
time="2020-08-24T17:05:48Z" level=debug msg="Engine controller ip-172-31-11-66 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:05:48Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 state, old state , new state stopped"
time="2020-08-24T17:05:48Z" level=debug msg="Engine controller ip-172-31-11-66 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf"
time="2020-08-24T17:05:48Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf state, old state , new state stopped"
time="2020-08-24T17:05:48Z" level=debug msg="Engine controller ip-172-31-11-66 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0"
time="2020-08-24T17:05:48Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 state, old state , new state stopped"
time="2020-08-24T17:05:48Z" level=debug msg="Created volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6: {Size:53687091200 Frontend:blockdev FromBackup: NumberOfReplicas:3 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
10.42.2.1 - - [24/Aug/2020:17:05:48 +0000] "POST /v1/volumes HTTP/1.1" 200 973 "" "Go-http-client/1.1"
time="2020-08-24T17:05:48Z" level=debug msg="Volume controller ip-172-31-11-66 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6"
time="2020-08-24T17:05:49Z" level=debug msg="Engine controller ip-172-31-11-66 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438"
time="2020-08-24T17:05:49Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 state, old state , new state stopped"
time="2020-08-24T17:05:49Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f"
time="2020-08-24T17:05:49Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f state, old state , new state stopped"
time="2020-08-24T17:05:49Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f"
time="2020-08-24T17:05:49Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f state, old state , new state stopped"
time="2020-08-24T17:05:50Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307"
time="2020-08-24T17:05:50Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 state, old state , new state stopped"
time="2020-08-24T17:05:50Z" level=debug msg="Engine controller ip-172-31-2-6 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195"
time="2020-08-24T17:05:50Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 state, old state , new state stopped"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 973 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 1316 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 1316 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 1316 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 1316 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 1316 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:50 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 1563 "" "Go-http-client/1.1"
time="2020-08-24T17:05:50Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b"
time="2020-08-24T17:05:50Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-760af78d"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-8e79550a"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-c0745bf8"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692"
time="2020-08-24T17:05:51Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 state, old state , new state stopped"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-d259c3f5"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-90752d14"
time="2020-08-24T17:05:51Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-cc199358"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 2057 "" "Go-http-client/1.1"
time="2020-08-24T17:05:52Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24"
time="2020-08-24T17:05:52Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 state, old state , new state stopped"
time="2020-08-24T17:05:52Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4"
time="2020-08-24T17:05:52Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 state, old state , new state stopped"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 1562 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 1562 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 1810 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 1809 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:52 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 1809 "" "Go-http-client/1.1"
time="2020-08-24T17:05:52Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a"
time="2020-08-24T17:05:52Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a state, old state , new state stopped"
time="2020-08-24T17:05:52Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-b9fd047c"
time="2020-08-24T17:05:52Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-04692f96"
time="2020-08-24T17:05:52Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-ea2d1c87"
time="2020-08-24T17:05:53Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c"
time="2020-08-24T17:05:53Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c state, old state , new state stopped"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-64fddd1b"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-f53f3383"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-00917f4f"
time="2020-08-24T17:05:53Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59"
time="2020-08-24T17:05:53Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 state, old state , new state stopped"
time="2020-08-24T17:05:53Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e"
time="2020-08-24T17:05:53Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e state, old state , new state stopped"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-5291cac0"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-7ff685b8"
time="2020-08-24T17:05:53Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-03c6dd34"
time="2020-08-24T17:05:53Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671"
time="2020-08-24T17:05:53Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 state, old state , new state stopped"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-b5f8cc6b"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-858005df"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-ea70e920"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-6952c441"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-104824fb"
time="2020-08-24T17:05:54Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-b3de60e4"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 1809 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 2057 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:54 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 2057 "" "Go-http-client/1.1"
time="2020-08-24T17:05:54Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb"
time="2020-08-24T17:05:54Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb state, old state , new state stopped"
time="2020-08-24T17:05:54Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139"
time="2020-08-24T17:05:54Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 state, old state , new state stopped"
time="2020-08-24T17:05:55Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d"
time="2020-08-24T17:05:55Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d state, old state , new state stopped"
time="2020-08-24T17:05:55Z" level=debug msg="Requeue longhorn-system/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-0013ef9f"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e93a24d3"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-b476e0a2"
time="2020-08-24T17:05:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4\", UID:\"077b73cb-0d79-41ac-8399-933afe7e3709\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206238\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4 has been detached"
time="2020-08-24T17:05:55Z" level=debug msg="Replica controller ip-172-31-6-234 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68"
time="2020-08-24T17:05:55Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 state, old state , new state stopped"
time="2020-08-24T17:05:55Z" level=debug msg="Requeue longhorn-system/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-8ca03e70"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-3e117327"
time="2020-08-24T17:05:55Z" level=debug msg="Schedule replica pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-5ee597b3"
time="2020-08-24T17:05:55Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6\", UID:\"4dd48821-45cd-4f0e-bacf-f385ac89d106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206322\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 has been detached"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
time="2020-08-24T17:05:56Z" level=debug msg="Requeue longhorn-system/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:56Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-f77bed8f"
time="2020-08-24T17:05:56Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-149a0487"
time="2020-08-24T17:05:56Z" level=debug msg="Schedule replica pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-bf792e04"
time="2020-08-24T17:05:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e\", UID:\"a981f0f1-ca27-46e0-a51a-fa9883932459\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206253\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e has been detached"
10.42.2.1 - - [24/Aug/2020:17:05:56 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 2297 "" "Go-http-client/1.1"
time="2020-08-24T17:05:57Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671"
time="2020-08-24T17:05:57Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 2415 "" "Go-http-client/1.1"
time="2020-08-24T17:05:58Z" level=debug msg="Requeue longhorn-system/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-9c541ed4"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-c817a03b"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-3ad96bd4"
time="2020-08-24T17:05:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020\", UID:\"6424c51f-ea6f-49bd-982c-b8d3a0754674\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206236\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 has been detached"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 2413 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
time="2020-08-24T17:05:58Z" level=debug msg="Replica controller ip-172-31-23-91 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3761 "" "Go-http-client/1.1"
time="2020-08-24T17:05:58Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f"
time="2020-08-24T17:05:58Z" level=debug msg="Requeue longhorn-system/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-784fe531"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-6c098a70"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-a91f1b2b"
time="2020-08-24T17:05:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8\", UID:\"7837ecb1-f0b1-4f0a-84e0-a85eb73be659\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206250\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 has been detached"
time="2020-08-24T17:05:58Z" level=debug msg="Requeue longhorn-system/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-76048223"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 to node ip-172-31-2-6, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-cbe75e44"
time="2020-08-24T17:05:58Z" level=debug msg="Schedule replica pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-db158afb"
time="2020-08-24T17:05:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d\", UID:\"2f065d53-a34e-4223-a3b2-2ea0fd0ab441\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206251\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d has been detached"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
time="2020-08-24T17:05:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020\", UID:\"6424c51f-ea6f-49bd-982c-b8d3a0754674\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206236\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 started to use/reuse Longhorn volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020"
10.42.2.1 - - [24/Aug/2020:17:05:58 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 3761 "" "Go-http-client/1.1"
time="2020-08-24T17:05:59Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f"
time="2020-08-24T17:05:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4\", UID:\"077b73cb-0d79-41ac-8399-933afe7e3709\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206972\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4 started to use/reuse Longhorn volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4"
10.42.2.1 - - [24/Aug/2020:17:05:59 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:59 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
time="2020-08-24T17:05:59Z" level=debug msg="Requeue longhorn-system/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:05:59Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e to node ip-172-31-23-91, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-4a54a0b4"
time="2020-08-24T17:05:59Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e to node ip-172-31-11-66, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-b5cf35e8"
time="2020-08-24T17:05:59Z" level=debug msg="Schedule replica pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 to node ip-172-31-6-234, disk default-disk-60bdabb5fa61ea16, datapath /var/lib/longhorn/replicas/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-03bd1718"
time="2020-08-24T17:05:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb\", UID:\"3d2183f1-d156-4ee6-a144-e474e7d6e0b7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206252\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb has been detached"
10.42.2.1 - - [24/Aug/2020:17:05:59 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:59 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3760 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:05:59 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3761 "" "Go-http-client/1.1"
time="2020-08-24T17:05:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8\", UID:\"7837ecb1-f0b1-4f0a-84e0-a85eb73be659\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206250\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 started to use/reuse Longhorn volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8"
time="2020-08-24T17:05:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb\", UID:\"3d2183f1-d156-4ee6-a144-e474e7d6e0b7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206252\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb started to use/reuse Longhorn volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb"
time="2020-08-24T17:05:59Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 3979 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2056 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2176 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2223 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2176 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:00 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3760 "" "Go-http-client/1.1"
time="2020-08-24T17:06:00Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d\", UID:\"2f065d53-a34e-4223-a3b2-2ea0fd0ab441\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206251\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d started to use/reuse Longhorn volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d"
time="2020-08-24T17:06:00Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e\", UID:\"a981f0f1-ca27-46e0-a51a-fa9883932459\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206989\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e started to use/reuse Longhorn volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e"
time="2020-08-24T17:06:00Z" level=debug msg="Replica controller ip-172-31-23-91 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307"
time="2020-08-24T17:06:00Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6\", UID:\"4dd48821-45cd-4f0e-bacf-f385ac89d106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"206939\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 started to use/reuse Longhorn volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6"
time="2020-08-24T17:06:00Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24"
time="2020-08-24T17:06:00Z" level=debug msg="Replica controller ip-172-31-23-91 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71"
time="2020-08-24T17:06:01Z" level=debug msg="Replica controller ip-172-31-2-6 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1"
10.42.2.1 - - [24/Aug/2020:17:06:02 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2223 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:02 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2294 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:02 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2498 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:02 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2343 "" "Go-http-client/1.1"
time="2020-08-24T17:06:02Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb"
time="2020-08-24T17:06:03Z" level=debug msg="Replica controller ip-172-31-23-91 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3957 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/nodes/ip-172-31-23-91 HTTP/1.1" 200 1785 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 3979 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/nodes/ip-172-31-2-6 HTTP/1.1" 200 1903 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2637 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 2618 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 2649 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:04 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 2650 "" "Go-http-client/1.1"
time="2020-08-24T17:06:04Z" level=debug msg="Requeue longhorn-system/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 due to error Operation cannot be fulfilled on volumes.longhorn.io \"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020\": the object has been modified; please apply your changes to the latest version and try again or <nil>"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020\", UID:\"6424c51f-ea6f-49bd-982c-b8d3a0754674\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207084\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 has been detached"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f\", UID:\"e0459827-6eb1-41ce-97f3-ff1337a7bb0e\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207154\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f"
time="2020-08-24T17:06:04Z" level=debug msg="Attaching volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e to ip-172-31-23-91 with disableFrontend set false"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "POST /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e?action=attach HTTP/1.1" 200 2515 "" "Go-http-client/1.1"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67\", UID:\"419658fe-dba5-4111-8396-8615128bbcf2\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207159\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67"
time="2020-08-24T17:06:04Z" level=debug msg="Attaching volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4 to ip-172-31-2-6 with disableFrontend set false"
10.42.0.1 - - [24/Aug/2020:17:06:04 +0000] "POST /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4?action=attach HTTP/1.1" 200 2537 "" "Go-http-client/1.1"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a\", UID:\"aa5d8e15-546f-4991-b7ba-9f32c582db8d\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207164\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692\", UID:\"3e57b6cf-02d9-42d8-a7bc-ba63e8b4a161\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207163\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b\", UID:\"161f1ba6-01f6-4555-9b10-f24907d873fb\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207169\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b"
time="2020-08-24T17:06:04Z" level=debug msg="Requeue longhorn-system/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 due to error Operation cannot be fulfilled on volumes.longhorn.io \"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8\": the object has been modified; please apply your changes to the latest version and try again or <nil>"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8\", UID:\"7837ecb1-f0b1-4f0a-84e0-a85eb73be659\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207108\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 has been detached"
time="2020-08-24T17:06:04Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f starts running, IP 10.42.1.35"
time="2020-08-24T17:06:04Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f starts running, Port 10000"
time="2020-08-24T17:06:04Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-efd5b07f state, old state stopped, new state running"
time="2020-08-24T17:06:04Z" level=debug msg="Prepare to create instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f\", UID:\"468987fd-edbc-4736-ac2f-ee756ee2907e\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207176\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f"
time="2020-08-24T17:06:04Z" level=debug msg="Replica controller ip-172-31-11-66 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e"
time="2020-08-24T17:06:04Z" level=debug msg="Requeue longhorn-system/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d due to error Operation cannot be fulfilled on volumes.longhorn.io \"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d\": the object has been modified; please apply your changes to the latest version and try again or <nil>"
time="2020-08-24T17:06:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d\", UID:\"2f065d53-a34e-4223-a3b2-2ea0fd0ab441\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207131\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d has been detached"
time="2020-08-24T17:06:05Z" level=debug msg="Requeue longhorn-system/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb due to error Operation cannot be fulfilled on volumes.longhorn.io \"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb\": the object has been modified; please apply your changes to the latest version and try again or <nil>"
time="2020-08-24T17:06:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb\", UID:\"3d2183f1-d156-4ee6-a144-e474e7d6e0b7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207120\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb has been detached"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 starts running, IP 10.42.0.28"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 starts running, Port 10000"
time="2020-08-24T17:06:05Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-afef2e67 state, old state stopped, new state running"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a starts running, IP 10.42.2.40"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a starts running, Port 10000"
time="2020-08-24T17:06:05Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-r-6dadbf0a state, old state stopped, new state running"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 starts running, IP 10.42.2.40"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 starts running, Port 10015"
time="2020-08-24T17:06:05Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-236d4692 state, old state stopped, new state running"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f starts running, IP 10.42.1.35"
time="2020-08-24T17:06:05Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f starts running, Port 10015"
time="2020-08-24T17:06:05Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-6cfd811f state, old state stopped, new state running"
time="2020-08-24T17:06:05Z" level=debug msg="Prepare to create instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf"
time="2020-08-24T17:06:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf\", UID:\"8e86de4a-e2ba-4ebf-b569-f49b432a7945\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207197\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf"
10.42.2.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 2637 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 3995 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 3964 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 3996 "" "Go-http-client/1.1"
time="2020-08-24T17:06:06Z" level=debug msg="Instance process pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf had been created, need to wait for instance manager update"
10.42.0.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3378 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:06 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 3316 "" "Go-http-client/1.1"
time="2020-08-24T17:06:06Z" level=debug msg="Volume controller ip-172-31-23-91 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e"
time="2020-08-24T17:06:06Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b starts running, IP 10.42.3.8"
time="2020-08-24T17:06:06Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b starts running, Port 10000"
time="2020-08-24T17:06:06Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-r-12ff243b state, old state stopped, new state running"
time="2020-08-24T17:06:07Z" level=debug msg="Instance process pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf had been created, need to wait for instance manager update"
time="2020-08-24T17:06:07Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf state, old state stopped, new state starting"
time="2020-08-24T17:06:07Z" level=debug msg="Prepare to create instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0"
time="2020-08-24T17:06:07Z" level=debug msg="Engine controller ip-172-31-2-6 picked up pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0"
time="2020-08-24T17:06:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0\", UID:\"c40e3fdc-c68a-4045-809d-80560a6e5b64\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207228\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0"
time="2020-08-24T17:06:07Z" level=debug msg="Instance process pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:07Z" level=debug msg="Instance process pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:07Z" level=debug msg="Instance process pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:07Z" level=debug msg="Instance process pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:07Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 state, old state stopped, new state starting"
10.42.2.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3983 "" "Go-http-client/1.1"
time="2020-08-24T17:06:08Z" level=debug msg="Requeue longhorn-system/pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 due to conflict: Operation cannot be fulfilled on engines.longhorn.io \"pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0\": the object has been modified; please apply your changes to the latest version and try again"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 3964 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/nodes/ip-172-31-6-234 HTTP/1.1" 200 1982 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3989 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/nodes/ip-172-31-11-66 HTTP/1.1" 200 1980 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 3996 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/nodes/ip-172-31-23-91 HTTP/1.1" 200 1850 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3981 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/nodes/ip-172-31-11-66 HTTP/1.1" 200 1980 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 3995 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/nodes/ip-172-31-6-234 HTTP/1.1" 200 1982 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3405 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "POST /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020?action=attach HTTP/1.1" 200 2522 "" "Go-http-client/1.1"
time="2020-08-24T17:06:08Z" level=debug msg="Attaching volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 to ip-172-31-6-234 with disableFrontend set false"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "POST /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6?action=attach HTTP/1.1" 200 2547 "" "Go-http-client/1.1"
time="2020-08-24T17:06:08Z" level=debug msg="Attaching volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 to ip-172-31-11-66 with disableFrontend set false"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 3424 "" "Go-http-client/1.1"
time="2020-08-24T17:06:08Z" level=debug msg="Volume controller ip-172-31-6-234 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020"
time="2020-08-24T17:06:08Z" level=debug msg="Prepare to create instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24"
time="2020-08-24T17:06:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24\", UID:\"a51f6b96-251f-49d4-b166-7e6f7fdd7728\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207279\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24"
time="2020-08-24T17:06:08Z" level=debug msg="Engine controller ip-172-31-23-91 picked up pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 starts running, IP 10.42.1.35"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 starts running, Port 10030"
time="2020-08-24T17:06:09Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c68ad24 state, old state stopped, new state running"
time="2020-08-24T17:06:08Z" level=debug msg="Prepare to create instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb"
time="2020-08-24T17:06:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb\", UID:\"8b5bd3de-7ae9-4d56-9900-85be02bb2cc0\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207286\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf starts running, IP 10.42.3.7"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf starts running, Port 10000"
time="2020-08-24T17:06:09Z" level=debug msg="Instance handler updated instance pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf state, old state starting, new state running"
time="2020-08-24T17:06:09Z" level=debug msg="Start monitoring pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e-e-421b0fdf"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307\", UID:\"5545f2c8-69cc-412e-8b61-f288d63d96b8\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207293\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "POST /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8?action=attach HTTP/1.1" 200 2550 "" "Go-http-client/1.1"
time="2020-08-24T17:06:09Z" level=debug msg="Attaching volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 to ip-172-31-6-234 with disableFrontend set false"
time="2020-08-24T17:06:09Z" level=debug msg="Attaching volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb to ip-172-31-11-66 with disableFrontend set false"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "POST /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb?action=attach HTTP/1.1" 200 2539 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:08 +0000] "POST /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d?action=attach HTTP/1.1" 200 2554 "" "Go-http-client/1.1"
time="2020-08-24T17:06:09Z" level=debug msg="Attaching volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d to ip-172-31-23-91 with disableFrontend set false"
time="2020-08-24T17:06:09Z" level=debug msg="Volume controller ip-172-31-6-234 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139\", UID:\"f72c0214-1d2b-4371-9294-591f49caec2b\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207303\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e\", UID:\"a981f0f1-ca27-46e0-a51a-fa9883932459\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207212\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e has been attached to ip-172-31-23-91"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4\", UID:\"8d4aac2a-1dab-4c80-8e3f-ab2f8ea5aa34\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207307\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71\", UID:\"7a5ffd87-7abc-4245-8c96-86addba901d1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207316\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c\", UID:\"399ab6c9-091d-4ae4-8ac6-86df77af28d2\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207319\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e\", UID:\"db807197-3dc1-4de3-be3d-16b5331c8d37\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207323\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb starts running, IP 10.42.0.28"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb starts running, Port 10015"
time="2020-08-24T17:06:09Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-cb0a4fcb state, old state stopped, new state running"
time="2020-08-24T17:06:09Z" level=debug msg="Instance process pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e had been created, need to wait for instance manager update"
time="2020-08-24T17:06:09Z" level=debug msg="Instance process pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:09Z" level=debug msg="Instance process pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c had been created, need to wait for instance manager update"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 starts running, IP 10.42.1.34"
time="2020-08-24T17:06:09Z" level=debug msg="Instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 starts running, Port 10000"
time="2020-08-24T17:06:09Z" level=debug msg="Instance handler updated instance pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0 state, old state starting, new state running"
time="2020-08-24T17:06:09Z" level=debug msg="Start monitoring pvc-52286726-ac7e-49df-ba29-4be2a89192b4-e-c3e793f0"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-52286726-ac7e-49df-ba29-4be2a89192b4\", UID:\"077b73cb-0d79-41ac-8399-933afe7e3709\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207183\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-52286726-ac7e-49df-ba29-4be2a89192b4 has been attached to ip-172-31-2-6"
time="2020-08-24T17:06:09Z" level=debug msg="Volume controller ip-172-31-11-66 picked up pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb"
time="2020-08-24T17:06:09Z" level=debug msg="Prepare to create instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d"
time="2020-08-24T17:06:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d\", UID:\"f9805992-1402-4c98-8d5f-a86469f46c42\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207332\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d"
time="2020-08-24T17:06:10Z" level=debug msg="Requeue longhorn-system/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:06:10Z" level=debug msg="Prepare to create instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671"
time="2020-08-24T17:06:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671\", UID:\"9fdc4f06-39c6-4374-b937-2a9df8a72f08\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207338\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671"
10.42.0.1 - - [24/Aug/2020:17:06:10 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 3446 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:10 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 3232 "" "Go-http-client/1.1"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 starts running, IP 10.42.2.40"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 starts running, Port 10030"
time="2020-08-24T17:06:10Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139 state, old state stopped, new state running"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d starts running, IP 10.42.2.40"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d starts running, Port 10045"
time="2020-08-24T17:06:10Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-d2cbb81d state, old state stopped, new state running"
10.42.0.1 - - [24/Aug/2020:17:06:10 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 5133 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:10 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3989 "" "Go-http-client/1.1"
time="2020-08-24T17:06:10Z" level=debug msg="Prepare to create instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68"
time="2020-08-24T17:06:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68\", UID:\"599108aa-d431-4640-b3f6-62f6032f3967\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207352\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 starts running, IP 10.42.3.8"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 starts running, Port 10015"
time="2020-08-24T17:06:10Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-r-7c661307 state, old state stopped, new state running"
time="2020-08-24T17:06:10Z" level=debug msg="Instance process pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 starts running, IP 10.42.1.35"
time="2020-08-24T17:06:10Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 starts running, Port 10045"
time="2020-08-24T17:06:10Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-b81a0671 state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 starts running, IP 10.42.0.28"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 starts running, Port 10030"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-80ac91d4 state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c starts running, IP 10.42.0.28"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c starts running, Port 10045"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-eedff39c state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e starts running, IP 10.42.0.28"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e starts running, Port 10060"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-d0cff55e state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Prepare to create instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468"
time="2020-08-24T17:06:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468\", UID:\"140e958a-ab86-4338-8c23-a7b0d82c2d61\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207361\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468"
time="2020-08-24T17:06:11Z" level=debug msg="Prepare to create instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59"
time="2020-08-24T17:06:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59\", UID:\"bb86eb21-9fcd-45ff-a63c-a3b3336a9814\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207362\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59"
10.42.0.1 - - [24/Aug/2020:17:06:11 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 3263 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:11 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3983 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:11 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 4065 "" "Go-http-client/1.1"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 starts running, IP 10.42.2.40"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 starts running, Port 10075"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-r-8436e468 state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 starts running, IP 10.42.2.40"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 starts running, Port 10060"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-e6fe6d68 state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Prepare to create instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e"
time="2020-08-24T17:06:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e\", UID:\"c18abbfa-40a7-43e6-9a1b-219d1bc8e573\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207368\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e"
10.42.0.1 - - [24/Aug/2020:17:06:11 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 5133 "" "Go-http-client/1.1"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 starts running, IP 10.42.3.8"
time="2020-08-24T17:06:11Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 starts running, Port 10030"
time="2020-08-24T17:06:11Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-6a29ea71 state, old state stopped, new state running"
time="2020-08-24T17:06:11Z" level=debug msg="Instance process pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e had been created, need to wait for instance manager update"
time="2020-08-24T17:06:12Z" level=debug msg="Prepare to create instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6"
time="2020-08-24T17:06:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6\", UID:\"f7979b04-b0c2-42a0-ba9e-83ecbb76720e\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207380\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6"
time="2020-08-24T17:06:12Z" level=debug msg="Engine controller ip-172-31-6-234 picked up pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6"
time="2020-08-24T17:06:12Z" level=debug msg="Instance process pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:12Z" level=debug msg="Requeue longhorn-system/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:06:12Z" level=debug msg="Instance process pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 had been created, need to wait for instance manager update"
10.42.0.1 - - [24/Aug/2020:17:06:12 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 5113 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:12 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 3384 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:12 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 5133 "" "Go-http-client/1.1"
time="2020-08-24T17:06:12Z" level=debug msg="Instance process pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:12Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 state, old state stopped, new state starting"
10.42.0.1 - - [24/Aug/2020:17:06:12 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3395 "" "Go-http-client/1.1"
time="2020-08-24T17:06:12Z" level=debug msg="Prepare to create instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1"
time="2020-08-24T17:06:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1\", UID:\"a437d623-a1e3-4117-9112-2f53060f5325\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207393\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1"
time="2020-08-24T17:06:12Z" level=debug msg="Requeue longhorn-system/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 due to error <nil> or Operation cannot be fulfilled on replicas.longhorn.io \"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-r-7ea51139\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:06:12Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 starts running, IP 10.42.1.35"
time="2020-08-24T17:06:12Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 starts running, Port 10060"
time="2020-08-24T17:06:12Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-b7c9dfe1 state, old state stopped, new state running"
time="2020-08-24T17:06:13Z" level=debug msg="Instance process pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:13Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 state, old state stopped, new state starting"
time="2020-08-24T17:06:13Z" level=debug msg="Engine controller ip-172-31-6-234 picked up pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:06:13Z" level=debug msg="Prepare to create instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:06:13Z" level=debug msg="Prepare to create instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:06:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9\", UID:\"8211c991-1677-4631-8ecd-2c3ff376e5e7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207404\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:06:13Z" level=debug msg="Instance process pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:13Z" level=debug msg="Prepare to create instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195"
time="2020-08-24T17:06:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195\", UID:\"ece12c4e-4649-4d4e-923f-552a5b7d2463\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207409\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195"
time="2020-08-24T17:06:13Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e starts running, IP 10.42.3.8"
time="2020-08-24T17:06:13Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e starts running, Port 10045"
time="2020-08-24T17:06:13Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-r-4a204f7e state, old state stopped, new state running"
10.42.0.1 - - [24/Aug/2020:17:06:13 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 3415 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:13 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3320 "" "Go-http-client/1.1"
time="2020-08-24T17:06:13Z" level=debug msg="Instance process pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 had been created, need to wait for instance manager update"
10.42.0.1 - - [24/Aug/2020:17:06:13 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 4134 "" "Go-http-client/1.1"
time="2020-08-24T17:06:13Z" level=debug msg="Instance process pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:13Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 state, old state stopped, new state starting"
time="2020-08-24T17:06:13Z" level=debug msg="Requeue longhorn-system/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 due to conflict: Operation cannot be fulfilled on engines.longhorn.io \"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6\": the object has been modified; please apply your changes to the latest version and try again"
10.42.0.1 - - [24/Aug/2020:17:06:13 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 5133 "" "Go-http-client/1.1"
time="2020-08-24T17:06:13Z" level=debug msg="Requeue longhorn-system/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 due to conflict: Operation cannot be fulfilled on engines.longhorn.io \"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:06:14Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 starts running, IP 10.42.0.28"
time="2020-08-24T17:06:14Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 starts running, Port 10075"
time="2020-08-24T17:06:14Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-r-8d0c0a59 state, old state starting, new state running"
10.42.0.1 - - [24/Aug/2020:17:06:14 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 3410 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:14 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3410 "" "Go-http-client/1.1"
time="2020-08-24T17:06:14Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 starts running, IP 10.42.2.39"
time="2020-08-24T17:06:14Z" level=debug msg="Instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 starts running, Port 10000"
time="2020-08-24T17:06:14Z" level=debug msg="Instance handler updated instance pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6 state, old state starting, new state running"
time="2020-08-24T17:06:14Z" level=debug msg="Start monitoring pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020-e-7f7b96b6"
time="2020-08-24T17:06:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020\", UID:\"6424c51f-ea6f-49bd-982c-b8d3a0754674\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207302\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 has been attached to ip-172-31-6-234"
time="2020-08-24T17:06:14Z" level=debug msg="Prepare to create instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438"
time="2020-08-24T17:06:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438\", UID:\"e554675f-5e85-4a68-b978-a13fd4aec433\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207434\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438"
time="2020-08-24T17:06:15Z" level=debug msg="Instance process pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:15Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 state, old state stopped, new state starting"
time="2020-08-24T17:06:15Z" level=debug msg="Prepare to create instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0"
time="2020-08-24T17:06:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0\", UID:\"4a3d26e8-6cb8-4d9f-b34b-002debe00680\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207448\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0"
10.42.0.1 - - [24/Aug/2020:17:06:15 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 3442 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:15 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3402 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:15 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 3417 "" "Go-http-client/1.1"
time="2020-08-24T17:06:15Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 starts running, IP 10.42.2.39"
time="2020-08-24T17:06:15Z" level=debug msg="Instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 starts running, Port 10001"
time="2020-08-24T17:06:15Z" level=debug msg="Instance handler updated instance pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9 state, old state starting, new state running"
time="2020-08-24T17:06:15Z" level=debug msg="Start monitoring pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8-e-8e9d07c9"
time="2020-08-24T17:06:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8\", UID:\"7837ecb1-f0b1-4f0a-84e0-a85eb73be659\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207327\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 has been attached to ip-172-31-6-234"
10.42.0.1 - - [24/Aug/2020:17:06:15 +0000] "GET /v1/volumes/pvc-52286726-ac7e-49df-ba29-4be2a89192b4 HTTP/1.1" 200 6228 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:15 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
time="2020-08-24T17:06:15Z" level=debug msg="Instance process pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:15Z" level=debug msg="Instance process pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:15Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 state, old state stopped, new state starting"
time="2020-08-24T17:06:16Z" level=debug msg="Engine controller ip-172-31-23-91 picked up pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0"
time="2020-08-24T17:06:16Z" level=debug msg="Instance process pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:16Z" level=debug msg="Requeue longhorn-system/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 due to conflict: Operation cannot be fulfilled on engines.longhorn.io \"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-24T17:06:16Z" level=debug msg="Engine controller ip-172-31-11-66 picked up pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195"
10.42.0.1 - - [24/Aug/2020:17:06:16 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 5119 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:16 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 3437 "" "Go-http-client/1.1"
time="2020-08-24T17:06:16Z" level=debug msg="Instance process pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 had been created, need to wait for instance manager update"
time="2020-08-24T17:06:16Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 state, old state stopped, new state starting"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 starts running, IP 10.42.0.27"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 starts running, Port 10000"
time="2020-08-24T17:06:17Z" level=debug msg="Instance handler updated instance pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195 state, old state starting, new state running"
time="2020-08-24T17:06:17Z" level=debug msg="Start monitoring pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6-e-bd74f195"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 starts running, IP 10.42.0.27"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 starts running, Port 10001"
time="2020-08-24T17:06:17Z" level=debug msg="Instance handler updated instance pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438 state, old state starting, new state running"
time="2020-08-24T17:06:17Z" level=debug msg="Start monitoring pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb-e-9cf92438"
10.42.0.1 - - [24/Aug/2020:17:06:17 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 5151 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:17 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 3472 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:17 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 3444 "" "Go-http-client/1.1"
time="2020-08-24T17:06:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6\", UID:\"4dd48821-45cd-4f0e-bacf-f385ac89d106\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207392\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 has been attached to ip-172-31-11-66"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 starts running, IP 10.42.3.7"
time="2020-08-24T17:06:17Z" level=debug msg="Instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 starts running, Port 10001"
time="2020-08-24T17:06:17Z" level=debug msg="Instance handler updated instance pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0 state, old state starting, new state running"
time="2020-08-24T17:06:17Z" level=debug msg="Start monitoring pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d-e-eceaa7b0"
time="2020-08-24T17:06:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb\", UID:\"3d2183f1-d156-4ee6-a144-e474e7d6e0b7\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207397\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb has been attached to ip-172-31-11-66"
time="2020-08-24T17:06:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d\", UID:\"2f065d53-a34e-4223-a3b2-2ea0fd0ab441\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"207418\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d has been attached to ip-172-31-23-91"
10.42.0.1 - - [24/Aug/2020:17:06:18 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 5146 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:18 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 5146 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:18 +0000] "GET /v1/nodes/ip-172-31-11-66 HTTP/1.1" 200 1980 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:19 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 5139 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:19 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 5139 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:19 +0000] "GET /v1/nodes/ip-172-31-11-66 HTTP/1.1" 200 1979 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:19 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 5152 "" "Go-http-client/1.1"
10.42.3.11 - - [24/Aug/2020:17:06:20 +0000] "GET /v1/volumes/pvc-6db2f638-0d4d-4e8d-bfa5-1f694267812e HTTP/1.1" 200 6207 "" "Go-http-client/1.1"
10.42.3.11 - - [24/Aug/2020:17:06:20 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:20 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 5146 "" "Go-http-client/1.1"
10.42.0.1 - - [24/Aug/2020:17:06:21 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 5139 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:23 +0000] "GET /v1/volumes/pvc-1e6116e6-1c3a-45b9-9753-9d8005cc1020 HTTP/1.1" 200 6214 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:23 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
10.42.3.11 - - [24/Aug/2020:17:06:23 +0000] "GET /v1/volumes/pvc-a567c1f4-e1a4-4eae-b547-eb8d59c6ba9d HTTP/1.1" 200 6246 "" "Go-http-client/1.1"
10.42.3.11 - - [24/Aug/2020:17:06:23 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/volumes/pvc-759c0a0b-2233-43f4-89d6-8f00ec203cc6 HTTP/1.1" 200 6240 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/volumes/pvc-88ffc31c-5d17-49a4-92fe-b2991edd85d8 HTTP/1.1" 200 6245 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/volumes/pvc-41c8fd4d-861a-4412-a7b6-8bf4071dfdfb HTTP/1.1" 200 6233 "" "Go-http-client/1.1"
10.42.2.1 - - [24/Aug/2020:17:06:24 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"

@boknowswiki
Copy link
Contributor

@manhhailua , I think the issue is 'There's no available disk for replica pvc-dc8a0fc4-0483-474c-96b7-84ee14781fe7-r-b632a64f, size 2147483648'. You might need to enable the Replica Node Level Soft Anti-Affinity, here is the document about it as reference Replica Node Level Soft Anti-Affinity.

@manhhailua
Copy link
Author

@boknowswiki I upgraded longhorn release with --set defaultSettings.replicaSoftAntiAffinity=true but nothing changed.
But when I tested simple bitnami/postgresql chart, it worked (sudo helm install postgresql-dev bitnami/postgresql). I think longhorn is fine, I'll keep looking on my chart configurations for the defect.

@adarmanto
Copy link

FYI, I stuck with a similar issue after installing bitnami/redis using longhorn on k8s. By including --set volumePermissions.enabled=true solved my permission denied issue on Redis, it is also stated on the readme section. HTH

@darox
Copy link

darox commented Sep 29, 2021

I'm facing a similar issue on the Prometheus Helm Chart. Provisioning was through Longhorn Auto provisioning, but Prometheus Pods are crashing due to permission issues. I wonder if there's also a parameter for Prometheus to make it working.

@modzilla99
Copy link

I'm facing a similar issue on the Prometheus Helm Chart. Provisioning was through Longhorn Auto provisioning, but Prometheus Pods are crashing due to permission issues. I wonder if there's also a parameter for Prometheus to make it working.

Same here. I did a dirty workaround by scaling the deployment to 0, attaching the volume via the GUI to a node, mounting it the volume on the node to /mnt and then just chowning /mnt to the specific user 65534:65534. After that unmount that volume and scale the deploy to 1.

You can find the device-mapper Endpoint in the UI when clicking on the pvc. It should be under /dev/longhorn/.

@maximemoreillon
Copy link

maximemoreillon commented May 6, 2024

I am currently having the same issue.

I deployed Longhorn using the Helm chart on a 3 node Microk8s cluster composed of 3 Ubuntu servers.

open-iscsi is installed on all nodes.

When I deploy either Grafana or Postgres to the cluster with the Longhorn storage class, the pods fail with permission denied errors.

I followed this guide for my install: https://github.com/balchua/do-microk8s/blob/master/docs/longhorn.md

microk8s helm3 upgrade --install \
  longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --set defaultSettings.defaultDataPath="/longhorn" \
  --set csi.kubeletRootDir="/var/snap/microk8s/common/var/lib/kubelet"

Edit:

I managed to solve the issue by adding a "securityContext" section to my pod specification. Here's the .yml for Grafana:

securityContext:
  runAsUser: 472
  fsGroup: 472

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/question Please use `discussion` to ask questions instead
Projects
None yet
Development

No branches or pull requests

7 participants