Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] Repetitive warnings and errors in a new longhorn setup #6257

Closed
jaqua opened this issue Jul 7, 2023 · 11 comments
Closed

[QUESTION] Repetitive warnings and errors in a new longhorn setup #6257

jaqua opened this issue Jul 7, 2023 · 11 comments
Assignees
Labels
area/performance System, volume performance area/troubleshoot Troubleshoot related backport/1.5.1 component/longhorn-manager Longhorn manager (control plane) kind/improvement Request for improvement of existing function priority/0 Must be fixed in this release (managed by PO)
Milestone

Comments

@jaqua
Copy link

jaqua commented Jul 7, 2023

I've setup a completely new k8s cluster with longhorn 1.5.0.
There is nothing special running on that cluster, but I do get multiple warnings like:

time="2023-07-07T14:51:16Z" level=warning msg="Ignoring recurring job labels on Volume pvc-1e1565de-60a0-4dd8-9e7b-30b5d5a90772 PVC prometheus-community-server due to missing source label" controller=longhorn-volume node=worker3

and repetitive errors like this:

time="2023-07-07T14:49:46Z" level=info msg="Forwarding request to 10.42.2.155:9500"
time="2023-07-07T14:49:46Z" level=info msg="CreateSnapshot: req: {\"name\":\"snapshot-275ba4a0-10aa-4024-87d8-2f82a37fb5b0\",\"parameters\":{\"type\":\"bak\"},\"source_volume_id\":\"pvc-7c074207-69ba-44de-af57-fba9cc564130\"}"
time="2023-07-07T14:49:46Z" level=info msg="GetPluginInfo: rsp: {\"name\":\"driver.longhorn.io\",\"vendor_version\":\"v1.5.0\"}"
time="2023-07-07T14:49:46Z" level=info msg="GetPluginInfo: req: {}"
I0707 14:49:46.203466       1 snapshot_controller.go:291] createSnapshotWrapper: Creating snapshot for content snapcontent-275ba4a0-10aa-4024-87d8-2f82a37fb5b0 through the plugin ...
E0707 14:49:46.203420       1 snapshot_controller_base.go:283] could not sync content "snapcontent-275ba4a0-10aa-4024-87d8-2f82a37fb5b0": failed to take snapshot of the volume pvc-7c074207-69ba-44de-af57-fba9cc564130: "rpc error: code = NotFound desc = volume pvc-7c074207-69ba-44de-af57-fba9cc564130 not found"
E0707 14:49:46.203323       1 snapshot_controller.go:124] checkandUpdateContentStatus [snapcontent-275ba4a0-10aa-4024-87d8-2f82a37fb5b0]: error occurred failed to take snapshot of the volume pvc-7c074207-69ba-44de-af57-fba9cc564130: "rpc error: code = NotFound desc = volume pvc-7c074207-69ba-44de-af57-fba9cc564130 not found"
time="2023-07-07T14:49:46Z" level=error msg="CreateSnapshot: err: rpc error: code = NotFound desc = volume pvc-7c074207-69ba-44de-af57-fba9cc564130 not found"
time="2023-07-07T14:49:46Z" level=error msg="Failed to create snapshot" component=csi-controller-server error="rpc error: code = NotFound desc = volume pvc-7c074207-69ba-44de-af57-fba9cc564130 not found" function=CreateSnapshot
I0707 14:49:46.191116       1 snapshot_controller.go:325] createSnapshotWrapper: CreateSnapshot for content snapcontent-275ba4a0-10aa-4024-87d8-2f82a37fb5b0 returned error: rpc error: code = NotFound desc = volume pvc-7c074207-69ba-44de-af57-fba9cc564130 not found

I don't understand the warning and the error. Obviously the volume is not existing, but why is it trying to create a snapshot at all and why is it trying this on a volume which is not existing?

I guess both parts are an issue of misconfiguration.

My values.yaml file:

persistence:
  defaultFsType: xfs
networkPolicies:
  enabled: true
  type: rke2
defaultSettings:
  storageOverProvisioningPercentage: 150
  removeSnapshotsDuringFilesystemTrim: enabled

Environment

  • Longhorn version: 1.5.0
  • Kubernetes version: 1.27.2
  • Node config
    • OS type and version: ubuntu 22
    • CPU per node: 4
    • Memory per node: 8
    • Disk type NVMe
  • Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
@jaqua jaqua added the kind/question Please use `discussion` to ask questions instead label Jul 7, 2023
@derekbit
Copy link
Member

derekbit commented Jul 8, 2023

cc @PhanLe1010

@derekbit
Copy link
Member

derekbit commented Jul 8, 2023

time="2023-07-07T14:51:16Z" level=warning msg="Ignoring recurring job labels on Volume pvc-1e1565de-60a0-4dd8-9e7b-30b5d5a90772 PVC prometheus-community-server due to missing source label" controller=longhorn-volume node=worker3

The warning is fine.

and repetitive errors like this:

Can you provide a support bundle?

@jaqua
Copy link
Author

jaqua commented Jul 8, 2023

@derekbit Thank you for your reply. Here is the support bundle

@derekbit
Copy link
Member

derekbit commented Jul 9, 2023

These volumes such as pvc-d259794e-f52d-41fc-8d43-71f0b8af3ccd and pvc-7c074207-69ba-44de-af57-fba9cc564130 don't exist. That's why csi-snapshotter complains the failed to take snapshot of the volume...not found.

@derekbit
Copy link
Member

derekbit commented Jul 9, 2023

Do you have outdated configurations and volumesnapshot for taking snapshots of these volumes?

@jemand771
Copy link

heya, I don't mean to steal this issue but the first warning might be a broader problem:

Since upgrading from v1.4.2 to v1.5.0, I'm getting a lot of them, for what feels like all (52; 38 attached) of my volumes. Since the number of log lines seem to spike every 4 hours, I assume they're related to my 4h-ly snapshot job:

image

here's a short excerpt from the longhorn-manager logs:

click to expand...
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-fc9dfcfe-c68d-4656-b626-d320b01bb7a0 PVC data-minio-3 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-23c0bee8-18a2-4582-90d5-881d258bda91 PVC data-loki-write-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	10.42.247.162 - - [09/Jul/2023:10:49:39 +0000] "GET /v1/volumes/pvc-96836591-4725-461e-bf6e-e3ff776bee2e HTTP/1.1" 200 16659 "" "Go-http-client/1.1"
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-c5d2a149-83c6-4180-b64a-093d0b212dfe PVC data-minio-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f7463826-edda-4e90-81fe-beb5e31af78c PVC smp5-data-staging due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f0ad7c5b-3b10-4ef6-baba-afa251371d55 PVC data-loki-write-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-47b76234-55af-4dec-9385-a59bcfb6458b PVC kps-grafana due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-62627677-d937-45c9-952f-a4876a248194 PVC redis-data-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f7463826-edda-4e90-81fe-beb5e31af78c PVC smp5-data-staging due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-c5d2a149-83c6-4180-b64a-093d0b212dfe PVC data-minio-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:39+02:00	time="2023-07-09T10:49:39Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f0ad7c5b-3b10-4ef6-baba-afa251371d55 PVC data-loki-write-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:40+02:00	time="2023-07-09T10:49:40Z" level=warning msg="Ignoring recurring job labels on Volume pvc-1dc16f90-9b3e-4f46-8655-1e0ac7674693 PVC data-minio-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-1dc16f90-9b3e-4f46-8655-1e0ac7674693 PVC data-minio-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-33e5ec26-4bf3-4c3f-a974-fa1c933a5b8a PVC redis-data-authentik-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-95527332-1510-4546-9d68-433e7a17c8b5 PVC data-coder-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-acce2bf0-0f0a-4361-9588-f8b43f8d6959 PVC paperless-data due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-33e5ec26-4bf3-4c3f-a974-fa1c933a5b8a PVC redis-data-authentik-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-95527332-1510-4546-9d68-433e7a17c8b5 PVC data-coder-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:41+02:00	time="2023-07-09T10:49:41Z" level=warning msg="Ignoring recurring job labels on Volume pvc-95527332-1510-4546-9d68-433e7a17c8b5 PVC data-coder-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:42+02:00	time="2023-07-09T10:49:42Z" level=warning msg="Ignoring recurring job labels on Volume pvc-654216e9-6977-4ee0-a4bf-ceaead69816b PVC data-loki-write-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:42+02:00	time="2023-07-09T10:49:42Z" level=warning msg="Ignoring recurring job labels on Volume pvc-654216e9-6977-4ee0-a4bf-ceaead69816b PVC data-loki-write-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:43+02:00	10.42.247.162 - - [09/Jul/2023:10:49:43 +0000] "GET /v1/volumes/pvc-7fb0f815-c718-4f69-a436-c8be5dc8ecb3 HTTP/1.1" 200 15377 "" "Go-http-client/1.1"
2023-07-09T12:49:43+02:00	10.42.247.162 - - [09/Jul/2023:10:49:43 +0000] "GET /v1/volumes/pvc-acce2bf0-0f0a-4361-9588-f8b43f8d6959 HTTP/1.1" 200 15216 "" "Go-http-client/1.1"
2023-07-09T12:49:43+02:00	time="2023-07-09T10:49:43Z" level=warning msg="Ignoring recurring job labels on Volume pvc-80172caf-ba5e-43f9-a7d5-4dda55738ab0 PVC prometheus-kps-kube-prometheus-stack-db-prometheus-kps-kube-prometheus-stack-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:43+02:00	time="2023-07-09T10:49:43Z" level=warning msg="Ignoring recurring job labels on Volume pvc-80172caf-ba5e-43f9-a7d5-4dda55738ab0 PVC prometheus-kps-kube-prometheus-stack-db-prometheus-kps-kube-prometheus-stack-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-23c0bee8-18a2-4582-90d5-881d258bda91 PVC data-loki-write-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-23c0bee8-18a2-4582-90d5-881d258bda91 PVC data-loki-write-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f0ad7c5b-3b10-4ef6-baba-afa251371d55 PVC data-loki-write-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-62627677-d937-45c9-952f-a4876a248194 PVC redis-data-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f7463826-edda-4e90-81fe-beb5e31af78c PVC smp5-data-staging due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-47b76234-55af-4dec-9385-a59bcfb6458b PVC kps-grafana due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f0ad7c5b-3b10-4ef6-baba-afa251371d55 PVC data-loki-write-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:44+02:00	time="2023-07-09T10:49:44Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f7463826-edda-4e90-81fe-beb5e31af78c PVC smp5-data-staging due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-96836591-4725-461e-bf6e-e3ff776bee2e PVC mailu-storage due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-b66e781b-0bae-4e64-a132-7df47a70d2d4 PVC data-minio-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-6054334b-7ba2-426b-b3cf-5c35ba767cf3 PVC data-nginx-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-bc2bc1e7-fb6c-4e98-9ee6-93108da99cb6 PVC kuma due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-0dff2a1a-1dfc-4763-bf5b-45d5e726e479 PVC data-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-a8940a9b-c9f5-4a31-adf2-f6ba6c54dc14 PVC coder-willy-ravy-mopsdisplay-workspace due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-baa03094-7be1-4916-b8b9-086e1f9d4750 PVC data-nginx-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-678e058a-24b8-4042-b099-c14363e40739 PVC data-mailu-clamav-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-cfaf9618-50f1-45fd-b4b7-032208e28abc PVC coder-willy-coder-test-workspace due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-47b76234-55af-4dec-9385-a59bcfb6458b PVC kps-grafana due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-777a26fc-1169-4248-9272-cd34dbf2ca7f PVC warpgate-data due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-95527332-1510-4546-9d68-433e7a17c8b5 PVC data-coder-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-a911a62a-d7b7-48f3-abc4-9c4f3880290a PVC data-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-b65d795a-ffa5-4622-8d18-c5246737639c PVC smp5-data due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f0ad7c5b-3b10-4ef6-baba-afa251371d55 PVC data-loki-write-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-0bfe2332-9a04-4f10-98f4-7eab03e04ef4 PVC data-authentik-postgresql-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-33e5ec26-4bf3-4c3f-a974-fa1c933a5b8a PVC redis-data-authentik-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-6abb67a9-5389-4839-b938-bb34aa81665b PVC wordpress due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-cccb4827-9682-48eb-b191-c9a587905285 PVC data-mariadb-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-e0841e07-d3a1-4403-99e3-bc2f5012faa6 PVC data-nginx-3 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-525808e2-e67c-4df8-8c89-44a2838263bb PVC data-wordpress-mariadb-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-93aa7172-a139-4b89-a9ea-c684510067d1 PVC redis-data-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-80172caf-ba5e-43f9-a7d5-4dda55738ab0 PVC prometheus-kps-kube-prometheus-stack-db-prometheus-kps-kube-prometheus-stack-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-a9bf45db-5756-4465-8b48-d62d37aad1d3 PVC mysql due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-c5758520-9e29-440d-8191-71d601ddbee5 PVC redis-data-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-0670996d-cd3e-4eef-bf85-941d870416e4 PVC redis-data-redis-master-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-c5d2a149-83c6-4180-b64a-093d0b212dfe PVC data-minio-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-23c0bee8-18a2-4582-90d5-881d258bda91 PVC data-loki-write-1 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-654216e9-6977-4ee0-a4bf-ceaead69816b PVC data-loki-write-0 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-663e19b7-85c6-4dd1-b5af-eaafd1c0f9c9 PVC data-loki-backend-2 due to missing source label" controller=longhorn-volume node=k3s-master-0
2023-07-09T12:49:45+02:00	time="2023-07-09T10:49:45Z" level=warning msg="Ignoring recurring job labels on Volume pvc-f7463826-edda-4e90-81fe-beb5e31af78c PVC smp5-data-staging due to missing source label" controller=longhorn-volume node=k3s-master-0

...you get the idea.

this is the job's definition:

apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: 4h-snapshot
  namespace: longhorn-system
spec:
  concurrency: 1
  cron: 0 0/4 * * ?
  groups:
    - default
  retain: 12
  task: snapshot

It's meant to run on all volumes unless specified otherwise, which works.

From digging through the code a bit, it looks like the recurring-job.longhorn.io/source label is missing somewhere, but I couldn't figure out where exactly or what value it should have.

Is this something longhorn should patch automatically? Unless I've misconfigured something, to me it looks like anyone upgrading to (and per the OP's description: anyone freshly installing) longhorn v1.5.0 will get this message on recurring job runs?

let me know if you need a support bundle from me, 75MB is too big to just attach to a github issue >_<

@derekbit
Copy link
Member

derekbit commented Jul 9, 2023

heya, I don't mean to steal this issue but the first warning might be a broader problem:

Since upgrading from v1.4.2 to v1.5.0, I'm getting a lot of them, for what feels like all (52; 38 attached) of my volumes. Since the number of log lines seem to spike every 4 hours, I assume they're related to my 4h-ly snapshot job:

image

here's a short excerpt from the longhorn-manager logs:

click to expand...
this is the job's definition:

apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: 4h-snapshot
  namespace: longhorn-system
spec:
  concurrency: 1
  cron: 0 0/4 * * ?
  groups:
    - default
  retain: 12
  task: snapshot

It's meant to run on all volumes unless specified otherwise, which works.

From digging through the code a bit, it looks like the recurring-job.longhorn.io/source label is missing somewhere, but I couldn't figure out where exactly or what value it should have.

Is this something longhorn should patch automatically? Unless I've misconfigured something, to me it looks like anyone upgrading to (and per the OP's description: anyone freshly installing) longhorn v1.5.0 will get this message on recurring job runs?

let me know if you need a support bundle from me, 75MB is too big to just attach to a github issue >_<

Using trace or debug log level might be better to avoid flooding messages
WDYT? @ChanYiLin @c3y1huang @innobead

@innobead
Copy link
Member

innobead commented Jul 9, 2023

@c3y1huang Please help with this. Need to get to 1.5.1.

@longhorn-io-github-bot
Copy link

longhorn-io-github-bot commented Jul 10, 2023

Pre Ready-For-Testing Checklist

  • Where is the reproduce steps/test steps documented?
    The reproduce steps/test steps are at:

  • Is there a workaround for the issue? If so, where is it documented?
    The workaround is at:

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at:
    The PR for the chart change is at: fix(chart): update default setting log level #6267

  • Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
    The PR is at:

  • Which areas/issues this PR might have potential impacts on?
    Area manager
    Issues

  • If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
    The LEP PR is at

  • If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
    The UI issue/PR is at

  • If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
    The documentation issue/PR is at

  • If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
    The automation skeleton PR is at
    The automation test case PR is at
    The issue of automation test case implementation is at (please create by the template)

  • If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
    The engine automation PR is at

  • If labeled: require/manual-test-plan Has the manual test plan been documented?
    The updated manual test plan is at

  • If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
    The compatibility issue is filed at

@c3y1huang
Copy link
Contributor

Removing the v1.3 and v1.4 backport labels because the log is introduced in v1.5.

@roger-ryao
Copy link

Verified on master-head 20230815

Result Passed

After Helm installation and manifests, I have checked the following items:

  • The logLevel parameter can be configured in the values.yaml file and will be reflected as log-level.
  • The value of log-level can be modified.
  • The default value of log-level is set to Info.
  • We couldn't see any logs related to Ignoring recurring job labels on Volume when the log-level is set to Info. However, after changing the log-level value to Debug, we can observe logs related to Ignoring recurring job labels on Volume.

@innobead innobead changed the title [QUESTION] Repetetive warnings and errors in a new longhorn setup [QUESTION] Repetitive warnings and errors in a new longhorn setup Feb 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/performance System, volume performance area/troubleshoot Troubleshoot related backport/1.5.1 component/longhorn-manager Longhorn manager (control plane) kind/improvement Request for improvement of existing function priority/0 Must be fixed in this release (managed by PO)
Projects
Status: Resolved/Scheduled
Development

No branches or pull requests

7 participants