Nullpointer Exception on Backup

[sashokbg]

Description

When trying to run a manual backup (see yaml bellow) I get a strange null pointer exception.

Additional Context

• My cluster:

✗ kubectl get nodes
NAME    STATUS   ROLES                         AGE     VERSION
node1   Ready    control-plane,master,worker   7d19h   v1.23.7
node2   Ready    worker                        7d19h   v1.23.7
node3   Ready    worker                        7d19h   v1.23.7

• k8up operator helm chart: k8up-4.2.2
• Restores works fine
• I am using NFS with nfs-subdir-external-provisioner

Logs

2023-05-21T11:49:11Z	INFO	k8up.restic.restic.backup	starting backup
2023-05-21T11:49:11Z	INFO	k8up.restic.restic.backup	starting backup for folder	{"foldername": "data-nextcloud"}
2023-05-21T11:49:11Z	INFO	k8up.restic.restic.backup.command	restic command	{"path": "/usr/local/bin/restic", "args": ["backup", "--option", "", "--json", "--host", "home-cloud", "/data/data-nextcloud", "--tag", "manual"]}
2023-05-21T11:49:11Z	INFO	k8up.restic.restic.backup.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
2023-05-21T11:49:16Z	INFO	k8up.restic.restic.backup.progress	progress of backup	{"percentage": "0.00%"}
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.backup.progress	backup finished	{"new files": 0, "changed files": 0, "errors": 0}
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.backup.progress	stats	{"time": 10.738905407, "bytes added": 353, "bytes processed": 4063429439}
2023-05-21T11:49:23Z	INFO	k8up.restic.statsHandler.promStats	sending prometheus stats	{"url": "http://127.0.0.1/"}
2023-05-21T11:49:23Z	ERROR	k8up.restic.restic	prometheus send failed	{"error": "Post \"http://127.0.0.1/metrics/job/restic_backup/instance/home-cloud\": dial tcp 127.0.0.1:80: connect: connection refused"}
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats
	/home/runner/work/k8up/k8up/restic/cli/backup.go:95
github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out
	/home/runner/work/k8up/k8up/restic/logging/logging.go:162
github.com/k8up-io/k8up/v2/restic/logging.writer.Write
	/home/runner/work/k8up/k8up/restic/logging/logging.go:103
io.copyBuffer
	/opt/hostedtoolcache/go/1.19.9/x64/src/io/io.go:429
io.Copy
	/opt/hostedtoolcache/go/1.19.9/x64/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1
	/opt/hostedtoolcache/go/1.19.9/x64/src/os/exec/exec.go:407
os/exec.(*Cmd).Start.func1
	/opt/hostedtoolcache/go/1.19.9/x64/src/os/exec/exec.go:544
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.backup	backup finished, sending snapshot list
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.snapshots	getting list of snapshots
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.snapshots.command	restic command	{"path": "/usr/local/bin/restic", "args": ["snapshots", "--option", "", "--json"]}
2023-05-21T11:49:23Z	INFO	k8up.restic.restic.snapshots.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x141887e]

goroutine 1 [running]:
github.com/k8up-io/k8up/v2/restic/kubernetes.filterByRepo(...)
	/home/runner/work/k8up/k8up/restic/kubernetes/snapshots.go:123
github.com/k8up-io/k8up/v2/restic/kubernetes.SyncSnapshotList({0x1a42388, 0xc00060cf80}, {0xc0004e5500, 0x87, 0x0?}, {0xc000044009, 0xa}, {0xc000048012, 0x29})
	/home/runner/work/k8up/k8up/restic/kubernetes/snapshots.go:29 +0x35e
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendSnapshotList(0xc0002ac080)
	/home/runner/work/k8up/k8up/restic/cli/backup.go:116 +0x19b
github.com/k8up-io/k8up/v2/restic/cli.(*Restic).Backup(0xc0002ac080, {0x17bcee5, 0x5}, {0xc000173b90, 0x1, 0x1})
	/home/runner/work/k8up/k8up/restic/cli/backup.go:46 +0x2da
github.com/k8up-io/k8up/v2/cmd/restic.doBackup({0x1a423c0?, 0xc0000c8000?}, 0xc000380810?, {{0x1a44e50?, 0xc000380810?}, 0xc000380900?})
	/home/runner/work/k8up/k8up/cmd/restic/main.go:234 +0x105
github.com/k8up-io/k8up/v2/cmd/restic.run({0x1a423c0, 0xc0000c8000}, 0x1a44e50?, {{0x1a44e50?, 0xc000380810?}, 0x1a30ec8?})
	/home/runner/work/k8up/k8up/cmd/restic/main.go:129 +0xbf
github.com/k8up-io/k8up/v2/cmd/restic.resticMain(0xc00060c440)
	/home/runner/work/k8up/k8up/cmd/restic/main.go:113 +0x2a7
github.com/urfave/cli/v2.(*Command).Run(0x2598960, 0xc00060c440, {0xc000380750, 0x3, 0x3})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/command.go:271 +0xa42
github.com/urfave/cli/v2.(*Command).Run(0xc00038c640, 0xc0003b0e80, {0xc0000c6040, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/command.go:264 +0xc97
github.com/urfave/cli/v2.(*App).RunContext(0xc000388000, {0x1a423c0?, 0xc0000c8000}, {0xc0000c6040, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/app.go:333 +0x665
github.com/urfave/cli/v2.(*App).Run(...)
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.23.7/app.go:310
main.main()
	/home/runner/work/k8up/k8up/cmd/k8up/main.go:30 +0x45

Expected Behavior

Backup should work

Steps To Reproduce

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-after-server-change
spec:
  tags:
    - manual
  promURL: ""
  podSecurityContext:
    runAsUser: 0
    runAsGroup: 0
  backend:
    envFrom:
      - secretRef:
          name: "backup-secret"
    repoPasswordSecretRef:
      name: "restic-repository-password"
      key: "password"
    swift:
      path: /archives
      container: home-cloud-backup-storage
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 2

Version of K8up

v4.2.2

Version of Kubernetes

v1.23.7

Distribution of Kubernetes

Bare metal

[Kidswiss]

Hi @sashokbg

Do you have any snapshot objects in that namespace? kubectl get snapshots? If you do, could you please post the yaml output of one of those?

It looks to me that a field is not correctly populated in the snapshot object when using the swift backend.

[sashokbg]

Hello @Kidswiss thank you for your message here are some yamls from the snapshots:

apiVersion: k8up.io/v1
kind: Snapshot
metadata:
  creationTimestamp: "2023-05-14T02:00:26Z"
  generation: 1
  name: f4856c6f
  namespace: home-cloud
  resourceVersion: "85254"
  uid: cc1e2de6-35e1-4dcd-934c-e36b06914cb3
spec:
  date: "2023-03-27T02:00:20Z"
  id: f4856c6fe842d6421187f9368171089fde4316bcf5c7a9053cceba10f192f339
  paths:
  - /home-cloud-postgresql.sql

apiVersion: k8up.io/v1
kind: Snapshot
metadata:
  creationTimestamp: "2023-05-14T02:00:28Z"
  generation: 1
  name: f0cd4ef4
  namespace: home-cloud
  resourceVersion: "85267"
  uid: d5a29422-c0b5-4d79-b4e4-a3f573592155
spec:
  date: "2023-04-18T02:00:12Z"
  id: f0cd4ef4c1f4d26b0eb823310303c2357ef14e7dbaf0f714b18b26abb2603ae0
  paths:
  - /data/data-nextcloud

[sashokbg]

Hello @Kidswiss I have worked on adding the swift support via EnvFrom variables in #657 can you please give me some directions where the NPE might be comming from ? I am not well versed in GO unfortunately :(

[Kidswiss]

Hi @sashokbg

Sorry was a bit busy...

I just checked your snapshots and I see that the field spec.repository is missing. That's what's causing the nilpointer.

It should take the value of RESTIC_REPOSITORY in the backup pod to populate this field. Could you please post a yaml of the backup pod? I'd like to check if all environment variables are available as expected.

EDIT: it might start to work if you use envFrom to inject following envVar:

RESTIC_REPOSITORY=swift:home-cloud-backup-storage/archives

Then you delete the existing snapshot objects in the namespace and trigger a backup again.

[sashokbg]

Hello @Kidswiss

Good news your suggestion worked perfectly !!

I checked the snapshots and saw they did not have the repository field that was added recently so I checked my CRDs and found out I forgot to update them (silly me).

I updated the CRDs and then deleted all snapshots as you suggested and everything worked.

Thank you very much for your kind help