Velero pod enters "CrashLoopBackOff" with powerprotect-velero-dd plugin

[Frank51]

Using Velero v1.7.0-rc.1 cli binary to deploy Velero server pod in "Velero" namespace, including powerprotect-velero-dd as a plugin. The deployment of Velero pod failed.
And the log shows it relates registering plugin. But same operation is successful when using Velero 1.6.3.
k get pods -n velero-ppdm
NAME READY STATUS RESTARTS AGE
velero-688b667d77-8vxhn 0/1 CrashLoopBackOff 8 20m

The velero server pod enters "CrashLoopBackOff" status.

Command:
./velero install --namespace velero-ppdm --image velero/velero:v1.7.0-rc.1 --no-secret --no-default-backup-location --use-volume-snapshots=false --plugins dellemc/powerprotect-velero-dd:19.9.0-12 --features EnableLocalMode

The Velero pod should be running successfully on the Velero namespace.

• kubectl logs deployment/velero -n velero
  https://gist.github.com/Frank51/072e2f4a041184618617ec05dfab34d4
• Velero version (use velero version): Version: v1.7.0-rc.1
• Kubernetes version (use kubectl version): v1.22.0 (client), v1.21.1(server)
• OS (e.g. from /etc/os-release):Ubuntu 18.04 LTS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[reasonerjt]

I'm not sure how the plugin was implemented, but some changes made in v1.7:

1. It was compiled by go 1.16
2. It was built upon distroless, so a lot of software are removed from the velero image.

Hope that could help you identify the reason the plugin failed.

[phuongatemc]

This is critical. It will block us from upgrade to Velero 1.7.0

[phuongatemc]

I turned on debug mode and it spilled out more details on the error. The Velero hit panic because the velerodd plugin trying to load DDBoost library and failed. See error log below:
...
time="2021-09-24T14:36:09Z" level=debug msg="plugin started" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velerodd pid=25
time="2021-09-24T14:36:09Z" level=debug msg="waiting for RPC address" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" path=/plugins/velerodd
time="2021-09-24T14:36:09Z" level=debug msg="time="2021-09-24T14:36:09Z" level=info msg="Initializing libDDBoost"" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="time="2021-09-24T14:36:09Z" level=info msg="Opening log for DDBoost package ..." fields.level=TRACE prefix="TRACE: "" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="time="2021-09-24T14:36:09Z" level=info msg="Opened log for DDBoost package." fields.level=INFO prefix="INFO: "" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="time="2021-09-24T14:36:09Z" level=info msg="Unable to initialize DDBoost library '/plugins/': [1] error in loading libDDBoost library" fields.level=ERROR prefix="ERROR: "" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="time="2021-09-24T14:36:09Z" level=panic msg="DDBoost initialization error Unable to initialize DDBoost library '/plugins/': [1] error in loading libDDBoost library"" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="panic: (*logrus.Entry) 0xc00046d180" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="goroutine 1 [running]:" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="github.com/sirupsen/logrus.Entry.log(0xc00046cd20, 0xc000281c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
time="2021-09-24T14:36:09Z" level=debug msg="\t/space/jenkins/workspace/vproxy-kubernetes_release_19.9.0/golang/velerodd/.external-resources/goModCache/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345" cmd=/plugins/velerodd logSource="pkg/plugin/clientmgmt/logrus_adapter.go:74" pluginName=velerodd
...

[reasonerjt]

@phuongatemc
So could any reason mentioned in this comment cause such failure?
#4192 (comment)

[dsu-igeek]

With the switch to distroless there are a lot fewer libraries in the base image. Check to see if there are libraries you need that are not in the distroless image. If that's the case, you can include them in the plugin and copy them to the /plugins dir as part of the plugin's init. When the plugins are invoked, LD_LIBRARY_PATH is set to include /plugins. We use that in the vSphere plugin to include vSphere specific libraries.

[phuongatemc]

Sound like the root cause of this problem is the ddboost library itself requires a few libraries that are not currently in the distroless image or in the /plugins directory (where the LD_LIBRARY_PATH points to). We will investigate further and update you.

[dsu-igeek]

@phuongatemc That sounds right. When you discover which libraries you're missing, include them in your plug-in image and copy them into the /plugins directory with your init container.

[phuongatemc]

We have added all the libraries needed but the plug in still failed. The logs didn't show much info.
To debug, we copy the plugin binary to the scratch instead and then executed the plugin directly on the velero container using exec, we got the error 139
phuong@irv-vm-90-22:~/vproxy-kubernetes/docker/velerodd> k exec -n velero-ppdm velero-858c5859d7-bxsdc -c velero -- /scratch/velerodd
command terminated with exit code 139

[dsu-igeek]

I think you're going to need to add some debug code into the plugin or figure out how to run it in the debugger and single-step through.

[phuongatemc]

The problem resolves after adding the missing libraries to the velerodd plugin. It's OK to close this issue.

[Frank51]

Solved it. Close.

[phuongatemc]

The reason why we added "all missing libraries" and still failed because we happened to add libraries that already in the Velero container. After we eliminated the duplicate libraries and added the missing ones that not in Velero container, the plugin was executed successfully.