Make the CSI deployment default for bpfd-operator

[astoycos]

With #727 we have preliminary support for implementing the Container Storage interface Spec to allow us to more easily share bpf maps with their userspace applications, we should now make this the default for our k8s deployment and all of the examples that we maintain. This will allow us to simplify the examples AND make them truly unprivileged. Also we'll need to update all of the documentation for these as well.

[Billy99]

Moving to CSI may limit us to Kubernetes v1.26 and later. This v1.26 PR (kubernetes/kubernetes#112597) addresses a gRPC Protocol Error that was seen in the CSI client code and it doesn't appear to have been backported.

When bpfd starts with CSI enabled and an older version of Kubernetes, the node-driver-registrar container in the bpfd-daemon fails to start:

$ kubectl get pods -A
NAMESPACE            NAME                                                    READY   STATUS             RESTARTS      AGE
bpfd                 bpfd-daemon-9nfg4                                       2/3     CrashLoopBackOff   3 (15s ago)   74s
bpfd                 bpfd-operator-7d7fd8f565-mgpsw                          2/2     Running            0             78s
:

And logs:

$ kubectl logs -n bpfd bpfd-daemon-9nfg4 -c node-driver-registrar
I1030 16:26:25.067698       1 main.go:135] Version: v2.9.0
I1030 16:26:25.067821       1 main.go:136] Running node-driver-registrar in mode=
I1030 16:26:25.067848       1 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock"
I1030 16:26:25.067893       1 connection.go:213] Connecting to unix:///csi/csi.sock
I1030 16:26:25.068740       1 main.go:164] Calling CSI driver to discover driver name
I1030 16:26:25.068904       1 connection.go:242] GRPC call: /csi.v1.Identity/GetPluginInfo
I1030 16:26:25.068952       1 connection.go:243] GRPC request: {}
I1030 16:26:25.074610       1 connection.go:249] GRPC response: {"name":"csi.bpfd.dev","vendor_version":"0.3.0"}
I1030 16:26:25.074629       1 connection.go:250] GRPC error: <nil>
I1030 16:26:25.074636       1 main.go:173] CSI driver name: "csi.bpfd.dev"
I1030 16:26:25.074690       1 node_register.go:55] Starting Registration Server at: /registration/csi.bpfd.dev-reg.sock
I1030 16:26:25.074922       1 node_register.go:64] Registration Server started at: /registration/csi.bpfd.dev-reg.sock
I1030 16:26:25.075088       1 node_register.go:88] Skipping HTTP server because endpoint is set to: ""
I1030 16:26:26.277559       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1030 16:26:26.285800       1 main.go:101] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR,}
E1030 16:26:26.285873       1 main.go:103] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR, restarting registration container.