Proposal: Make net-conf and net-plugin directory configurable

[yifan-gu]

Problem

rkt supports CNI plugins for setting up networks. However, currently we cannot configure where to find the network config files or network plugin binaries:

• network config files located under $local-config/net.d (https://github.com/coreos/rkt/blob/v1.1.0/networking/podenv.go#L230)
• network plugin binaries located under $local-config/net.d, /usr/lib/rkt/plugins/net or $pod-stage1-rootfs/usr/lib/rkt/plugins/net (https://github.com/coreos/rkt/blob/v1.1.0/networking/net_plugin.go#L83-L85)

They are all sort of hardcoded.

Proposal

My proposal would be adding two ways to configure the network config dir. and network plugin binary dir:

• Add flags --net-conf-dir and --net-plugin-dir to rkt run/run-prepared.
• Add two paths fields (net-conf and net-plugin in the paths configuration)

The way it works:

• If nothing mentioned above is set, use the defaults as today.
• If paths fields are set, then use them as the source of truth to find net config files and plugin binaries.
• If CLI flags are set, then use them as the source of truth, overriding the paths fields.

The reason behind this is that users are expected to write config files under local-config in production. And CLI flags are more used for manual debugging and testing purpose, so they are expected to override the config files.

Previous work:
#1992
#2013

Also ref: kubernetes/kubernetes#21047 (comment)

cc @jonboulle @krnowak @steveej @iaguis @philips

[yifan-gu]

Any feedback?

[jonboulle]

This seems fine to me. @steveej ?

[yifan-gu]

Sent a PR for the implementation for the CLI flag part. #2270

[yifan-gu]

Discussed with @euank on kubernetes/kubernetes#24688 (comment)

We think symlinking the net.d directory to the directory provided by the --network-plugin-dir flag is not an ideal long term solution for using CNI plugins provided by kubelet users.

rkt should still provide a way to specify the network directory that contains CNI config files.

cc @krnowak

[krnowak]

@yifan-gu

I was trying to move away from CNI config files in rkt in favor of configuration that is consistent with other rkt configuration (using rktVersion and rktKind). Not sure if that flies well with you (for rktnetes, I mean), I have no idea about it.

About additional paths for configuration, I guess you can't just use --user-config flag to point to your directory with the network config?

[yifan-gu]

About additional paths for configuration, I guess you can't just use --user-config flag to point to your directory with the network config?

Nope, the --user-config=$foo will have a structure like $foo/net.d, and rkt looks up CNI config files under $foo/net.d. I need it to look up CNI config files at an arbitrary path.

[yifan-gu]

I was trying to move away from CNI config files in rkt in favor of configuration that is consistent with other rkt configuration

Why we want to do this? This makes the CNI configs not usable by rkt unless we modify it to meet the rkt configuration format.

I think CNI configs should be recognizable by rkt, this makes it easy to manage when another program and rkt share one CNI config.

[s-urbaniak]

xref #2312

[s-urbaniak]

didn't make it due to OCI/Fest activity

[jonboulle]

@yifan-gu @s-urbaniak can you sync and update on the status of this one please

[s-urbaniak]

@jonboulle thanks for the reminder, we'll meet today anyways, I'll discuss this today.

[euank]

My understanding is that our setting up the network namespace in the kubelet instead of rkt (kubernetes/kubernetes#25062) means that this is less of priority for kubernetes integration in the short-to-mid term at the least.

I'm less certain if we'd want this for rktnetes long term since I'm unsure what the longer term vision for kubelet container networking / sandboxing is there (and I think we need to wait on some dust to settle).

Because of that, I've re-prioritized, feel free to change it further.

I've left it open because it seems like it could be useful in other use-cases, but that can also be up for discussion.