Specify Pod default network in Annotations

[pliurh]

Allow pod to specify cluster default network in annotations, When more than one cluster network is available.
Create a new field 'k8s.v1.cni.cncf.io/default-network' in pod annotations, which is used to overwrite the cluster default network defined in Multus config file.

Fix #158

[coveralls]

Pull Request Test Coverage Report for Build 524

• 29 of 45 (64.44%) changed or added relevant lines in 1 file are covered.
• 109 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+2.7%) to 52.152%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
k8sclient/k8sclient.go	29	45	64.44%

Files with Coverage Reduction	New Missed Lines	%
types/conf.go	32	27.93%
multus/multus.go	77	54.36%

Totals
Change from base Build 495:	2.7%
Covered Lines:	630
Relevant Lines:	1208

---

💛 - Coveralls

[kannanvr]

@rkamudhan @dougbtv Request you to merge this PR as early as possible. This PR would solve our use case. FOllowing is our Use case.
Multiple tenants in networks - where every tenants / NS will be configured with a different subnet and any pod in a NS will have the IPAddr from that configured subnet and pods across Namespaces should not connect to each other. There is no management IP / subnet concept

[pliurh]

@s1061123 @rkamudhan @dougbtv could you please review it? Thanks.

[s1061123]

Could you change "multus-cni.io/default-network" to "multus-cni.io/v1/default-network"?
Once it fixed we merge it! Thank you for your cooperation!

[dougbtv]

I'm OK with moving forward on this one, there's no rush and it shouldn't block the merge, but... Let's follow up with some documentation -- one that includes a big warning for usage. We can also understate it in the documentation -- as to not encourage people to use this functionality for non-standard use-cases. I created the issue in #200. Again, no big priority on the documentation, but, should be a follow up at some point. Thanks!

[joshlreese]

We've been keeping up with this PR and have been testing the changes as they've come through. The recommendation of using multus-cni.io/v1/default-network makes sense, but is not a qualified name, which k8s requires here. Perhaps something along the lines of v1.multus-cni.io/default-network (similar to k8s.v1.cni.cncf.io) would work.

Below is the error encountered when attempting to use the recommended annotation name:

Error creating pod: Pod "test" is invalid: metadata.annotations: Invalid value: "multus-cni.io/v1/default-network": a qualified name must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]') with an optional DNS subdomain prefix and '/' (e.g. 'example.com/MyName')

[pliurh]

@dougbtv, thanks for reminding. I'll prepare the document in another patch.

[s1061123]

Looks good to me, thank you for your work!