Multitenancy Support using vlanid #156

tamilmani1989 · 2018-05-30T02:11:02Z

What this PR does / why we need it:
This PR adds multitenancy support in both cnm and cni. The user has the option to pass vlanid. If vlanid is specified, then OVS network will be setup and the packets coming out of container will be tagged with corresponding vlan id. In case of CNM, the vlanid can be specified as option in create network request. CNI will get vlanid after querying CNS for podname and namespace.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note:

…working into multitenancy

…working into multitenancy # Conflicts: # cns/dnccontract.go

…iner-networking into multitenancy # Conflicts: # cns/dnccontract.go # cns/restserver/restserver.go

Added getnetworkcontainerbyorchestrator context api and relevant test cases

…e and namespace

…working into multitenancy # Conflicts: # cns/dnccontract.go # cns/restserver/restserver.go

…iner-networking into multitenancy # Conflicts: # client/cnsclient/cnsclient.go # cni/network/network.go

…working into multitenancy # Conflicts: # client/cnsclient/cnsclient.go # cni/network/network.go # cns/service/main.go

saiyan86 · 2018-05-31T00:14:24Z

cni/network/network.go

+	if ipAddr.To4() != nil {
+		resultIpconfig.Version = "4"
+		resultIpconfig.Address = net.IPNet{IP: ipAddr, Mask: net.CIDRMask(int(ipconfig.IPSubnet.PrefixLength), 32)}
+	} else {


saiyan86 · 2018-05-31T00:17:57Z

cni/network/network.go

+			gwIP := net.ParseIP(route.GatewayIPAddress)
+			result.Routes = append(result.Routes, &types.Route{Dst: *routeIPnet, GW: gwIP})
+		}
+	} else {


We can return route here instead of having else.

saiyan86 · 2018-05-31T00:18:25Z

cni/network/network.go

+	} else {
+		gwIP := net.ParseIP(networkConfig.IPConfiguration.GatewayIPAddress)
+		dstIP := net.IPNet{IP: net.ParseIP("0.0.0.0"), Mask: resultIpconfig.Address.Mask}
+		result.Routes = append(result.Routes, &types.Route{Dst: dstIP, GW: gwIP})


What happens if result.Routes==nil?

saiyan86 · 2018-06-01T17:48:04Z

cni/network/network.go

+	return convertToCniResult(networkConfig), networkConfig.MultiTenancyInfo.ID, *subnetPrefix, nil
+}
+
+func getPodNameWithoutSuffix(podName string) string {


Suggestion:

func getPodNameWithoutSuffix(podName string) string { nameSplit := strings.Split(podName, "-") if len(nameSplit) < 2 { return podName } return strings.Join(nameSplit[:2], "-") }

…working into ovsabstract # Conflicts: # cni/network/network.go # cns/service/main.go # network/endpoint.go # network/endpoint_linux.go

…working into ovsabstract

…container-networking into ovsabstract

sharmasushant · 2018-06-22T23:31:12Z

client/cnsclient/cnsclient.go

+		return nil, err
+	}
+
+	defer res.Body.Close()


Check if we need to close in case we get err!=nil

sharmasushant · 2018-06-22T23:32:37Z

client/cnsclient/cnsclient.go

@@ -0,0 +1,85 @@
+package cnsclient


move it under cns package

sharmasushant · 2018-06-22T23:40:36Z

cni/azure-linux.conflist

         "type":"azure-vnet",
         "mode":"bridge",
         "bridge":"azure0",
+         "multiTenancy":true,


Create a new conflist for this

sharmasushant · 2018-06-22T23:43:48Z

cni/azure-linux.conflist

         "type":"azure-vnet",
         "mode":"bridge",
         "bridge":"azure0",
+         "multiTenancy":true,


add enablesnat: true

sharmasushant · 2018-06-22T23:45:55Z

cni/network/network.go

 	// Plugin name.
-	name = "azure-vnet"
+	name                = "azure-vnet"
+	namespaceKey        = "K8S_POD_NAMESPACE"


remove it as it is not used

sharmasushant · 2018-06-23T00:38:57Z

cns/restserver/restserver.go

+}
+
+func (service *httpRestService) getNetworkContainerResponse(req cns.GetNetworkContainerRequest) cns.GetNetworkContainerResponse {
+	var containerID string


log at beginning and end

this is internal function

sharmasushant · 2018-06-23T00:39:33Z

cns/restserver/restserver.go

+}
+
+func (service *httpRestService) getNetworkContainerByOrchestratorContext(w http.ResponseWriter, r *http.Request) {
+	log.Printf("[Azure CNS] getNetworkContainer")


sharmasushant · 2018-06-23T00:45:59Z

cns/service/main.go

+	{
+		Name:         acn.OptStopAzureVnet,
+		Shorthand:    acn.OptStopAzureVnetAlias,
+		Description:  "Start Azure-CNM if flag is true",


fix description or rename flag

sharmasushant · 2018-06-23T00:50:30Z

common/utils.go

 	return err
 }
+
+func ExecuteShellCommand(command string) (string, error) {


platform specific

sharmasushant · 2018-06-23T00:56:05Z

network/api.go


-	OptVethName = "vethname"
+	OptVethName         = "vethname"
+	OptEnableSnatOnHost = "enableSnatOnHost"


…s set to true without reboot

sharmasushant · 2018-07-02T21:37:40Z

cni/network/network.go

+}
+
+func getContainerNetworkConfiguration(namespace string, podName string) (*cniTypesCurr.Result, *cns.GetNetworkContainerResponse, net.IPNet, error) {
+	cnsClient, err := cnsclient.NewCnsClient("")


Lets leave it upto the caller to give CNS address.. ideally, it should come from cni config (when CNI is used)

sharmasushant · 2018-07-02T21:40:36Z

cni/network/network.go

+	podInfo := cns.KubernetesPodInfo{PodName: podName, PodNamespace: namespace}
+	orchestratorContext, err := json.Marshal(podInfo)
+	if err != nil {
+		log.Printf("Marshalling azure container instance info failed with %v", err)


"Marshalling azure container instance info failed with %v" --> "Marshalling cns.KubernetesPodInfo failed with %v"

sharmasushant · 2018-07-02T21:41:13Z

cni/network/network.go

+
+	networkConfig, err := cnsClient.GetNetworkConfiguration(orchestratorContext)
+	if err != nil {
+		log.Printf("GetNetworkConfiguration failed with %v", err)


Lets print %v orchestrator context to help debug

sharmasushant · 2018-07-02T21:44:04Z

cni/network/network.go

+		return podName
+	}
+
+	log.Printf("Final namesplit %v", nameSplit)


"Pod name after splitting based on - : %v"

sharmasushant · 2018-07-02T21:46:19Z

cni/network/network.go

 			Name: args.IfName,
 		}
+
+		snatIface := &cniTypesCurr.Interface{


Move it out to a linux specific method

sharmasushant · 2018-07-02T23:39:17Z

network/ovs_networkclient_linux.go

+	}
+
+	if client.enableSnatOnHost {
+


remove empty line

sharmasushant · 2018-07-02T23:41:57Z

network/ovs_networkclient_linux.go

+
+	_, err := net.InterfaceByName(internetBridgeName) 
+	if err == nil {
+		log.Printf("Internet Bridge already exists")


Replace internet with SNAT everywhere...

sharmasushant · 2018-07-02T23:45:20Z

network/ovs_networkclient_linux.go

+}
+
+func (client *OVSNetworkClient) AddBridgeRules(extIf *externalInterface) error {
+	//primary := extIf.IPAddresses[0].IP.String()


remove dead code

sharmasushant · 2018-07-02T23:47:06Z

network/ovs_networkclient_linux.go

+	return nil
+}
+
+func (client *OVSNetworkClient) AddBridgeRules(extIf *externalInterface) error {


sharmasushant · 2018-07-02T23:51:33Z

ovsctl/ovsctl.go

+)
+
+const (
+	macAddress = "12:34:56:78:9a:bc"


defaultMacForArpResponse

jaer-tsun · 2018-07-02T21:24:14Z

cni/network/network_linux.go

+		epInfo.Data[network.VlanIDKey] = cnsNwConfig.MultiTenancyInfo.ID
+		epInfo.Data[network.LocalIPKey] = cnsNwConfig.LocalIPConfiguration.IPSubnet.IPAddress + "/" + strconv.Itoa(int(cnsNwConfig.LocalIPConfiguration.IPSubnet.PrefixLength))
+		epInfo.Data[network.InternetBridgeIPKey] = cnsNwConfig.LocalIPConfiguration.GatewayIPAddress + "/" + strconv.Itoa(int(cnsNwConfig.LocalIPConfiguration.IPSubnet.PrefixLength))
+


remove space

jaer-tsun · 2018-07-02T21:31:31Z

cns/NetworkContainerContract.go

 	PrimaryInterfaceIdentifier string // Primary CA.
 	AuthorizationToken         string
-	OrchestratorInfo           OrchestratorInfo
+	LocalIPConfiguration	   IPConfiguration


What's this? Is this from your dnc change?

i changed this both in vendor and in my branch

jaer-tsun · 2018-07-02T21:38:54Z

cns/cnsclient/cnsclient.go

+
+	defer res.Body.Close()
+
+	if res.StatusCode != 200 {


http.StatusOK instead of 200

jaer-tsun · 2018-07-02T21:52:12Z

cns/restserver/restserver.go

+	getNetworkContainerResponse = cns.GetNetworkContainerResponse{
+		IPConfiguration:            savedReq.IPConfiguration,
+		Routes:                     savedReq.Routes,
+		CnetAddressSpace:			savedReq.CnetAddressSpace,


fix spacing

jaer-tsun · 2018-07-02T22:25:07Z

cns/service/main.go


 	// Create the key value store.
-	pluginConfig.Store, err = store.NewJsonFileStore(platform.CNMRuntimePath + pluginName + ".json")
+


remove space

jaer-tsun · 2018-07-03T15:15:46Z

network/ovs_networkclient_linux.go

+}
+
+func createInternetBridge(internetBridgeIP string, mainInterface string) error {
+


remove space

jaer-tsun · 2018-07-03T15:18:30Z

network/ovs_networkclient_linux.go

+
+	ip, addr, _ := net.ParseCIDR(internetBridgeIP)
+
+	log.Printf("Assigning %v on internet bridge", internetBridgeIP)


move ParseCIDR below log.Printf

jaer-tsun · 2018-07-03T15:20:39Z

network/ovs_networkclient_linux.go

+		log.Printf("[net] Failed to delete bridge %v, err:%v.", internetBridgeName, err)
+	}
+
+	return nil


return err here?

jaer-tsun · 2018-07-03T15:21:05Z

network/ovs_networkclient_linux.go

+	mac := extIf.MacAddress.String()
+	macHex := strings.Replace(mac, ":", "", -1)
+
+	/*if err := ovsctl.AddVMIpAcceptRule(client.bridgeName, primary, mac); err != nil {


remove commented section if not necessary

jaer-tsun · 2018-07-03T15:22:59Z

ovsctl/ovsctl.go

+
+	ofport = strings.Trim(ofport, "\n")
+
+	return ofport, nil


return strings.Trim(ofport, "\n"), nil

pjohnst5 · 2024-03-07T18:10:55Z

cni/network/mutlitenancy.go

+	return result
+}
+
+func getPodNameWithoutSuffix(podName string) string {


What was the use case of this feature?

tamilmani1989 and others added 30 commits March 4, 2018 19:50

Temporary commit. Added vlan support

fb257f8

checkpoint

3cd7300

fixes

0c05120

Added code for cleaning up ovs rules on deleting endpoint

ef6d539

Merge branch 'master' of https://github.com/Azure/azure-container-net…

035462d

…working into multitenancy

Merge branch 'master' of https://github.com/Azure/azure-container-net…

72f34ff

…working into multitenancy

Merge branch 'master' of https://github.com/Azure/azure-container-net…

7a7ba69

…working into multitenancy

Added code to get network config

09122b0

Added cnsclient

9bea2e8

Merge branch 'master' of https://github.com/Azure/azure-container-net…

149ce20

…working into multitenancy # Conflicts: # cns/dnccontract.go

Added setorchestrator api

eecc9ec

Fixed styling

9a21055

Merge branch 'master' of https://github.com/tamilmani1989/azure-conta…

d433550

…iner-networking into multitenancy # Conflicts: # cns/dnccontract.go # cns/restserver/restserver.go

Modified networkcontainercreate request to save podinfo (#115)

f7ab7ea

Added GetNetworkContainerByContext Api (#116)

40b87bc

Added getnetworkcontainerbyorchestrator context api and relevant test cases

Added testcase to check if state is cleaneup after delete call

1a439cb

Resued existing errorcode

0df120d

Added code in CNI to retrieve network config from CNS based on podnam…

70e8074

…e and namespace

Added option in cns to not to start cnm by default.

46fc36f

Fixed style and spacing

37f8114

Merge branch 'devaci' of https://github.com/Azure/azure-container-net…

7f730fc

…working into multitenancy # Conflicts: # cns/dnccontract.go # cns/restserver/restserver.go

Merge branch 'master' of https://github.com/tamilmani1989/azure-conta…

b4eaf38

…iner-networking into multitenancy # Conflicts: # client/cnsclient/cnsclient.go # cni/network/network.go

fixed merge issues

765cac9

CNI gets networkconfig from CNS (#117)

00522ae

Merge branch 'devaci' of https://github.com/Azure/azure-container-net…

bba9c4a

…working into multitenancy # Conflicts: # client/cnsclient/cnsclient.go # cni/network/network.go # cns/service/main.go

Fixed issues and tested cnm and cni

3858cec

Added comments

adde6f2

Fixed hardcoded values

679dc94

removed unnecessary checks

fad692c

check for network container type (#121)

82486a8

Fixed review comments and added rule to drop packet if already tagged

323c49f

saiyan86 suggested changes Jun 1, 2018

View reviewed changes

tamilmani1989 and others added 18 commits June 8, 2018 12:27

Merge branch 'master' of https://github.com/Azure/azure-container-net…

5b39198

…working into ovsabstract # Conflicts: # cni/network/network.go # cns/service/main.go # network/endpoint.go # network/endpoint_linux.go

fixed merge conflict bugs

82c0df4

jaeryn changes in cns

e7f0136

Missing changes from devaci branch.

08fdbdc

fixed route issue

b6682bd

Merge branch 'master' of https://github.com/Azure/azure-container-net…

3499ad5

…working into ovsabstract

updated ovs config to clean ovs setup on bootup

385d9c7

Merge branch 'ovsabstract' of https://github.com/tamilmani1989/azure-…

2106013

…container-networking into ovsabstract

fixed the hardcoded part

ed58e92

Added multiwrite option for windows logger

c76a859

fixed getting podname without suffix

8313b04

Merge branch 'ovsabstract' of https://github.com/tamilmani1989/azure-…

9d32e49

…container-networking into ovsabstract

added logs

1151913

Added snat support for outbound traffic

e2ad9fd

Added cnetspace routes

eaec5d9

snat changes

fa0fd46

snat changes integrated with cns

5d190d9

moved linux specific code to appropriate files

265c285

sharmasushant suggested changes Jun 27, 2018

View reviewed changes

tamilmani1989 added 2 commits June 27, 2018 18:32

addressed review comments and fixed issues

11aeb20

handled failure case and create internet bridge if enablesnatonhost i…

bab14a5

…s set to true without reboot

sharmasushant suggested changes Jul 3, 2018

View reviewed changes

jaer-tsun reviewed Jul 3, 2018

View reviewed changes

tamilmani1989 added 2 commits July 3, 2018 17:23

addressed review comments and tested for regression

99770ae

fixed review comments

d1f405e

sharmasushant approved these changes Jul 6, 2018

View reviewed changes

sharmasushant merged commit 147c438 into Azure:master Jul 6, 2018

pjohnst5 reviewed Mar 7, 2024

View reviewed changes


		// Create the key value store.
		pluginConfig.Store, err = store.NewJsonFileStore(platform.CNMRuntimePath + pluginName + ".json")

		}

		func createInternetBridge(internetBridgeIP string, mainInterface string) error {


		ip, addr, _ := net.ParseCIDR(internetBridgeIP)

		log.Printf("Assigning %v on internet bridge", internetBridgeIP)

Multitenancy Support using vlanid #156

Multitenancy Support using vlanid #156

Uh oh!

Conversation

tamilmani1989 commented May 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!