Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow specifying on which interface the Azure IPAM should allocate IPs on #10875

Merged
merged 1 commit into from Apr 9, 2020
Merged

Allow specifying on which interface the Azure IPAM should allocate IPs on #10875

merged 1 commit into from Apr 9, 2020

Conversation

ungureanuvladvictor
Copy link
Member

Signed-off-by: Vlad Ungureanu vladu@palantir.com

Add an azure field to the CiliumNode which should hold Azure specific IPAM settings. Starting with interface-name which should be used to override on which interface on the VM the Azure IPAM will allocate IPs. Currently it walks the interfaces in order and picks where it has space. This is useful in situations where the interfaces are in different subnets.

Doing it the same way as the AWS IPAM is configured -- passing the interface name via the cni config which the agent can read when it registers the CiliumNode in the k8s apiserver.

Allow specifying on which interface the Azure IPAM should allocate IPs on

@ungureanuvladvictor ungureanuvladvictor added release-note/minor This PR changes functionality that users may find relevant to operating Cilium. area/operator Impacts the cilium-operator component integration/cloud Related to integration with cloud environments such as AKS, EKS, GKE, etc. sig/ipam IP address management, including cloud IPAM labels Apr 7, 2020
@ungureanuvladvictor ungureanuvladvictor requested review from a team as code owners April 7, 2020 14:17
@ungureanuvladvictor ungureanuvladvictor requested a review from a team April 7, 2020 14:17
@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.8.0 Apr 7, 2020
@ungureanuvladvictor
Copy link
Member Author

I still need to do some testing of this -- will comment out when that is done so people can review it.

ungureanuvladvictor
ungureanuvladvictor commented Apr 7, 2020
View reviewed changes
pkg/k8s/apis/cilium.io/v2/register.go
@@ -398,6 +398,10 @@ func createNodeCRD(clientset apiextensionsclient.Interface) error {
Type: "string",
Description: "instance-id is the Azure specific identifier of the node",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this instance-id field being here is outdated since it has been pulled out in the main spec part via 948440d.

Is there any thing we can do here related to validation to keep the CRD validation in-line with the actual structs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. I didn't find anything in k8s upstream either besides code review.

@coveralls
Copy link

coveralls commented Apr 7, 2020 •
edited

Coverage Status

Coverage decreased (-0.01%) to 45.889% when pulling c2d4376 on pr/vladu/azure-netconf into 5d35686 on master.

@ungureanuvladvictor
Copy link
Member Author

test-me-please

@ungureanuvladvictor
Allow specifying on which interface the Azure IPAM should allocate IP…
c2d4376 
…s on

Signed-off-by: Vlad Ungureanu <vladu@palantir.com>
@ungureanuvladvictor
Copy link
Member Author

test-me-please

@ungureanuvladvictor
Copy link
Member Author

I deployed it to my Azure stack and was able to get it to allocate on the specified interface. I tested with the following cni config:

{
  "cniVersion":"0.3.1",
  "name":"cilium",
  "plugins": [
    {
      "cniVersion":"0.3.1",
      "type":"cilium-cni",
      "ipam": {
        "pre-allocate": 1,
        "min-allocate": 13
      },
      "azure": {
          "interface-name": "eth1"
      }
    }
  ]
}

and the final CiliumNode after allocation looks like:

apiVersion: cilium.io/v2
kind: CiliumNode
metadata:
  creationTimestamp: "2020-04-08T03:31:47Z"
  generation: 4
  managedFields:
  - apiVersion: cilium.io/v2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences: {}
      f:spec:
        .: {}
        f:addresses: {}
        f:azure:
          .: {}
          f:interface-name: {}
        f:encryption: {}
        f:eni: {}
        f:health: {}
        f:instance-id: {}
        f:ipam:
          .: {}
          f:min-allocate: {}
          f:podCIDRs: {}
          f:pre-allocate: {}
      f:status:
        f:ipam:
          f:used:
            .: {}
            f:10.0.2.10:
              .: {}
              f:owner: {}
              f:resource: {}
            f:10.0.2.35:
              .: {}
              f:owner: {}
              f:resource: {}
            f:10.0.2.42:
              .: {}
              f:owner: {}
              f:resource: {}
    manager: cilium-agent
    operation: Update
    time: "2020-04-08T03:36:52Z"
  - apiVersion: cilium.io/v2
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:ipam:
          f:pool:
            .: {}
            f:10.0.2.9:
              .: {}
              f:resource: {}
            f:10.0.2.10:
              .: {}
              f:resource: {}
            f:10.0.2.32:
              .: {}
              f:resource: {}
            f:10.0.2.33:
              .: {}
              f:resource: {}
            f:10.0.2.34:
              .: {}
              f:resource: {}
            f:10.0.2.35:
              .: {}
              f:resource: {}
            f:10.0.2.36:
              .: {}
              f:resource: {}
            f:10.0.2.37:
              .: {}
              f:resource: {}
            f:10.0.2.38:
              .: {}
              f:resource: {}
            f:10.0.2.39:
              .: {}
              f:resource: {}
            f:10.0.2.40:
              .: {}
              f:resource: {}
            f:10.0.2.41:
              .: {}
              f:resource: {}
            f:10.0.2.42:
              .: {}
              f:resource: {}
            f:10.0.2.43:
              .: {}
              f:resource: {}
      f:status:
        .: {}
        f:azure:
          .: {}
          f:interfaces: {}
        f:eni: {}
        f:ipam: {}
    manager: cilium-operator
    operation: Update
    time: "2020-04-08T03:50:41Z"
  name: vault000003
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: vault000003
    uid: 7437e49a-a9d2-4252-a2f3-8bd97a97ae27
  resourceVersion: "7169754"
  selfLink: /apis/cilium.io/v2/ciliumnodes/vault000003
  uid: 852d44d0-6424-4d89-aa80-b375a5d25a00
spec:
  addresses:
  - ip: 10.0.2.9
    type: InternalIP
  - ip: 10.0.2.10
    type: CiliumInternalIP
  azure:
    interface-name: eth1
  encryption: {}
  eni: {}
  health: {}
  instance-id: /subscriptions/$REDACTED/resourcegroups/$REDACTED/providers/microsoft.compute/virtualmachinescalesets/vault/virtualmachines/3
  ipam:
    min-allocate: 13
    podCIDRs:
    - 10.9.0.0/16
    pool:
      10.0.2.9:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth0
      10.0.2.10:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.32:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.33:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.34:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.35:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.36:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.37:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.38:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.39:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.40:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.41:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.42:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.43:
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
    pre-allocate: 1
status:
  azure:
    interfaces:
    - addresses:
      - ip: 10.0.2.10
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.32
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.33
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.34
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.35
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.36
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.37
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.38
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.39
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.40
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.41
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.42
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      - ip: 10.0.2.43
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      id: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      mac: 00:0D:3A:1C:19:96
      name: eth1
    - addresses:
      - ip: 10.0.2.9
        state: succeeded
        subnet: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Network/virtualNetworks/main/subnets/security
      id: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth0
      mac: 00:0D:3A:1C:15:52
      name: eth0
  eni: {}
  ipam:
    used:
      10.0.2.10:
        owner: router
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.35:
        owner: default/pod-a
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1
      10.0.2.42:
        owner: default/pod-b
        resource: /subscriptions/$REDACTED/resourceGroups/$REDACTED/providers/Microsoft.Compute/virtualMachineScaleSets/vault/virtualMachines/3/networkInterfaces/eth1

tgraf
tgraf approved these changes Apr 9, 2020
View reviewed changes
pkg/k8s/apis/cilium.io/v2/register.go
@@ -398,6 +398,10 @@ func createNodeCRD(clientset apiextensionsclient.Interface) error {
Type: "string",
Description: "instance-id is the Azure specific identifier of the node",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. I didn't find anything in k8s upstream either besides code review.

@tgraf tgraf merged commit 070baff into master Apr 9, 2020
1.8.0 automation moved this from In progress to Merged Apr 9, 2020
@tgraf tgraf deleted the pr/vladu/azure-netconf branch April 9, 2020 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tgraf tgraf tgraf approved these changes

Assignees
No one assigned
Labels
area/operator Impacts the cilium-operator component integration/cloud Related to integration with cloud environments such as AKS, EKS, GKE, etc. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. sig/ipam IP address management, including cloud IPAM
Projects
No open projects
1.8.0
  
Merged
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ungureanuvladvictor @coveralls @tgraf