An IP Address Management (IPAM) CNI plugin that assigns IP addresses cluster-wide.
If you need a way to assign IP addresses dynamically across your cluster -- Whereabouts is the tool for you. If you've found that you like how the host-local CNI plugin works, but, you need something that works across all the nodes in your cluster (host-local only knows how to assign IPs to pods on the same node) -- Whereabouts is just what you're looking for.
Whereabouts can be used for both IPv4 & IPv6 addressing.
This version of whereabouts is an experimental attempt to integrate with openwisp-ipam so that IP address will be allocated from a central place. In this way whereabouts talk to a central openwisp-ipam/phpipam server to get the IP for the multus interface making the administration easy. openwisp-ipam and phpipam supports API which will be used by whereabouts to allocate/deallocate IP. for more info on openwisp-ipam, refer to https://github.com/openwisp/openwisp-ipam for more info on phpipam refer to https://phpipam.net/
Many thanks to all the contributors of the main whereabouts cni project.
To be added- config needed in openwisp
PHPIPAM: Once phpipam is configured for non tls use, create an appid for API usage and then create subnet section in it like below for test. The description for each pool is used as pool name which is referenced va network attachment definition.
APPID config for API enablement
Pool subnet config for whereabouts usage
For now if the annotation podname: "" is added to a deployment, then the annotation value is used as the reference in the ipam database. so if for some reason the pod moves to a different node and starts up, it will still get same IP if a IPAM delete was not sent for the failed pod. In this way IP doesnt change for pod migration due to node failure etc.
Example pod definition
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: test
name: test
spec:
replicas: 1
selector:
matchLabels:
app: test
strategy: {}
template:
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: whereabouts-conf
podname: test234 <----unique name for this deployment across the whole IPAM subnet
creationTimestamp: null
labels:
app: test
spec:
containers:
- image: nginx
imagePullPolicy: IfNotPresent
name: nginx
resources: {}
Pool definition which supports pool name definition using the CNI Args convention
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: whereabouts-conf
spec:
config: '{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"args": {"pool": "pool1"}, <--- pool name to be configured in IPAM provider from which IP is allocated, description in phpipam for a subnet
"type": "macvlan",
"master": "ens38",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"range": "192.168.2.225/28", <--- Ignored when ipam provider is used, the above pool name will be used instead
"log_file": "/tmp/whereabouts.log",
"log_level": "debug"
}
}'
CNI (Container Network Interface) plugins typically have a configuration element named ipam. CNI IPAM plugins can assign IP addresses, and Whereabouts assigns IP addresses within a range -- without having to use a DHCP server.
Whereabouts takes an address range, like 192.168.2.0/24 in CIDR notation, and will assign IP addresses within that range. In this case, it will assign IP addresses from 192.168.2.1 to 192.168.2.255. When an IP address is assigned to a pod, Whereabouts tracks that IP address in a data store for the lifetime of that pod. When the pod is removed, Whereabouts then frees the address and makes it available to assign on subsequent requests. Whereabouts always assigns the lowest value address that's available in the range.
You can also specify ranges to exclude from assignment, so if for example you'd like to assign IP addresses within the range 192.168.2.0/24, you can exclude IP addresses within it by adding them to an exclude list. For example, if you decide to exclude the range 192.168.2.0/28, the first IP address assigned in the range will be 192.168.2.16.
In respect to the old equipment out there that doesn't think that IP addresses that end in .0 are valid -- Whereabouts will not assign addresses that end in .0.
The original inspiration for Whereabouts comes from when users have tried to use the samples from Multus CNI (a CNI plugin that attaches multiple network interfaces to your pods), which includes examples that use the host-local plugin, and they find that it's... Almost the right thing. Sometimes people even assume it'll work across nodes -- and then wind up with IP address collisions.
Whereabouts is designed with Kubernetes in mind, but, isn't limited to use in just Kubernetes.
To track which IP addresses are in use between nodes, Whereabouts uses etcd or a Kubernetes Custom Resource as a backend. The goal is to make Whereabouts more flexible and to use additional storage backends, we welcome any contributions towards this goal.
Issues and PRs are welcome! Some of the known limitations are found at the bottom of the README.
There's two steps to installing Whereabouts:
- Installing Whereabouts itself (it's just a binary on disk).
- Creating IPAM CNI configurations.
Further installation options (including etcd usage) and configuration parameters can be found in the extended configuration document.
You can install this plugin with a Daemonset, using:
git clone https://github.com/dougbtv/whereabouts && cd whereabouts
kubectl apply -f ./doc/daemonset-install.yaml -f ./doc/whereabouts.cni.cncf.io_ippools.yaml
The daemonset installation requires Kubernetes Version 1.16 or later.
You can also install multus and whereabouts with helm 3 (helm 2 is not supported)
git clone https://github.com/k8snetworkplumbingwg/helm-charts.git
cd helm-charts
helm upgrade --install multus ./multus --namespace kube-system
helm upgrade --install whereabouts ./whereabouts --namespace kube-system
Helm will install the crd as well as the daemonset
Included here is an entire CNI configuration. Whereabouts only cares about the ipam section of the CNI config. In particular this example uses the macvlan CNI plugin. (If you decide to copy this block and try it too, make sure that the master setting is set to a network interface name that exists on your nodes). Typically, you'll already have a CNI configuration for an existing CNI plugin in your cluster, and you'll just copy the ipam section and modify the values there.
{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"range": "192.168.2.225/28",
"exclude": [
"192.168.2.229/30",
"192.168.2.236/32"
]
}
}
Whereabouts is particularly useful in scenarios where you're using additional network interfaces for Kubernetes. A NetworkAttachmentDefinition custom resource can be used with a CNI meta plugin such as Multus CNI to attach multiple interfaces to your pods in Kubernetes.
In short, a NetworkAttachmentDefinition contains a CNI configuration packaged into a custom resource. Here's an example of a NetworkAttachmentDefinition containing a CNI configuration which uses Whereabouts for IPAM:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: whereabouts-conf
spec:
config: '{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"range": "192.168.2.225/28"
}
}'
The same applies for the usage of IPv6:
{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"range": "2001::0/116",
"gateway": "2001::f:1"
}
}
Required
These parameters are required:
type: This should be set towhereabouts.range: This specifies the range in which IP addresses will be allocated.
If for example the range is set to 192.168.2.225/28, this will allocate IP addresses in the range excluding the first network address and the last broadcast address.
If you need a tool to figure out the range of a given CIDR address, try this online tool, subnet-calculator.com or an IPv6 subnet calculator.
Range end syntax
Additionally, the range parameter can support a CIDR notation that includes the last IP to use. Example: range: "192.168.2.225-192.168.2.230/28".
Optional
The following parameters are optional:
range_start: First IP to use when allocating from therange. Optional, if unset is inferred from therange.range_end: Last IP to use when allocating from therange. Optional, if unset the last ip within the range is determined.exclude: This is a list of CIDRs to be excluded from being allocated.
In the example, we exclude IP addresses in the range 192.168.2.229/30 from being allocated (in this case it's 3 addresses, .229, .230, .231), as well as 192.168.2.236/32 (just a single address).
Note: It's up to you to properly set exclusion ranges that are within your subnet, there's no double checking for you (other than that the CIDR notation parses).
Additionally -- you can set the route, gateway and DNS using anything from the configurations for the static IPAM plugin (as well as additional static IP addresses).
Run the build command from the ./hack directory:
./hack/build-go.sh
Thanks big time to Tomofumi Hayashi, I utilized his static CNI IPAM plugin as a basis for this project to give me a head start!
The typeface used in the logo is AZONIX, by MixoFX.
- If you specify overlapping ranges -- you're almost certain to have collisions, so if you specify one config with
192.168.0.0/16and another with192.168.0.0/24, you'll have collisions.- This could be fixed with an admission controller.
- And admission controller could also prevent you from starting a pod in a given range if you were out of addresses within that range.
- There's probably a lot of comparison of IP addresses that could be optimized, lots of string conversion.
- The etcd method has a number of limitations, in that it uses an all ASCII methodology. If this was binary, it could probably store more and have more efficient IP address comparison.
- Unlikely to work in Canada, apparently it would have to be "where aboots?" for Canadians to be able to operate it.