Networking OptionsThis section offers a brief overview of each concept in
networking for Compute.
- In Compute, users organize their cloud resources in
- projects. A Compute project consists of a number of VM
- instances created by a user. For each VM instance, Compute
- assigns to it a private IP address. (Currently, Nova only
- supports Linux bridge networking that allows the virtual
- interfaces to connect to the outside network through the
- physical interface.)
+ In Compute, users organize their cloud resources in projects. A Compute project
+ consists of a number of VM instances created by a user. For each VM instance, Compute
+ assigns to it a private IP address. (Currently, Compute only supports Linux bridge
+ networking that allows the virtual interfaces to connect to the outside network through
+ the physical interface.)The Network Controller provides virtual networks to
enable compute servers to interact with each other and
with the public network.
- Currently, Nova supports three kinds of networks,
- implemented in three “Network Manager” types respectively:
- Flat Network Manager, Flat DHCP Network Manager, and VLAN
- Network Manager. The three kinds of networks can co-exist
- in a cloud system. However, since you can't yet select the
- type of network for a given project, you cannot configure
- more than one type of network in a given Compute
- installation.
- Nova has a concept of Fixed IPs and Floating IPs. Fixed
- IPs are assigned to an instance on creation and stay the
- same until the instance is explicitly terminated. Floating
- IPs are IP addresses that can be dynamically associated
- with an instance. This address can be disassociated and
- associated with another instance at any time. A user can
- reserve a floating IP for their project.
- In Flat Mode, a network administrator specifies a
- subnet. The IP addresses for VM instances are grabbed from
- the subnet, and then injected into the image on launch.
- Each instance receives a fixed IP address from the pool of
- available addresses. A network administrator must
- configure the Linux networking bridge (named br100) both
- on the network controller hosting the network and on the
- cloud controllers hosting the instances. All instances of
- the system are attached to the same bridge, configured
- manually by the network administrator.
+ Currently, Compute supports three kinds of networks, implemented in three “Network
+ Manager” types:
+
+ Flat Network Manager
+
+
+ Flat DHCP Network Manager
+
+
+ VLAN Network Manager
+
+ The three kinds of networks can co-exist in a cloud system. However,
+ since you can't yet select the type of network for a given project, you cannot configure
+ more than one type of network in a given Compute installation.
+ Compute makes a distinction between fixed IPs and
+ floating IPs for VM instances. Fixed IPs are IP
+ addresses that are assigned to an instance on creation and stay the same until the
+ instance is explicitly terminated. By contrast, floating IPs are addresses that can be
+ dynamically associated with an instance. A floating IP address can be disassociated and
+ associated with another instance at any time. A user can reserve a floating IP for their
+ project.
+ In Flat Mode, a network administrator specifies a subnet. The IP addresses for VM
+ instances are grabbed from the subnet, and then injected into the image on launch. Each
+ instance receives a fixed IP address from the pool of available addresses. A network
+ administrator must configure the Linux networking bridge (typically named
+ br100, although this configurable) both on the network controller
+ hosting the network and on the cloud controllers hosting the instances. All instances of
+ the system are attached to the same bridge, configured manually by the network
+ administrator.
- The configuration injection currently only works
- on Linux-style systems that keep networking
- configuration in /etc/network/interfaces.
+ The configuration injection currently only works on Linux-style systems that
+ keep networking configuration in
+ /etc/network/interfaces.In Flat DHCP Mode, you start a DHCP server to pass out
@@ -344,15 +346,12 @@ public_interface=eth0
xml:id="outbound-traffic-flow-with-any-flat-networking">
Outbound Traffic Flow with Any Flat
Networking
- In any set up with FlatNetworking, the host with
- nova-network on it is responsible for forwarding
- traffic from the private network configured with the
- --fixed_range=... directive in
- nova.conf. This host needs to have br100 configured
- and talking to any other nodes that are hosting VMs.
- With either of the Flat Networking options, the
- default gateway for the virtual machines is set to the
- host which is running nova-network.
+ In any set up with FlatNetworking, the host with nova-network on it is
+ responsible for forwarding traffic from the private network configured with the
+ fixed_range=... directive in nova.conf.
+ This host needs to have a bridge interface (e.g., br100) configured and talking to any other
+ nodes that are hosting VMs. With either of the Flat Networking options, the default
+ gateway for the virtual machines is set to the host which is running nova-network. When a virtual machine sends traffic out to the
public networks, it sends it first to its default
gateway, which is where nova-network is configured.
@@ -390,189 +389,209 @@ public_interface=eth0
Configuring VLAN Networking
- In some networking environments, you may have a
- large IP space which is cut up into smaller subnets.
- The smaller subnets are then trunked together at the
- switch level (dividing layer 3 by layer 2) so that all
- machines in the larger IP space can communicate. The
- purpose of this is generally to control the size of
- broadcast domains.
- Using projects as a way to logically separate each
- VLAN, we can setup our cloud in this environment.
- Please note that you must have IP forwarding enabled
- for this network mode to work.
-
- Obtain the parameters for each network. You may need
- to ask a network administrator for this information,
- including netmask, broadcast, gateway, ethernet device
- and VLAN ID.
- You need to have networking hardware that supports
- VLAN tagging.
- Please note that currently eth0 is hardcoded as the
- vlan_interface in the default options. If you need to
- attach your bridges to a device other than eth0, you
- will need to add following option to
- /etc/nova/nova.conf:
-
-
-vlan_interface=eth1
-
- In VLAN mode, the setting for --network_size is the
- number of IPs per project as opposed to the FlatDHCP
- mode where --network_size indicates number of IPs in
- the entire fixed range. For VLAN, the settings in
- nova.conf that affect networking are also
- --fixed_range, where the space is divided up into
- subnets of --network_size.
- VLAN is the default networking mode for Compute, so
- if you have no --network_manager entry in your
- nova.conf file, you are set up for VLAN. To set your
- nova.conf file to VLAN, use this option in
- /etc/nova/nova.conf:
-
-network_manager=nova.network.manager.VlanManager
-
- For the purposes of this example walk-through, we
- will use the following settings. These are
- intentionally complex in an attempt to cover most
- situations:
-
-
-
-
- VLANs: 171, 172, 173 and 174
-
-
- IP Blocks: 10.1.171.0/24, 10.1.172.0/24,
- 10.1.173.0/24 and 10.1.174.0/24
-
-
- Each VLAN maps to its corresponding /24 (171
- = 10.1.171.0/24, etc)
-
-
- Each VLAN will get its own bridge device,
- which is in the format br_$VLANID
-
-
- Each /24 has an upstream default gateway on
- .1
-
+ Compute can be configured so that the virtual machine instances of different
+ projects (tenants) are in different subnets, with each subnet having a different
+ VLAN tag. This can be useful in networking environments where you have a large IP
+ space which is cut up into smaller subnets. The smaller subnets are then trunked
+ together at the switch level (dividing layer 3 by layer 2) so that all machines in
+ the larger IP space can communicate. The purpose of this is generally to control the
+ size of broadcast domains. It can also be useful to provide an additional layer of
+ isolation in a multi-tenant environment.
+
+ The terms network and subnet are often used interchangeably in
+ discussions of VLAN mode. In all cases, we are referring to a range of IP
+ addresses specified by a subnet (e.g.,
+ 172.16.20.0/24) that are on the same VLAN (layer 2
+ network).
+
+ Running in VLAN mode is more complex than the other network modes. In particular:
+
+ IP forwarding must be enabled
+
- The first 6 IPs in each /24 are
- reserved
+ The hosts running nova-network and nova-compute must have the
+ 8021q kernel module loaded
-
- First, create the networks that Compute can pull
- from using nova-manage commands:
-
-$nova-manage --flagfile=/etc/nova/nova.conf network create private 10.1.171.0/24 1 256
-$nova-manage --flagfile=/etc/nova/nova.conf network create private 10.1.172.0/24 1 256
-$nova-manage --flagfile=/etc/nova/nova.conf network create private 10.1.173.0/24 1 256
-$nova-manage --flagfile=/etc/nova/nova.conf network create private 10.1.174.0/24 1 256
-
-
- Log in to the nova database to determine the network
- ID assigned to each VLAN:
-
-$select id,cidr from networks;
-
- Update the DB to match your network settings. The
- following script will generate SQL based on the
- predetermined settings for this example. You
- will need to modify this database update to fit
- your environment.
-
-if [ -z $1 ]; then
- echo "You need to specify the vlan to modify"
-fi
-
-if [ -z $2 ]; then
- echo "You need to specify a network id number (check the DB for the network you want to update)"
-fi
+
+ Your networking switches must support VLAN tagging
+
+
+ Your networking switches must be configured to enable the specific
+ VLAN tags you specify in your Compute setup
+
+
+ You will need information about your networking setup from your
+ network administrator to configure Compute properly (e.g., netmask,
+ broadcast, gateway, ethernet device, VLAN IDs)
+
+
+ To configure your nodes to support VLAN tagging, install the
+ vlan package and load the 8021q kernel
+ module, as
+ root:#apt-get install vlan
+#modprobe 8021q
+ To have this kernel module loaded on boot, add the following line to
+ /etc/modules:8021q
+ Here is an example of settings from /etc/nova/nova.conf for a
+ host configured to run nova-network in VLAN mode
-VLAN=$1
-ID=$2
+ network_manager=nova.network.manager.VlanManager
+vlan_interface=eth0
+fixed_range=172.16.0.0/12
+network_size=256
+The network_manager=nova.network.manager.VlanManager option specifies VLAN
+ mode, which happens to be the default networking mode.
+ The bridges that are created by the network manager will be attached to the
+ interface specified by vlan_interface, the example above uses the
+ eth0 interface, which is the default.
+ The fixed_range option is a CIDR block which describes the IP
+ address space for all of the instances: this space will be divided up into subnets.
+ This range is typically a private
+ network. The example above uses the private range
+ 172.16.0.0/12.
+ The network_size option refers to the default number of IP
+ addresses in each network, although this can be overriden at network creation time .
+ The example above uses a network size of 256, whicih corresponds
+ to a /24 network.
+ Networks are created with the nova-manage network create
+ command. Here is an example of how to create a network consistent with the above
+ example configuration options, as
+ root:#nova-manage network create --label=example-net --fixed_range_v4=172.16.169.0/24 --vlan=169 --bridge=br169 --project_id=a421ae28356b4cc3a25e1429a0b02e98
+ This creates a network called example-net associated with
+ tenant a421ae28356b4cc3a25e1429a0b02e98. The subnet is
+ 172.16.169.0/24 with a VLAN tag of 169
+ (the VLAN tag does not need to match the third byte of the address, though it is a
+ useful convention to remember the association). This will create a bridge interface
+ device called br169 on the host running the nova-network service.
+ This device will appear in the output of an ifconfig
+ command.
+ Each network must be associated with one tenant, specified using the
+ --project_id flag which corresponds to the tenant ID. Use the
+ keystone tenant-list command to list the tenants and
+ corresponding IDs that you have already created.
+ The nova-manage network create command supports many
+ configuration options, which are displayed when called with the
+ --help flag:
+ Usage: nova-manage network create <args> [options]
-cat > vlan.sql << __EOF_
-update networks set vlan = '$VLAN' where id = $ID;
-update networks set bridge = 'br_$VLAN' where id = $ID;
-update networks set gateway = '10.1.$VLAN.7' where id = $ID;
-update networks set dhcp_start = '10.1.$VLAN.8' where id = $ID;
-update fixed_ips set reserved = 1 where address in ('10.1.$VLAN.1','10.1.$VLAN.2','10.1.$VLAN.3','10.1.$VLAN.4','10.1.$VLAN.5','10.1.$VLAN.6','10.1.$VLAN.7');
-__EOF_
+Options:
+ -h, --help show this help message and exit
+ --label=<label> Label for network (ex: public)
+ --fixed_range_v4=<x.x.x.x/yy>
+ IPv4 subnet (ex: 10.0.0.0/8)
+ --num_networks=<number>
+ Number of networks to create
+ --network_size=<number>
+ Number of IPs per network
+ --vlan=<vlan id> vlan id
+ --vpn=VPN_START vpn start
+ --fixed_range_v6=FIXED_RANGE_V6
+ IPv6 subnet (ex: fe80::/64
+ --gateway=GATEWAY gateway
+ --gateway_v6=GATEWAY_V6
+ ipv6 gateway
+ --bridge=<bridge> VIFs on this network are connected to this bridge
+ --bridge_interface=<bridge interface>
+ the bridge is connected to this interface
+ --multi_host=<'T'|'F'>
+ Multi host
+ --dns1=<DNS Address> First DNS
+ --dns2=<DNS Address> Second DNS
+ --uuid=<network uuid>
+ Network UUID
+ --fixed_cidr=<x.x.x.x/yy>
+ IPv4 subnet for fixed IPS (ex: 10.20.0.0/16)
+ --project_id=<project id>
+ Project id
+ --priority=<number> Network interface priority
- After verifying that the above SQL will work for
- your environment, run it against the nova database,
- once for every VLAN you have in the
- environment.
+ In particular, flags to the nova-mange network create command
+ can be used to override settings from nova.conf:
+
+ --network_size
+
+ Overrides the network_size configuration
+ option
+
+
+
+ --bridge_interface
+
+ Overrides the vlan_interface configuration
+ option
+
+
+
+ To view a list of the networks that have been created, as
+ root:#nova-manage network list
+ To modify an existing network, use the nova-manage network
+ modify command, as
+ root:#nova-manage network modify --help
+Usage: nova-manage network modify <args> [options]
- Next, create a project manager for the Compute
- project:
+Options:
+ -h, --help show this help message and exit
+ --fixed_range=<x.x.x.x/yy>
+ Network to modify
+ --project=<project name>
+ Project name to associate
+ --host=<host> Host to associate
+ --disassociate-project
+ Disassociate Network from Project
+ --disassociate-host Disassociate Host from Project
+ To delete a network, use nova-manage network delete, as
+ root:#nova-manage network delete --help
+Usage: nova-manage network delete <args> [options]
-
-$nova-manage --flagfile=/etc/nova/nova.conf user admin $username
-
-
- Then create a project and assign that user as the
- admin user:
-
-
-$nova-manage --flagfile=/etc/nova/nova.conf project create $projectname $username
-
-
- Finally, get the credentials for the user just
- created, which also assigns one of the networks to
- this project:)
-
-
-$nova-manage --flagfile=/etc/nova/nova.conf project zipfile $projectname $username
-
-
- When you start nova-network, the bridge devices and
- associated VLAN tags will be created. When you create
- a new VM you must determine (either manually or
- programatically) which VLAN it should be a part of,
- and start the VM in the corresponding project.
+Options:
+ -h, --help show this help message and exit
+ --fixed_range=<x.x.x.x/yy>
+ Network to delete
+ --uuid=<uuid> UUID of network to delete
+ Note that a network must first be disassociated from a project using the
+ nova-manage network modify command before it can be
+ deleted.
+ Creating a network will automatically cause the Compute database to populate with
+ a list of available fixed IP addresses. You can view the list of fixed IP addresses
+ and their associations with active virtual machines by doing, as
+ root:#nova-manage fix list
+
+
+ Due to Compute bug #754900, deleting a network with the
+ nova-manage network delete command does not delete
+ the associated fixed IP addresses. As a workaround, these fixed IP addresses
+ can be deleted by connecting to the nova database and
+ issuing the SQL query (this example assumes the deleted network id is
+ 1DELETE from fixed_ips where network_id=1;
+
+
- In certain cases, the network manager may not
- properly tear down bridges and VLANs when it is
- stopped. If you attempt to restart the network manager
- and it does not start, check the logs for errors
- indicating that a bridge device already exists. If
- this is the case, you will likely need to tear down
- the bridge and VLAN devices manually.
+ In certain cases, the network manager may not properly tear down bridges and VLANs
+ when it is stopped. If you attempt to restart the network manager and it does not
+ start, check the logs for errors indicating that a bridge device already exists. If
+ this is the case, you will likely need to tear down the bridge and VLAN devices
+ manually. It is also advisable to kill any remaining dnsmasq processes. These
+ commands would stop the service, manually tear down the bridge and VLAN from the
+ previous example, kill any remaining dnsmasq processes, and start the service up
+ again, as root:
-
-$vconfig rem vlanNNN
-$ifconfig br_NNN down
-$brctl delbr br_NNN
-
+ #stop nova-network
+#vconfig rem vlan169
+#ip link set br169 down
+#brctl delbr br169
+#killall dnsmasq
+#start nova-network
- Also, if users need to access the instances in their
- project across a VPN, a special VPN instance (code
- named cloudpipe) needs to be created as described. You
- can create the cloudpipe instance. The image is
- basically just a Linux instance with openvpn
- installed. It needs a simple script to grab user data
- from the metadata server, b64 decode it into a zip
- file, and run the autorun.sh script from inside the
- zip. The autorun script should configure and run
- openvpn to run using the data from Compute.
- For certificate management, it is also useful to
- have a cron script that will periodically download the
- metadata and copy the new Certificate Revocation List
- (CRL). This will keep revoked users from connecting
- and disconnects any users that are connected with
- revoked certificates when their connection is
- re-negotiated (every hour). You set the
- use_project_ca option in
- nova.conf for cloudpipes to
- work securely so that each project has its own
- Certificate Authority (CA).
+ If users need to access the instances in their project across a VPN, a special VPN
+ instance (code named cloudpipe) needs to be created as described in the section
+ titled Cloudpipe — Per Project
+ VPNs.
- Cloudpipe — Per Project Vpns
+ Cloudpipe — Per Project VPNs Cloudpipe is a method for connecting end users to
their project instances in VLAN networking mode.
@@ -600,7 +619,7 @@ __EOF_
Creating a Cloudpipe Image
- Making a cloudpipe image is relatively easy.
+ To make a cloudpipe image: install openvpn on a base ubuntu image.
@@ -706,16 +725,19 @@ force_dhcp_release=True
Certificates and Revocation
- If the use_project_ca config
- option is set (required to for cloudpipes to work
- securely), then each project has its own ca. This
- ca is used to sign the certificate for the vpn,
- and is also passed to the user for bundling
- images. When a certificate is revoked using
- nova-manage, a new Certificate Revocation List
- (crl) is generated. As long as cloudpipe has an
- updated crl, it will block revoked users from
- connecting to the vpn.
+ For certificate management, it is also useful to have a cron script that will
+ periodically download the metadata and copy the new Certificate Revocation List
+ (CRL). This will keep revoked users from connecting and disconnects any users
+ that are connected with revoked certificates when their connection is
+ re-negotiated (every hour). You set the use_project_ca option in nova.conf for
+ cloudpipes to work securely so that each project has its own Certificate
+ Authority (CA).
+ If the use_project_ca config option is set (required to for
+ cloudpipes to work securely), then each project has its own CA. This CA is used
+ to sign the certificate for the vpn, and is also passed to the user for bundling
+ images. When a certificate is revoked using nova-manage, a new Certificate
+ Revocation List (crl) is generated. As long as cloudpipe has an updated crl, it
+ will block revoked users from connecting to the vpn. The userdata for cloudpipe isn't currently
updated when certs are revoked, so it is necessary
to restart the cloudpipe instance if a user's