Dealing with the current Floating IP Behavior

[josephineSei]

The current Floating Ip behavior has two downsides:

1. When adding a floating IP to a VM there is no check, whether the floating IP is already associated to another VM, this makes it possible to "steal" the floating IP from another VM
2. When removing a floating IP from a VM there is no check, whether the correct IP was specified, because the the given server name or id is not checked.

There are many ways to deal with this behavior:

1. Talk to upstream about the whole situation, that in both cases, the association is not checked
2. Talk to upstream about the CLI command, that may need to be adjusted to better reflect the workflow (especially for the removing part)
3. Talk to upstream about the documentation of floating ip usage
4. Maybe have a guide / information about this behavior in the scs docs

[josephineSei]

Detailed Information

Going a little bit back in time in the OpenStack project, Networking was also done by Nova. It was eventually taken out and brought to a new project: Neutron (first Name: Quantum) but remnants remained in Nova. So it was for quite some time thath Nova and Neutron shared the responsibility for creating and attaching a Floating IP to the correct port.
Not too long ago that was also excluded from Nova and implemented in Neutron alone.

And that is a problem.

expected behavior

1. using openstack floating ip create gives me a (randomly assigned) IP from the pool, that I can freely use in my project.
2. using openstack server add floating ip attaches a free floating IP to the given VM
3. using openstack server remove floating ip detaches the given floating IP from the given server.
4. using openstack floating ip delete "deletes" an IP == i can and will no use this IP anymore

These steps explain what a user is expecting to do with floating ips, even though the create and delete may be a bit confusing.

This was also the behavior, when Nova was still responsible for Floating IPs

current behavior

1. using openstack floating ip create allocates an IP from the provider networks pool for the project.

   • this is basically technical implementation of the expected behavior

2. using openstack server add floating ip attaches a ANY floating IP allocated for the project to the given VM

   • this leads to the problem, that within the same project a floating IP can be stolen from a VM and attached to a new VM leaving the first VM unconnected.

$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID                                   | Name            | Status | Networks                                   | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server     | ACTIVE | client-internal=192.168.10.79              |       | S      |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107, 10.54.8.58 |       | S      |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 08600d00-cd66-48e4-9e69-9e6a5c932ef1 | 10.54.8.206         | None             | None                                 | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| 4a89a886-a47b-46d3-95cd-fb38820f56b3 | 10.54.8.171         | None             | None                                 | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| d8e827d0-4446-4217-9f3a-d30d10c74653 | 10.54.8.220         | None             | None                                 | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| f282badb-a1b6-4269-9783-5df9ce2fccc9 | 10.54.8.58          | 192.168.10.107   | 85921f34-8821-409c-80e8-81093e1e805d | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
$ openstack server add floating ip test-server 10.54.8.58
$ openstack server list
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+
| ID                                   | Name            | Status | Networks                                  | Image | Flavor |
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server     | ACTIVE | client-internal=192.168.10.79, 10.54.8.58 |       | S      |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107            |       | S      |
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+

Due to Neutron taking over the complete process the check whether a floating IP is already associated to a VM was skipped. This leaves us with this "stealing" behavior.

3. using openstack server remove floating ip detaches the given floating IP from the given server it was associated with
   • That means there is no check, whether the IP was really associated to the specified server
   • if there is a mismatch for the VM and floating IP pair, maybe due to a mistake from a user, the floating IP is detach from the wrong server

$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID                                   | Name            | Status | Networks                                   | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server     | ACTIVE | client-internal=192.168.10.79, 10.54.8.206 |       | S      |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107, 10.54.8.58 |       | S      |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack server remove floating ip test-server 10.54.8.58
$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID                                   | Name            | Status | Networks                                   | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server     | ACTIVE | client-internal=192.168.10.79, 10.54.8.206 |       | S      |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107             |       | S      |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack server remove floating ip this-server-does-not-exist 10.54.8.206
$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+
| ID                                   | Name            | Status | Networks                       | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server     | ACTIVE | client-internal=192.168.10.79  |       | S      |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107 |       | S      |
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+

A combination from removing a floating Ip from a VM (step 3) and deleting the floating IP from the project (step 4) would lead in the worst case to a DOS attack on a VM.
Because after deleting a floating IP, it cannot be ensured for a normal user to get the same IP address
- first because only admins can specify a certain ip address when allocating a floating IP for a project.
- second because when the IP is already taken, even admins cannot do anything

$ source openrc admin admin
$ openstack floating ip list
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| ID                                 | Floating IP Address | Fixed IP Address | Port                               | Floating Network                   | Project                          |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| 1875754d-7b9f-47c2-9c0d-           | 172.24.4.210        | 10.0.0.45        | d8387e3b-3b19-444a-9983-           | 73edb86b-d7ab-4db3-82b7-           | 15f2ab0eaa5b4372b759bde609e86224 |
| 83eafd1a0a76                       |                     |                  | 42b61b3d19c1                       | 25fa8b012e40                       |                                  |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
$ openstack floating ip create --floating-ip-address 172.24.4.222 public
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2024-03-13T08:48:54Z                 |
| description         |                                      |
| dns_domain          |                                      |
| dns_name            |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 172.24.4.222                         |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id                  | 3978a1f6-3af8-432f-978a-c7feafd88057 |
| name                | 172.24.4.222                         |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | 15f2ab0eaa5b4372b759bde609e86224     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2024-03-13T08:48:54Z                 |
+---------------------+--------------------------------------+
$ openstack floating ip list
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| ID                                 | Floating IP Address | Fixed IP Address | Port                               | Floating Network                   | Project                          |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| 1875754d-7b9f-47c2-9c0d-           | 172.24.4.210        | 10.0.0.45        | d8387e3b-3b19-444a-9983-           | 73edb86b-d7ab-4db3-82b7-           | 15f2ab0eaa5b4372b759bde609e86224 |
| 83eafd1a0a76                       |                     |                  | 42b61b3d19c1                       | 25fa8b012e40                       |                                  |
| 3978a1f6-3af8-432f-978a-           | 172.24.4.222        | None             | None                               | 73edb86b-d7ab-4db3-82b7-           | 15f2ab0eaa5b4372b759bde609e86224 |
| c7feafd88057                       |                     |                  |                                    | 25fa8b012e40                       |                                  |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
$ source openrc demo demo
$ openstack floating ip list

$ openstack floating ip create --floating-ip-address 172.24.4.222 public
Error while executing command: ForbiddenException: 403, (rule:create_floatingip and rule:create_floatingip:floating_ip_address) is disallowed by policy
$ openstack floating ip create public
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2024-03-13T08:49:51Z                 |
| description         |                                      |
| dns_domain          |                                      |
| dns_name            |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 172.24.4.155                         |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id                  | 0f340eb1-74c7-4cc0-8495-8f648ff7bc61 |
| name                | 172.24.4.155                         |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | f58edaee60ad484facd2436d31d9caff     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2024-03-13T08:49:51Z                 |
+---------------------+--------------------------------------+
$ source openrc admin admin
$ openstack floating ip create --floating-ip-address 172.24.4.155 public
Error while executing command: ConflictException: 409, IP address 172.24.4.155 already allocated in subnet 3e0206bc-53c8-44ca-a0f1-2c2548bba766

4. using openstack floating ip delete "deletes" an IP == i can and will no use this IP anymore
   • this works as expected
   • it should be noted, that the case above still is valid:
     • only admins can allocate a certain IP to a project
     • they can only do this, if it is not already allocated for another project.

[josephineSei]

I looked a bit through nova code, wrote a gist and put this whole topic on the PTG plan for Neutron.
I think it should be discussed there how to handle this whole issue.

[josephineSei]

After discussing this with Neutron I was asked to file 2 Bugs. One for each workflow:

• server add floating ip -> using already taken floating ip
• server remove floating ip -> does not check the server at all

Including detailed Requests from the CLI. So I am currently reproducing both cases and will describe them with the whole debug output.

After that I should ask the Nova people to look over this and they should decide (from Neutrons perspective) how to proceed, and which behavior they would like.

[josephineSei]

Here are the new bug reports:
https://bugs.launchpad.net/neutron/+bug/2060808
https://bugs.launchpad.net/nova/+bug/2060812

[josephineSei]

After looking through the reported bugs:

1. the association bug is confirmed for nova, which hopefully means, that they would like to change it.
2. the other bug has to be discussed.

There were no discussion about these bugs in the last weeks team meetings.
The IRC Nova meeting is held on Tuesday 16 UTC - unfortunately I will not be able to attend this, but I will try to reach out to the Nova team to discuss the "new" behavior.

[josephineSei]

I attended the Nova IRC meeting and asked about the bugs: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
They wanted to have some time looking into it and a more detailed discussion about this. So I will try to contact them in their IRC channel at a different time (UTC afternoon would work)