Insufficient external IP address declared prematurely

[askfongjojo]

The issue happens to an IP pool on rack2 that contains several discrete ranges:

$ oxide ip-pool range list --pool oxide-alt-pool
[
  {
    "id": "c3f126d5-cb46-466d-879d-b8237b0a3a26",
    "ip_pool_id": "8b924114-a9f4-4634-b6a4-956ff640807a",
    "range": {
      "first": "192.168.30.2",
      "last": "192.168.30.3"
    },
    "time_created": "2023-08-31T05:28:29.740339Z"
  }, {
    "id": "e9dcb77b-4850-4c6f-a4e2-dd75af40ebd7",
    "ip_pool_id": "8b924114-a9f4-4634-b6a4-956ff640807a",
    "range": {
      "first": "192.168.30.4",
      "last": "192.168.30.5"
    },
    "time_created": "2023-08-31T05:33:07.484888Z"
  }, {
    "id": "5217b1d9-15b9-4b19-b717-7445b066c57f",
    "ip_pool_id": "8b924114-a9f4-4634-b6a4-956ff640807a",
    "range": {
      "first": "192.168.30.6",
      "last": "192.168.30.7"
    },
    "time_created": "2025-11-05T05:48:47.384507Z"
  }, {
    "id": "58c40668-7c9d-416d-aa02-967ef525e7d8",
    "ip_pool_id": "8b924114-a9f4-4634-b6a4-956ff640807a",
    "range": {
      "first": "192.168.30.101",
      "last": "192.168.30.254"
    },
    "time_created": "2025-03-29T04:48:57.264797Z"
  }, {
    "id": "a231278f-3e9e-4c74-ade3-0fa6cbe92adc",
    "ip_pool_id": "8b924114-a9f4-4634-b6a4-956ff640807a",
    "range": {
      "first": "192.168.40.5",
      "last": "192.168.40.10"
    },
    "time_created": "2025-11-05T04:55:01.647901Z"
  }
]

$ oxide floating-ip create --pool oxide-alt-pool --project angela --name fileserver --description "fileserver fixed ip"
error
Error Response: status: 507 Insufficient Storage; headers: {"content-type": "application/json", "x-request-id": "4606727f-1e5f-40cf-8151-34038fee1d0f", "content-length": "172", "date": "Wed, 05 Nov 2025 05:59:09 GMT"}; value: Error { error_code: Some("InsufficientCapacity"), message: "Insufficient capacity: No external IP addresses available", request_id: "4606727f-1e5f-40cf-8151-34038fee1d0f" }

I know the IP pool isn't quite fully utilized so I queried the database table directly to see what is consuming these IPs but only saw 16 in use:

root@[fd00:1122:3344:105::3]:32221/omicron> select ip, first_port, last_port, kind, ip_pool_range_id from external_ip where time_deleted is null and ip_pool_id = '8b924114-a9f4-4634-b6a4-956ff640807a' order by 1,2,3;
        ip       | first_port | last_port |   kind    |           ip_pool_range_id
-----------------+------------+-----------+-----------+---------------------------------------
  192.168.30.2   |          0 |     65535 | floating  | c3f126d5-cb46-466d-879d-b8237b0a3a26
  192.168.30.3   |          0 |     65535 | ephemeral | c3f126d5-cb46-466d-879d-b8237b0a3a26
  192.168.30.4   |          0 |     65535 | floating  | e9dcb77b-4850-4c6f-a4e2-dd75af40ebd7
  192.168.30.5   |          0 |     65535 | floating  | e9dcb77b-4850-4c6f-a4e2-dd75af40ebd7
  192.168.30.101 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.102 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.103 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.104 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.105 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.109 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.110 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.111 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.114 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.116 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.119 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
  192.168.30.120 |          0 |     65535 | ephemeral | 58c40668-7c9d-416d-aa02-967ef525e7d8
(16 rows)

[askfongjojo]

The pool has the correct reservation_type and pool_type:

root@[fd00:1122:3344:105::3]:32221/omicron> select * from ip_pool where name in ('oxide-alt-pool', 'default');
                   id                  |      name      |     description      |         time_created          |         time_modified         | time_deleted | rcgen | ip_version | reservation_type | pool_type
---------------------------------------+----------------+----------------------+-------------------------------+-------------------------------+--------------+-------+------------+------------------+------------
  8b924114-a9f4-4634-b6a4-956ff640807a | oxide-alt-pool | oxide silo-only pool | 2023-08-31 05:15:27.63075+00  | 2025-06-28 04:31:07.877315+00 | NULL         |     8 | v4         | external_silos   | unicast
  cadb8535-e32f-4be2-9259-e45dec9fa3c7 | default        | default IP pool      | 2023-08-30 18:57:14.079765+00 | 2023-08-30 18:57:14.079765+00 | NULL         |     8 | v4         | external_silos   | unicast

[askfongjojo]

I tried another smaller IP pool and noticed the allocation seems to use only the first IP range in the pool. Pasting the error from the console for easier visualization:

[jgallagher]

Definitely need @bnaecker to double-check this, but I think the issue is in this subquery added in #9282. If I take out the LIMIT 1, the all_candidates subquery on this pool returns:

            ip_pool_range_id           |  candidate_ip  | candidate_first_port | candidate_last_port
---------------------------------------+----------------+----------------------+----------------------
  c3f126d5-cb46-466d-879d-b8237b0a3a26 | NULL           |                    0 |               65535
  e9dcb77b-4850-4c6f-a4e2-dd75af40ebd7 | NULL           |                    0 |               65535
  5217b1d9-15b9-4b19-b717-7445b066c57f | 192.168.30.6   |                    0 |               65535
  58c40668-7c9d-416d-aa02-967ef525e7d8 | 192.168.30.106 |                    0 |               65535
  a231278f-3e9e-4c74-ade3-0fa6cbe92adc | 192.168.40.5   |                    0 |               65535
(5 rows)

which looks right: the first two ranges in this pool are exhausted, and there are available IPs in the later three ranges. But the LIMIT 1 causes us to only look at the first range, which is exhausted:

            ip_pool_range_id           | candidate_ip | candidate_first_port | candidate_last_port
---------------------------------------+--------------+----------------------+----------------------
  c3f126d5-cb46-466d-879d-b8237b0a3a26 | NULL         |                    0 |               65535
(1 row)

therefore leading to "no available IPs". If I add a WHERE candidate_ip IS NOT NULL , we skip the first two ranges and should (?) return the next available IP:

            ip_pool_range_id           | candidate_ip | candidate_first_port | candidate_last_port
---------------------------------------+--------------+----------------------+----------------------
  5217b1d9-15b9-4b19-b717-7445b066c57f | 192.168.30.6 |                    0 |               65535
(1 row)

[david-crespo]

Claude agrees: