7.0.0[bc]: octavia (ovn) does not find existing subnet

[garloff]

Testing on a CiaB system with 7.0.0b and openstack-health-monitor:

New JHSUBNET: aeefbfec-fb5c-4b91-8fee-6cd99562f4bb
New SUBNET: dcf3b761-b3c8-44b7-b9d7-26b1186c1762  33e70699-87ae-4b21-8907-772497a8d2a9
[...]
New LBAAS:  on ciab/APIMonitor_1709379467/85 on deb12: 
[...]
ERROR: openstack loadbalancer create --vip-subnet-id aeefbfec-fb5c-4b91-8fee-6cd99562f4bb --name APIMonitor_1709379467_LB_0 --provider ovn => 1 Subnet aeefbfec-fb5c-4b91-8fee-6cd99562f4bb not found. (HTTP 400) (Request-ID: req-0dd11ee4-6ae2-44c7-9540-d069712d3e49)
ERROR: LBAAS creation failed

Clearly the subnet exists and belong to the project.
Somehow, the OVN loadbalancer does not seem to accept tenant subnets. Which it obviously should. And did before.

[garloff]

This might be specific to the CiaB setup, where amphorae are disabled, but maybe the removal -octavia_network_type: tenant was too much to achieve this?

[garloff]

The upgrade to 7.0.0c did not help.

OK, apparently, octavia does not accept the TLS certiificate from neutron:

2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base     raise exceptions.SSLError(msg)
2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/01b868dc-92be-4e77-8daa-71cabc423347: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/01b868dc-92be-4e77-8daa-71cabc423347 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base

[garloff]

This is really strange. Inside the octavia_api container, the cert validation with openssl s_client succeeds:

echo | openssl s_client -connect api.in-a-box.cloud:9696  2>/dev/null
CONNECTED(00000003)                                                            
---                                                                            
Certificate chain                                                              
 0 s:                                                                          
   i:CN = OSISM Cloud-in-a-Box CA                                              
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                       
   v:NotBefore: Nov 20 15:21:35 2022 GMT; NotAfter: Nov 17 16:21:35 2032 GMT   
 1 s:CN = OSISM Cloud-in-a-Box CA                                              
   i:CN = OSISM Cloud-in-a-Box CA                                              
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                       
   v:NotBefore: Nov 20 16:21:20 2022 GMT; NotAfter: Nov 17 16:21:20 2032 GMT   
---                                                                            
[...]
subject=
issuer=CN = OSISM Cloud-in-a-Box CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3424 bytes and written 400 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
[...]

The /etc/octavia/octavia.conf file mentions ca_certificates_file = /etc/ssl/certs/ca-certificates.crt numerous times. That file exists and does have the trusted CA cert for in-a-box.cloud in there.
So for whatever reason does octavia fail to consider it when talking to neutron?
But why?

[neutron]
region_name = RegionOne
endpoint_type = internal
ca_certificates_file = /etc/ssl/certs/ca-certificates.crt

[garloff]

Still puzzled.
octavia-api.log gives me:

2024-03-08 19:06:23.133 734 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.                                                                                                                                                                                                                                         
2024-03-08 19:06:35.522 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-03-08 19:06:35.523 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling  service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-03-08 19:06:35.523 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
2024-03-08 19:06:35.561 733 ERROR octavia.network.drivers.neutron.base [None req-e9128b28-dc8a-4fbfa8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Error retrieving subnet (subnet id: 98bb455b-387f-4d7c-a1e3-bdaf50b382ad.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

So I was reading through https://docs.openstack.org/octavia/latest/configuration/configref.html#neutron
to also understand the warning.
Adding a few cafile = /etc/ssl/.... settings and valid_interfaces = internal settings and restarting the octavia_api container does not change anything. (I can see in the config dump that the variables are read ... but the error messages don't go away.)

2024-03-08 19:10:20.284 733 INFO octavia.api.app [-] neutron.ca_certificates_file   = /etc/ssl/certs/ca-certificates.crt
2024-03-08 19:10:20.284 733 INFO octavia.api.app [-] neutron.cafile                 = /etc/ssl/certs/ca-certificates.crt
2024-03-08 19:10:20.286 733 INFO octavia.api.app [-] neutron.valid_interfaces       = ['internal']

Nova on the other hand ignores cafile and valid_interfaces. Looks like we are in an evil intermittent phase of a change?
The errors still occur:

2024-03-08 19:10:20.913 734 ERROR octavia.network.drivers.neutron.base [None req-d2aa63e8-b05c-469c-9bbe-e629a0921b4e - 26c88a620f834253ba7dc478af6ce12d - - default default] Error retrieving subnet (subnet id: 98bb455b-387f-4d7c-a1e3-bdaf50b382ad.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))     

[garloff]

And

[neutron]
insecure = True

in /etc/kolla/octavia-api/octavia.conf does not help either. This is getting more and more obscure.

[garloff]

So, when octavia-api talks to neutron, it does not seem to have its own configuration in mind.
Except when I use endpoint_override = https://api.in-a-box.cloud:9696.
Then it suddenly stops throwing those SSL errors.
The error with the missing subnets remains :-(
Similarly, setting an endpoint = https://api.in-a-box.cloud:8774/v2.1 for nova in the octavia.conf does make the nova, cinder, and glance (!!!) warnings go away. This is definitely strange behavior.

[garloff]

According to other testers of v7.0.0[bc], this does not affect full installations, so maybe the issue is limited to CiaB.
(I have not looked at testbed yet.)

[garloff]

Hi Christian,
OK, so we reenable the octavia amphora provider in CiaB deployments.
I'd still like to have a working octavia ovn provider, obviously.
Maybe, as a side-effect, we'll get this as well?

[garloff]

I upgraded to 7.0.0d.
The not found subnet still happens, with both ovn and amphora provider.
Also the disturbing log messages remain. (I can still fix them with cafile and endpoint_override, but that does not resolve the issue with not-found subnets.)

[garloff]

Upgrade to 7.0.0 did not make a difference: Still no access to the existing subnet (neither ovn nor amphora):

dragon@manager:/opt/configuration/environments/openstack$ openstack subnet list
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
| ID                                   | Name                              | Network                              | Subnet          |
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
| 98bb455b-387f-4d7c-a1e3-bdaf50b382ad | test-subnet                       | 7c7d9e93-5770-48c5-b29d-25d338d2831b | 192.168.64.0/24 |
| f9243dd5-59a0-4964-9d69-fefaa04820c7 | APIMonitor_1710528148_SUBNET_VM_0 | 14bba778-c773-4319-b797-2e44ba49cdf0 | 10.250.0.0/22   |
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
dragon@manager:/opt/configuration/environments/openstack$ openstack loadbalancer create --vip-subnet-id 98bb455b-387f-4d7c-a1e3-bdaf50b382ad --name testlb --provider ovn
Subnet 98bb455b-387f-4d7c-a1e3-bdaf50b382ad not found. (HTTP 400) (Request-ID: req-bd00a62c-dbcc-465d-b06b-ab1018431e50)
dragon@manager:/opt/configuration/environments/openstack$ openstack loadbalancer create --vip-subnet-id 98bb455b-387f-4d7c-a1e3-bdaf50b382ad --name testlb --provider amphora
Subnet 98bb455b-387f-4d7c-a1e3-bdaf50b382ad not found. (HTTP 400) (Request-ID: req-e7ff6ada-ddbd-411a-8a7d-da66d388b81b)
dragon@manager:/opt/configuration/environments/openstack$ 

Same SSL exceptions in /var/log/kolla/octavia/octavia-api.log.

[berendt]

Test of latest images including the bugfix:

dragon@testbed-manager:~$ openstack --os-cloud test loadbalancer create --provider ovn --vip-subnet-id subnet-test --name test
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2024-03-25T12:02:07                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 03f7df5f-4ba5-4eaf-b0f0-44e119ea1107 |
| listeners           |                                      |
| name                | test                                 |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 2b8f799708a44710855f5b6e1bdc99ed     |
| provider            | ovn                                  |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.200.251                      |
| vip_network_id      | 19eaf344-e5e7-42db-a1c8-b66aba552182 |
| vip_port_id         | a3ccaab9-53f7-476e-8314-54b51ef7da85 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 19d0fd77-da96-42f3-b4c9-b5b4ed42a9fa |
| tags                |                                      |
| additional_vips     | []                                   |
+---------------------+--------------------------------------+

dragon@testbed-manager:~$ openstack --os-cloud test loadbalancer list
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+
| id                                   | name | project_id                       | vip_address     | provisioning_status | operating_status | provider |
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+
| 03f7df5f-4ba5-4eaf-b0f0-44e119ea1107 | test | 2b8f799708a44710855f5b6e1bdc99ed | 192.168.200.251 | ACTIVE              | ONLINE           | ovn      |
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+

[berendt]

Included in 7.0.1. Also tested in the cloud-in-a-box, works there as well.

dragon@manager:/opt/configuration/environments/openstack$ openstack --os-cloud test loadbalancer create --provider ovn --vip-subnet-id subnet-test --name test
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2024-03-28T09:45:10                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 8c37ba47-9a23-4496-ab37-611c9f15f901 |
| listeners           |                                      |
| name                | test                                 |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | b31385a573fc44e489c8afb6109ee865     |
| provider            | ovn                                  |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.200.221                      |
| vip_network_id      | b8684359-35ca-4b08-9e3d-0001fd69bb78 |
| vip_port_id         | c33275fe-e2dd-4719-bce3-6f6b6ff189cb |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | e51bec68-5d9e-4179-b301-69ab7c247ff4 |
| vip_vnic_type       | normal                               |
| tags                |                                      |
| additional_vips     | []                                   |
+---------------------+--------------------------------------+
dragon@manager:/opt/configuration/environments/openstack$ openstack --os-cloud test loadbalancer list
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+
| id                        | name | project_id                | vip_address     | provisioning_status | operating_status | provider |
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+
| 8c37ba47-9a23-4496-ab37-  | test | b31385a573fc44e489c8afb61 | 192.168.200.221 | ACTIVE              | ONLINE           | ovn      |
| 611c9f15f901              |      | 09ee865                   |                 |                     |                  |          |
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+