stackhpc · jovial · Nov 3, 2023 · Apr 25, 2022 · Aug 6, 2022 · Aug 6, 2022
@@ -258,8 +258,6 @@
         NOVA_BACKEND: LVM
         # Do not waste time clearing volumes.
         LVM_VOLUME_CLEAR: none
-        # Disable SSH validation in tests to save time.
-        TEMPEST_RUN_VALIDATION: false
         # Increase the size of the swift loopback device to accommodate RAW
         # snapshots from the LV based instance disks.
         # See bug #1913451 for more details.

@@ -284,7 +284,7 @@ Troubleshooting
 Timeouts
 ~~~~~~~~
 
-Configure a :ref:`service user <user_token_timeout>` in case the user token
+Configure a :ref:`service user <service_user_token>` in case the user token
 times out, e.g. during the snapshot and download of a large server image.
 
 If RPC calls are timing out with a ``MessagingTimeout`` error in the logs,

@@ -19,6 +19,7 @@ A list of config options based on different topics can be found below:
 .. toctree::
    :maxdepth: 1
 
+   /admin/configuration/service-user-token
    /admin/configuration/api
    /admin/configuration/resize
    /admin/configuration/cross-cell-resize

@@ -0,0 +1,59 @@
+.. _service_user_token:
+
+===================
+Service User Tokens
+===================
+
+.. note::
+
+   Configuration of service user tokens is **required** for every Nova service
+   for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for
+   details.
+
+Configure Nova to send service user tokens alongside regular user tokens when
+making REST API calls to other services. The identity service (Keystone) will
+authenticate a request using the service user token if the regular user token
+has expired.
+
+This is important when long-running operations such as live migration or
+snapshot take long enough to exceed the expiry of the user token. Without the
+service token, if a long-running operation exceeds the expiry of the user
+token, post operations such as cleanup after a live migration could fail when
+Nova calls other service APIs like block-storage (Cinder) or networking
+(Neutron).
+
+The service token is also used by services to validate whether the API caller
+is a service. Some service APIs are restricted to service users only.
+
+To set up service tokens, create a ``nova`` service user and ``service`` role
+in the identity service (Keystone) and assign the ``service`` role to the
+``nova`` service user.
+
+Then, configure the :oslo.config:group:`service_user` section of the Nova
+configuration file, for example:
+
+.. code-block:: ini
+
+   [service_user]
+   send_service_user_token = true
+   auth_url = https://104.130.216.102/identity
+   auth_strategy = keystone
+   auth_type = password
+   project_domain_name = Default
+   project_name = service
+   user_domain_name = Default
+   username = nova
+   password = secretservice
+   ...
+
+And configure the other identity options as necessary for the service user,
+much like you would configure nova to work with the image service (Glance) or
+networking service (Neutron).
+
+.. note::
+
+   Please note that the role assigned to the :oslo.config:group:`service_user`
+   needs to be in the configured
+   :oslo.config:option:`keystone_authtoken.service_token_roles` of other
+   services such as block-storage (Cinder), image (Glance), and networking
+   (Neutron).
@@ -320,4 +320,4 @@ To make live-migration succeed, you have several options:
 
 If live migrations routinely timeout or fail during cleanup operations due
 to the user token timing out, consider configuring nova to use
-:ref:`service user tokens <user_token_timeout>`.
+:ref:`service user tokens <service_user_token>`.
@@ -67,7 +67,7 @@ Create a snapshot of the instance
 
    If snapshot operations routinely fail because the user token times out
    while uploading a large disk image, consider configuring nova to use
-   :ref:`service user tokens <user_token_timeout>`.
+   :ref:`service user tokens <service_user_token>`.
 
 #. Use the :command:`openstack image list` command to check the status
    until the status is ``ACTIVE``:

@@ -478,67 +478,3 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
 at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
 while the latter route is considered the legacy v2.0 compatibility-mode
 route which renders all requests as if they were made on the legacy v2.0 API.
-
-
-.. _user_token_timeout:
-
-User token times out during long-running operations
----------------------------------------------------
-
-Problem
-~~~~~~~
-
-Long-running operations such as live migration or snapshot can sometimes
-overrun the expiry of the user token. In such cases, post operations such
-as cleaning up after a live migration can fail when the nova-compute service
-needs to cleanup resources in other services, such as in the block-storage
-(cinder) or networking (neutron) services.
-
-For example:
-
-.. code-block:: console
-
-  2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
-  2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
-  2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
-  2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
-  2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
-
-Solution
-~~~~~~~~
-
-Configure nova to use service user tokens to supplement the regular user token
-used to initiate the operation. The identity service (keystone) will then
-authenticate a request using the service user token if the user token has
-already expired.
-
-To use, create a service user in the identity service similar as you would when
-creating the ``nova`` service user.
-
-Then configure the :oslo.config:group:`service_user` section of the nova
-configuration file, for example:
-
-.. code-block:: ini
-
-  [service_user]
-  send_service_user_token = True
-  auth_type = password
-  project_domain_name = Default
-  project_name = service
-  user_domain_name = Default
-  password = secretservice
-  username = nova
-  auth_url = https://104.130.216.102/identity
-  ...
-
-And configure the other identity options as necessary for the service user,
-much like you would configure nova to work with the image service (glance)
-or networking service.
-
-.. note::
-
-  Please note that the role of the :oslo.config:group:`service_user` you
-  configure needs to be a superset of
-  :oslo.config:option:`keystone_authtoken.service_token_roles` (The option
-  :oslo.config:option:`keystone_authtoken.service_token_roles` is configured
-  in cinder, glance and neutron).
@@ -197,7 +197,7 @@ Using fake computes for tests
 The number of instances supported by fake computes is not limited by physical
 constraints. It allows you to perform stress tests on a deployment with few
 resources (typically a laptop). Take care to avoid using scheduler filters
-that will limit the number of instances per compute, such as ``AggregateCoreFilter``.
+that will limit the number of instances per compute, such as ``NumInstancesFilter``.
 
 Fake computes can also be used in multi hypervisor-type deployments in order to
 take advantage of fake and "real" computes during tests:

@@ -92,6 +92,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

@@ -84,6 +84,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

@@ -74,6 +74,26 @@ Install and configure components
         Comment out or remove any other options in the
         ``[keystone_authtoken]`` section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

@@ -260,6 +260,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
      management interface IP address of the controller node:
 

@@ -247,6 +247,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
      management interface IP address of the controller node:
 

@@ -237,6 +237,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
      management interface IP address of the controller node:
 

@@ -93,7 +93,14 @@ def _remove_tenant_access(self, req, id, body):
 
         vals = body['removeTenantAccess']
         tenant = vals['tenant']
-        identity.verify_project_id(context, tenant)
+        # It doesn't really matter if project exists or not: we can delete
+        # it from flavor's access list in both cases.
+        try:
+            identity.verify_project_id(context, tenant)
+        except webob.exc.HTTPBadRequest as identity_exc:
+            msg = "Project ID %s is not a valid project." % tenant
+            if msg not in identity_exc.explanation:
+                raise
 
         # NOTE(gibi): We have to load a flavor from the db here as
         # flavor.remove_access() will try to emit a notification and that needs

@@ -56,6 +56,9 @@ def get_vnc_console(self, req, id, body):
             raise webob.exc.HTTPNotFound(explanation=e.format_message())
         except exception.InstanceNotReady as e:
             raise webob.exc.HTTPConflict(explanation=e.format_message())
+        except exception.InstanceInvalidState as e:
+            common.raise_http_conflict_for_instance_invalid_state(
+                e, 'get_vnc_console', id)
         except NotImplementedError:
             common.raise_feature_not_supported()
 

@@ -48,13 +48,10 @@ def __init__(self):
         self.actions = {"enable": self._enable,
                         "disable": self._disable,
                         "disable-log-reason": self._disable_log_reason}
-        self._placementclient = None  # Lazy-load on first access.
 
     @property
     def placementclient(self):
-        if self._placementclient is None:
-            self._placementclient = report.SchedulerReportClient()
-        return self._placementclient
+        return report.report_client_singleton()
 
     def _get_services(self, req):
         # The API services are filtered out since they are not RPC services
@@ -328,7 +325,7 @@ def delete(self, req, id):
                             "Failed to delete compute node resource provider "
                             "for compute node %s: %s",
                             compute_node.uuid, str(e))
-                # remove the host_mapping of this host.
+                # Remove the host_mapping of this host.
                 try:
                     hm = objects.HostMapping.get_by_host(context, service.host)
                     hm.destroy()