[EPIC] IaaS standards

[mbuechse]

General

Standard for Standards & Documentation

• Procedural improvement: no stabilization w/o test script #502
• Github Issue template for new Standard document #501
• [BUG] Standards table in docs page reports versions a being both stable and deprecated #446
• Draft 'Implementation and Testing notes' supplementary documents for select standards #437
• Automatically check front matter of standards documents via a Zuul job #385
• Extend implementation notes by findings from #426 #559
• Add (redundant) color coding to standards table in the docs page #445
• Differentiation between Decision Records (DR) and Standards #352

Mandatory Openstack Services

• [Feature Request] Define a list of mandatory and of optional (supported) SCS OpenStack services issues#528

Computing

Flavor naming, flavor selection, and flavor discoverability:

• SCS IaaS Standard flavors and Flavor naming discussion #271
• Split flavor naming and mandatory flavors in two separate standards  #267
• Flavor specs: Add extra_specs standardization #74
• maybe Supporting flavor aliases #228
• maybe Dynamic SCS flavor selection issues#287
• Extend Flavor Manager YAML into detailed, structured flavor descriptions issues#353
• Streamline command-line syntax of flavor-name-check.py #454
• Equip openstack-flavor-manager with the ability to set extra_specs #380
• [BUG] Extension syntax in flavor naming standard misleading #363
• flavor-manager-input.py uses strings instead of real types for default values #340
• CI tests for mandatory-flavor-list (#319) #339
• Revise flavor naming standard according to review criticism #327
• You may offer additional SCS- flavors ... #295
• Split flavor naming and mandatory flavors in two separate standards  #267
• Mention placement etc. in implementation notes on scs-0100-v3 #555
• Extend check for scs-0100-v3 to cover more properties #554
• [Standardization] GPU naming convention needs further refinements #366

Standard flavors

• [BUG] Misleading error messages in standard flavors check #504

OpenStack powered Compute 2022.11

• https://opendev.org/openinfra/interop/src/branch/master/guidelines/2022.11.json
• Refstack in Ref.Impl. but not generic
• Implement test script for Openinfra Interop Guidelines #300

Storage

Volume types

• Volume Type Standard and Discoverability #468

Network

public network

• IPv4: Need to standardize how public IPv4 networking behaves by default issues#167
• IPv6 networking is implemented, documented but not standardized (IPv6 support for IaaS issues#166)

Network Time Protocoll

• IaaS DR/Standard on NTP service issues#231

DNS

• IaaS DR/Standard for tenant VM DNS  issues#229
• IaaS DR/Standard for transparency on upstream DNS issues#230

L3 loadbalancer (OVN)

• Needed for good support of externalTrafficPolicy: Local
• Research: Implement/Use L3 load-balancing in OpenStack issues#251
• Research: Health Monitor for LB with OVN provider issues#268
• What would CSPs and Users would want to have standardized here?

Neutron Policy Standard

• Neutron Policies Standard #543

Images

Image Meta Data

• [BUG] Standard for image metadata omits unit for "min_disk_size" and "min_ram"  #367
• No longer recommend image signatures #328
• Standardize image names and source URLs #326
• [Feature Request] Add license to Image Metadata as mandatory attribute #369

Standard Images

• [BUG] namescheme for standard image "ubuntu-capi" wrong #537
• SCS Standard Images reference YAML not clearly communicated #534

Identity

Domain admin role: Allow project creation, user management as self-service (resellers)

• Domain Admin rights for SCS IaaS Customers issues#184

Identity federation via OIDC: Federate users from federated clouds

• https://scs.community/de/2023/01/05/sig-iam-openstack-cli-with-federation/
• https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0300-v1-requirements-for-sso-identity-federation.md

Security

Baseline security

Database(s)

• regulation regarding OpenStack database(s)
• Secure communication standard for IaaS infastructure #547

Entropy in VMs

• IaaS ADR on providing virtio-rng entropy source issues#234
• Write tests to check standards for entropy in VMs #268
• [BUG] SCS-compatible IaaS tests fail due to entropy test #456
• Entropy test sets up and tears down its network topology #317

Key Store

• Standardize List of allowed Barbican Plugins #509

Encryption

• Upstream Image Encryption Standardization #560

Security Groups

• Set of widely used security Groups in OpenStack #473
• Add standard for default security group rules #521

Backup and Redundancy

Taxonomy of Backups

• Taxonomy of backup / redundancy / failsafe levels #527

User Backup

• IaaS standard on user data backup #541

Definition of Availability Zones: Availability expectations when spreading over AZs

• Idea: Meaningful level of independence (power, net, fire, cooling, …)
• https://github.com/SovereignCloudStack/standards/blob/main/Drafts/SCS-Spec.md#availability-zones
• Availability Zones: standardized levels of independecies. #539

Definition of Region: What is shared?

• Idea: Share identities, replicate images

Unsorted/unclassified

Metadata source (w/ user-data, vendor-data)

• Required for customization of VMs

MetaData API

• Metadefs API #565

SCS compatibility assessment tool

• [Feature Request] Develop SCS compatibility assessment tool #313

Openinfra Interop Guidelines

• Implement test script for Openinfra Interop Guidelines #300

[fkr]

Discussion on how to best proceed with these standards / item that could be standardized in todays meeting with @josephineSei , @markus-hentsch and @mbuechse

We agreed to

0. Follow-up on the flavor/properties/traits discussion that happened as part of the Caracal vPTG - this topic is with @fkr
   1a) Work on the topics of DNS and NTP standards, this will be started in the next weeks in Team IaaS
   1b) Find out which items / features / functionality should be standardized for the loadbalancing

[josephineSei]

Somehow editing the description of this issue does not show me the correct version. So maybe one of you can include this:
Standardized list of OpenStack services: #469

[josephineSei]

I opened an issue for the Security Groups: #473 .

[markus-hentsch]

I discussed with @josephineSei which topics might not be covered yet by SCS standardization appropriately yet and what kind of potential might exist regarding those.

Key rotation

We thought about the possibility of creating a standard or guide for cryptographic key rotation in SCS and had a look at involved components.
We deemed the following topics relevant for key rotation.

Cinder Volumes:

• encryption uses Barbican keys, no rotation currently supported
• key rotation would require a new feature contribution to upstream
• rotation could work generating new secret in Barbican and swapping keys in the LUKS key slots
  • swapping key in LUKS slot is quick and atomic operation
• however, simply swapping keys in the LUKS slots is weak since copies of the old LUKS header (just 2MB) and the old secret can still unlock the current LUKS master key even if the key slot was replaced!
  • the effort of implementing this upstream would not be justified by the relatively low security gain
• alternative would be to offer online reencryption using LUKS v2¹
  • would not be an atomic operation anymore, can become error prone in a lot of ways
  • needs complex handling of the progress and state of the potentially lengthy reencryption process

Ephemeral Storage (Nova):

• encryption uses Barbican keys, no rotation currently supported
• key rotation would require a new feature contribution to upstream
• only LVM backend currently suppports encryption²
• more generic support is still WIP³ and not finished upstream
• uses the same LUKS encryption as Cinder volumes, hence the same cryptographic limitations apply, see the Cinder section above

Images (Glance):

• image encryption upstream contribution still WIP currently⁴
• encryption uses Barbican keys, no rotation currently supported
• encryption is not LUKS-based, key rotation could only work by reencrypting the full image file
  • could be done manually using API or client by downloading, reencrypting and reuploading the image
  • upstream implementation in Glance would be costly both in terms of required processing time and disk space

Keystone token provider:

• the default backend (Fernet) has a well-known rotation implementation using secondary and staged keys, the keystone-manage utility provides a rotation mechanism⁵
• upstream documentation could be referenced but is pretty sparse regarding the specific workflow of rotating and distributing the keys correctly
• SCS could document a better guide
• JWS rotation is easy, they're just individual keypairs per Keystone instance and well documented⁶

PKI / TLS

• for all API interfaces and backend channels (e.g. RabbitMQ, DB) that can be protected by TLS, a key rotation and certificate exchange can be established
• this is usually part of the PKI and seems out of scope for a key rotation standard or guide

---

Summary: LUKS encryption key rotation (concerns Nova and Cinder) would require an upstream contribution and would either be weak cryptographically (when changing only key slots) or require a lot of effort to get right (online reencryption). Glance image key rotation could be established using guidelines and a manual process but the implementation is not ready yet.
The only thing immediately actionable here would be creating a better guide for Keystone Fernet key rotation.

Backups

We briefly discussed the potential necessity of some form of backup guide for user data due to the following considerations:

• from a user's POV the backup mechanisms offered by OpenStack might be either insufficient or lacking in terms of clear and concise documentation
• Glance images can be downloaded
• Cinder volumes
  • there's openstack volume backup create but that requires Swift (who uses Swift?)
  • openstack server image create when used on a volume-based server will create volume snapshots but snapshots aren't genuine backups! (misleading?)
  • alternative: image from volume using imtermediate snapshots, known approach but poorly documented
• for Ephemeral Storage there's also openstack server image create which creates Glance images as backup

... so the state seems pretty messy. server image create does different things in a non-obvious way depending on whether Ephemeral Storage or volumes are used. volume backup create does not create genuine backups strictly speaking.

We see potential here to at least formulate a guide to better assist users seeking to implement a backup concept.

Footnotes

1. https://fossies.org/linux/cryptsetup/docs/v2.2.0-ReleaseNotes ↩

2. https://docs.openstack.org/security-guide/tenant-data/data-encryption.html#ephemeral-disk-encryption ↩

3. https://review.opendev.org/q/message:ephemeral-storage-encryption+status:open ↩

4. https://review.opendev.org/q/topic:%22image-encryption%22 ↩

5. https://docs.openstack.org/keystone/2023.2/admin/fernet-token-faq.html#what-are-the-different-types-of-keys ↩

6. https://docs.openstack.org/keystone/latest/admin/jws-key-rotation.html ↩

[josephineSei]

@markus-hentsch and I further discussed the possible need to:

1. standardize the usage of TLS and traffic encryption between the OpenStack services and the database at least.
2. deep-dive into the possibilty to create VPNaaS in Neutron (https://docs.openstack.org/neutron-vpnaas/latest/) and whether this would be needed as a standard.
3. deep dive into possibilities to isolate a Compute Host for a project, check if that would be a requirement for certain users and possibly make this feature better usable upstream