smo-v0.8.0

SMO 0.8.0

The security hardening release, and the operator console finished. A review of October 2026 (PR-SEC-15) found routes that trusted the caller's word, and they are closed: ownership and scope checks on the actions of a configuration job, on rApp Management, DME and AIMgF; an approval decided by the signed-in person the gateway vouches for, not by the request body; feature-group tokens given by reference; model checks in MLMR; escaped NETCONF XML; the MSAC credential checked; the console's sign-in lockout by source address; an audit path with no query; and a hardened shared library (the callback SSRF guard reads every spelling of an address, the rApp role patterns match the whole path, the SQLite fallback is an explicit opt-in, package size limits). The console was redesigned and completed: an alarm console with history and comments, KPIs over time by region, a containment tree on Topology, console users limited to regions and tenants, live counts from a new event feed, the whole console in German, change windows for CM jobs that start and expire by themselves, and a second approver for an rApp's change. The console, change-window, second-approver, console-scope and event-feed features are opt in; the security hardening is not, so read Upgrading from 0.7.0: the order of the upgrade matters for the approver identity, feature groups no longer return their token, a job can now be EXPIRED, and the introspection cache is on by default. What is still open is named under Known limitations.

Added

• An approved job starts by itself when its change window opens, and a waiting one expires when the window closes (PR-MGT-4 steps 4.4 and 4.5; schema revision 0044). Nothing changes for a job with no change window. Two tasks of the RAN NF OAM worker: start-due-windows (every RAN_NF_OAM_WINDOW_START_SECONDS, default 15) starts each SCHEDULED job whose window is open, once however many workers run, by the same code as POST /config-jobs/{id}/continue (approval recorded, window open, the rApp not stopped, and the scope claim of the caller that asked, the rate limit and the blast-radius and magnitude limits asked again, as POST /config-jobs asks them); expire-windows (every RAN_NF_OAM_WINDOW_EXPIRE_SECONDS, default 30) moves a job still PENDING_APPROVAL or SCHEDULED after its window_end to the new final state EXPIRED, every change REJECTED WINDOW_EXPIRED, nothing sent. The window opens at start and is over at window_end exactly (a job cannot start at that instant). If a safeguard refuses the start (the rApp that owns the job was stopped while it waited, an element left the caller's scope, the rApp's rate, blast-radius or magnitude limit now refuses it), the worker ends the job FAILED (haltedReason START_REFUSED, the refusal's code in haltedDetail) and records the refusal as any other, rather than trying again; a person's continue answers the refusal (403, or 429 for the rate) and leaves the job SCHEDULED. A start that fails with an unexpected error is tried again at the next run, and the fifth consecutive failure ends the job FAILED (haltedReason START_ERROR_LIMIT, the last error in haltedDetail; RAN_NF_OAM_WINDOW_START_ATTEMPTS, default 5; the count is kept on the job, so it survives a restart and is shared by replicas). No decision record or audit row is written for an expiry or a refused start in this release: the job and its state-change event carry the outcome. Revision 0044 widens the job status CHECK to admit EXPIRED and adds two nullable columns to write_config_job, requester_scope and start_attempts (expand only). The worker's two new intervals are read at start; the chart and compose need no change.
• The whole console in German (PR-GUI-6.4): every page, the sidebar, the top bar, the shared dialogs and the descriptions they show (flow steps, refusal and halt reasons, approval policies, form errors) are translated. In English nothing changes except that a count of one now reads in the singular where it did not ("1 alarm", not "1 alarms").
• The console can be shown in German (PR-GUI-6.3). Account → Preferences has a Language choice, English or Deutsch, saved with the user's other preferences (language on /api/me/preferences). The Preferences page is translated now; the other pages stay English until they are converted (GUI-6.4), and say so. Nothing changes for a user who does not choose German.
• The SMO publishes what happens as events, and the console's live counts follow them (PR-MSG-3 steps 3.1-3.3, 3.5, 3.6; PR-GUI-1.5; schema revision 0042). An alarm raised or cleared, an rApp instance's and a CM job's every state change are published as CloudEvents (docs/EVENTS.md, schema docs/schemas/event-envelope.schema.json) in the same transaction as the change, into the new shared table smo_event with a Postgres NOTIFY at commit. Platform components read them from GET /dme/smo-events (internal-only at R1; after, limit, wait up to 25 s, types). The console now shows such a change within about a second instead of at its next poll; if the feed cannot be read it polls as before. Events are kept 24 h (SMO_EVENT_RETENTION_HOURS) and purged by every worker's new shared task event-purge; SMO_EVENT_BUS=off stops publishing. Behind a transaction-pooling PgBouncer (SMO_DB_POOLER=transaction) a reader polls every 0.5 s; set SMO_EVENT_LISTEN_URL to a direct connection to have it woken at commit instead. Schema: Alembic head is 0042, expand only (one new table). Run scripts/db_roles.py after the migration where module roles are used (it grants the table and its sequence). Not yet: Kafka (MSG-3.4).
• Change windows for CM jobs, approved in the Approvals inbox (PR-MGT-4 steps 4.1 to 4.3, PR-GUI-7.1; schema revision 0041). Nothing changes for a job asked as before. POST /ran-nf-oam/config-jobs takes changeWindow {start?, end?} or requireApproval: true: the job is made PENDING_APPROVAL and sends nothing until someone other than its requester approves it (POST /config-jobs/{id}/approve, operator in the console; .../reject refuses it or withdraws an approved one). Approved inside its window (or without one) it runs at once; approved before the window it is SCHEDULED and an operator starts it in the window (POST .../continue, refused before the window opens unless forced, and after it closes). The Approvals page has a Change windows tab (counted in the sidebar badge), and the new config job form a "Hold for approval (change window)" option. Starting a scheduled job by itself and expiring one after its window follow (MGT-4.4, 4.5, below).
• A console user can be limited to regions and tenants (PR-GUI-5). Nothing changes for an existing user: every one stays unscoped. An admin sets a user's scope on the Users tab (scope on POST/PATCH /api/admin/users), or, with GUI_OIDC_SCOPE_CLAIM set, the identity provider's token gives it at every sign-in (a claim that breaks the rules refuses the sign-in). The console sends the claim with every call it makes for the user (X-R1-Acting-User-Scope, beside X-R1-Acting-User), R1 Termination forwards it only from the console's internal token and only with the person, and RAN NF OAM, DME and SME narrow the answer exactly as for a scoped rApp (a write outside it is refused SCOPE_DENIED). The console's counts, live updates, search and exports are asked and cached per claim. The user sees their limit beside the scope picker. Modules with nothing to match (rApp Management, AIMgF, MLMR, DME's types) are not narrowed, as for an rApp. Upgrade R1 Termination and the modules with or before the GUI backend: an older gateway drops the new header, and the user then sees what an unscoped user sees.
• KPIs over time, by region, in saved layouts (PR-GUI-4). The KPIs page's Overview has a KPIs over time box: up to eight KPIs, each a line over the Overview's range (1-minute steps for an hour, 15 for a day, an hour for a week), narrowed to a region and site cluster that start at the top bar's scope; the KPI tiles and the worst-elements list now follow the top bar's scope too. The KPIs, the range and the place can be saved as a named layout of the user's (at most 20), kept by the GUI backend so it follows the user to any browser. RAN NF OAM serves GET /kpis/{name}/series (at most 500 steps) and GET /kpis/{name} takes region and site_cluster. The GUI backend's new table gui_kpi_layout is made at start, like its other tables; no SMO schema change.
• The containment tree on the Topology page, with an alarm overlay (PR-GUI-3). A Containment tree box draws RAN NF OAM's managed-object tree (each element, its functions and cells), each node coloured by its worst open alarm; a folded node shows the worst alarm anywhere below it. A focused element (?me=) opens to its leaves; a node's name opens the element's page on its Managed objects tab. The top bar's scope narrows it. RAN NF OAM serves it as GET /topology/graph (at most 2000 nodes, default 500; the scope claim and MSAC apply). No schema change.
• Model gates in the Approvals inbox (PR-GUI-7.3). A Model gates tab lists the models waiting for a person (approve a training or validation, submit, approve or reject, certify), with the decision buttons each role may use; the sidebar's Approvals badge counts them too. AIMgF's GET /model-lifecycles takes awaiting_decision=true|false.
• An alarm's history and comments (PR-MGT-8.2, 8.3; PR-GUI-2.3, 2.4; schema revision 0039). RAN NF OAM records every change of an alarm (raised, acknowledged or withdrawn, re-graded, cleared, with who and when) and keeps operators' comments: GET /ran-nf-oam/alarms/{id}/history, GET/POST /ran-nf-oam/alarms/{id}/comments. The Alarms page's detail panel shows both and lets an operator add a comment. History starts with this release: an ...

smo-v0.7.0

SMO 0.7.0

The platform learns who an rApp may act on, and what it has to ask first. A signed and checked rApp package, a person's approval before an rApp's change is written (with a record of why, hashed into the audit chain), and a region and tenant an rApp is held to; rApps can now be written in Go and Java; a new element can be set up from a template and software can be rolled out in waves with a health gate; RAN NF OAM takes VES events; and Postgres can be reached over verified TLS. Everything is opt-in: nothing changes on upgrade until you set it. A few things are stand-ins and are named under Known limitations.

Added

• Signed rApp packages (PR-RAPP-1). Nothing changes until you set something. A package can carry a digest list of every file (TOSCA-Metadata/DIGESTS.sha256) and a detached ed25519 signature over it. ONBOARDING_TRUST_STORE (a PEM public-key file or a directory of <publisher>.pub, read for each package; chart: rappSigning.trustStoreConfigMap, a ConfigMap of your own) makes Onboarding verify a signed package against those keys and refuse one that was changed, had a file added or removed, is signed by an unknown publisher or with the wrong key, naming the rule in failureReason; ONBOARDING_REQUIRE_SIGNED_PACKAGES=true (chart: rappSigning.requireSigned) refuses an unsigned package too (and every package if no trust store is set). With neither set nothing is checked, signed packages included. scripts/csar_sign.py (keygen, sign, verify, digests) is the tool for a rApp developer and an operator. The four sample packages are now signed with a demo key whose private half is committed under samples/demo-signing/ on purpose: never put demo-publisher.pub in the trust store of a deployment that matters. signatureVerified keeps meaning "validated" while no trust store is set, and is true only for a verified package once one is. No dependency was added (ed25519 comes from cryptography, already locked). docs/RAPP_PACKAGING.md section 8.

• The rApp conformance pack (PR-RAPP-3, conformance/rapp). python -m conformance.rapp package my-rapp.csar checks a package file against Onboarding's own validation code, file by file, with advice (a missing ASD version, a test suite or a private key shipped), and signatures when given --trust; python -m conformance.rapp runtime --package-url URL --direct takes a package through onboard, register, heartbeat, R1 usage and terminate on a running stack and cleans up after itself. Both write report.json and report.md. CI runs both against the compose stack (the step was written without docker: confirm it on its first run).

• A runtime profile becomes CPU and memory requests and limits in the NFO descriptor (PR-RAPP-2.1). AIMgF's descriptors gain workloadTemplate.containerResources (beside the unchanged resources) and the descriptor Onboarding creates for a package gains containerResourcesByMode, each the resources: block of a Kubernetes container (requests equal limits; fractional CPU in millicores; GPUs not mapped). Nothing applies them yet: there is no deployment manager behind NFO.

• An egress NetworkPolicy for the rApp pods (PR-RAPP-2.3, chart value rappNetworkPolicy.enabled, default false). The reference rApps may reach R1 Termination, DNS, the bundled Postgres and the bundled Tempo and nothing else; extraPodSelectors and extraEgress extend it. Selects pods without a new pod label, so switching it on restarts nothing. Needs a CNI that enforces NetworkPolicy; try it in a lab first.

• A VES event receiver (PR-SB-7, steps SB-7.1 to 7.5). Nothing changes until you give it a password. An O1 adaptor that reports in VES (ONAP style) can post to RAN NF OAM's POST /ves/eventListener/v7 (and /eventBatch): a single event or an eventList of up to 500. A fault event becomes an alarm on the element named by sourceName (CRITICAL, MAJOR, MINOR and WARNING raise it, the same open condition is not raised twice and a new severity updates it, NORMAL clears it), a heartbeat is the adaptor endpoint's heartbeat, a measurement event and a 3GPP stndDefined 3GPP-PerformanceAssurance event become PM reports by the path POST /pm-reports takes (a PM subscription on the counter is needed). It is checked whole first (a bad header is a 400 that names the member, never the value, and nothing is applied); then each event is applied on its own and the 202 says what became of it, per event (APPLIED, PARTIAL, IGNORED, REJECTED, with fixed codes), because a refused event answered with an error would be sent again for ever. HTTP Basic: set RAN_NF_OAM_VES_PASSWORD or RAN_NF_OAM_VES_PASSWORD_FILE (and optionally RAN_NF_OAM_VES_USERNAME, default ves); without a password the route answers 404, a password set twice or an unreadable file answers 503. The route is not behind the R1 gateway (an adaptor posts to the module directly; expose it to the adaptors' network only). Not done: the Kafka variant (needs MSG-3.4), other VES domains, a commandList back-channel, and trying it against a real sender: the mapping was written from the schema. ran-nf-oam/README.md.

• MSAC beyond writes (PR-MGT-2, steps MGT-2.1 to 2.5). Off until you set RAN_NF_OAM_MSAC_REACH=on. The TS 28.319 access rules that decide who may write a managed object also guard, for a caller that is a registered MSAC Identity (matched on the invoker id the gateway vouches for): read on the configuration read, config history and diff, on PM and FM subscription create and on a PM file download; update on alarm acknowledge and clear; exec on a software-management job; read on the whole network (/*) for a file subscription. GET /files leaves out the files of elements the caller may not read. A denial is 403 MSAC_ACCESS_DENIED. A caller that is not an Identity is not asked, and a read takes no role of the caller's choosing. Switching it on without adding read rules to an Identity made for writes makes that Identity's reads fail: add them first. Not covered: list routes other than GET /files, subscription deletes (OPEN_ITEMS.md, MGT-2.6).

• A vendor profile on the O1 stub, and a first profile that is a stand-in (PR-SB-10; the mechanism, SB-10.1 to 10.4 for example-du). MOCK_O1_PROFILE=<name> makes mock-o1-adaptor one vendor's adaptor from mock-o1-adaptor/app/profiles/<name>/ (declaration, class defaults, no RESTCONF root for a NETCONF-only vendor); the same directory gives the body for RAN NF OAM's POST /vendor-onboarding (capability entry and the vendor's YANG-derived descriptor), lists the vendor's deviations in its README and carries the O1 conformance kit's report (a test fails when the report is stale). example-du is not a real vendor (no vendor lab was available; its content is invented), so the first real profile is still open (SB-10.5). Unset, the stub is unchanged. mock-o1-adaptor/app/profiles/example-du/README.md.

• Postgres over TLS, and every service checking it (PR-SEC-2.4). Opt in: nothing changes unless you ask. scripts/mtls_certs.py init now also makes a server certificate for Postgres (certs/mtls/postgres). With compose, docker compose -f docker-compose.yml -f docker-compose.pgtls.yml up -d starts Postgres with ssl=on and a pg_hba.conf that accepts the network only over TLS (a plain connection is rejected), and every service, worker, sample rApp and the migrate and backup containers connect with sslmode=verify-full against the development CA (PGSSLMODE and PGSSLROOTCERT; the URLs are unchanged). With the chart, postgres.tls.enabled=true does the same for the bundled Postgres with a Secret you make or a cert-manager Certificate (postgres.tls.certManager); use it on a new install. A wrong CA, or a certificate that does not name postgres, stops the connection. scripts/pg_tls_check.py proves it (right CA connects over TLS, another CA and another name are refused, plain is refused), against a throwaway local Postgres (--spawn) or a running one. PgBouncer can verify its own connection to Postgres (PGBOUNCER_SERVER_TLS_SSLMODE, PGBOUNCER_SERVER_TLS_CA_FILE); the services' connection to the pooler stays plain. Docs: docs/ARCHITECTURE.md "Postgres over TLS", deploy/helm/smo/README.md.

• The introspection cache is measured, and a comparison is part of the load lane (PR-SEC-5.4). scripts/load_run.py now reports how many token checks SME answered during the measured part and the gateway's cache hit ratio; the load workflow runs the same load again with R1_INTROSPECTION_CACHE_SECONDS=30 and scripts/introspection_compare.py sets the two side by side (and fails if the cache did not cut SME's introspections by half or either run had an error). First reading, on one machine without Docker (scripts/load_local.py): 98.4 introspections per 100 calls without the cache, 0.9 with it; 8.6 against 15.5 requests a second; no errors (docs/PERFORMANCE.md). The default stays 0 (off): 30 seconds is recommended and is the owner's decision (OPEN_ITEMS.md, SEC-5.5).

• A browser check of the pages rApps declare (PR-GUI-8, step GUI-8.7). scripts/gui_rapp_pages_e2e.py and a stub rApp (scripts/gui_rapp_stub.py) onboard a package after the GUI was built and drive its page in Chromium as the operator and as the viewer: panels, the row drawer, a button and a row action reaching the rApp with the signed-in user, no change button for the viewer, and the backend refusing a hand-made call (UNDECLARED_ROUTE, FORBIDDEN, RAPP_READ_ONLY). It is a step of the smo-gui-e2e job.

• A Go SDK for rApps written in Go, with an example rApp (sdk-go/, PR-RAPP-4, steps RAPP-4.2 to 4.5 for Go; the Java SDK is a separate change). The smosdk package is a thin client over R1 using only the Go standard library (no third-party module, so no go.sum to verify and nothing for Dependency review to flag): it discovers SME...

smo-v0.6.0: Release 0.6.0 (#368)

SMO 0.6.0

The operator's side of the platform grows up. The RAN O1 stub now emits what a real network function emits (alarms, performance reports and files, software phases, heartbeats) and the conformance kit checks that RAN NF OAM received it; the GUI has one rApps entry with a directory and a page per rApp that the rApp's own package declares, in place of one hand-written page per rApp, so a rApp onboarded at run time shows its page without rebuilding the GUI; sign-in gets a second factor (an OIDC-only mode, one-time codes for local accounts, mandatory for admins), the session token can be signed with a key pair, and the gateway can cache token checks. Retention periods, an External Secrets example, a plugfest plan and the SLO reference targets (docs/SLOS.md, accepted) are in. Nothing that deletes data or changes sign-in is on by default.

Added

• Signing keys for the GUI session token, and an optional cache of token checks at the gateway (PR-SEC-5, SEC-5.1 to 5.4). Nothing changes until you set something. (1) GUI_JWT_ALGORITHM=RS256 or ES256 with GUI_JWT_PRIVATE_KEY_FILE (a PEM file, a mounted secret; chart: gui.jwtKeySecretRef) signs the session token with a key pair instead of the shared GUI_JWT_SECRET; the token carries a kid, GUI_JWT_PREVIOUS_KEY_FILES keeps the previous keys verifying while you rotate, and GET /.well-known/jwks.json publishes the public keys (an empty set with the default HS256, never the secret). A token whose alg is not the configured one (none, HS256 signed with the public key) is refused. Switching the algorithm signs every user out once. Default HS256, unchanged: a session issued before the upgrade keeps working. (2) R1_INTROSPECTION_CACHE_SECONDS (default 0 = off) makes R1 Termination reuse SME's answer about a token for that many seconds; a token revoked at SME is honoured for at most that long (at once on the gateway replica the offboarding went through), a wrong token for 5 s at most, SME being down is still a 503. Metric smo_introspection_cache_total{result}. The load test that shows the SME calls saved has not been run yet. gui-bff/README.md 2.11, r1-termination/README.md "Introspection cache", docs/SECRETS.md.
• One rApps entry in the GUI: a searchable directory, a page per rApp that its own package declares, and pins (PR-GUI-8, steps GUI-8.3 to 8.6). rApps now opens a directory of every rApp instance (search by name, owner, version or id; filters for state, owner, "declares a page" and "pinned"), and a row opens /rapps/<instance>: the page the rApp's package declares in operatorUi (tables with sparklines and a drawer per row, key-value panels, KPI tiles, charts, buttons with inputs and a confirmation), drawn by one generic renderer, above the platform overview every rApp has (lifecycle, KPIs reported, safeguards, faults, version history). A rApp onboarded at run time shows its page with no GUI build; one that declares nothing has the overview. Each user can pin up to five rApps to the sidebar (stored by the GUI backend, so they follow the user). The GUI backend calls a rApp only on the routes its declaration lists, for that instance: a viewer's change, an undeclared route, another instance's id and any change on a readOnly page are refused (FORBIDDEN, UNDECLARED_ROUTE, RAPP_READ_ONLY), the request body is limited to the declared inputs and fixed values ("{user}" is the signed-in user), and every change is two audit rows, before the call is sent and after it answers. The browser never learns the rApp's address. Upgrading: a rApp needs a registered operator API for its declared panels to show data: pass operatorApiBase (an http or https URL) when the instance is created (the Deploy dialog has a field for it) or set it with PUT /rapp-mgmt/instances/{id}/operator-api; revision 0030 adds the nullable column rapp_instance.operator_api_base (expand only; nothing reads it until you set one). The gateway has a new prefix, /rapps/{instanceId}/operator/..., resolved per instance (cached R1_OPERATOR_API_CACHE_SECONDS, default 5) and forwarding none of the caller's credentials. The four sample rApps declare their pages (their packages are rebuilt) and the demo scripts pass operatorApiBase. Not done: the browser check against the compose stack (GUI-8.7).
• An example for supplying the chart's Secrets from Vault through the External Secrets Operator (deploy/external-secrets, SEC-4.7). A ClusterSecretStore, an ExternalSecret for each Secret the chart reads (smo-secrets, smo-role-secrets, the GUI's OIDC client secret and one-time-code key, a per-module mTLS Secret), the matching Helm values and a note on Vault Agent / the CSI driver. A test checks every name and key against the chart. It has not been applied to a cluster.
• A plugfest plan (docs/PLUGFEST.md, STD-3). Each interface the SMO implements mapped to the O-RAN specification family and test specification (cells not confirmed are marked as such), and a one-page plan: scope, what the stack offers a counterparty, what we need, entry and exit criteria, open questions. The plugfest itself has not been held.
• A rApp package can declare the page the operator sees for it (operatorUi in manifest.yaml, PR-GUI-8, steps GUI-8.1, 8.2, 8.8). Panels (table, keyValues, kpis, chart, actions), each bound to a GET route of the rApp's operator API, with buttons for the changes the rApp offers and, for a table, a drawer per row (rowDetail: up to 6 blocks of JSON, key-values, a table or a chart, from the row or from a per-row GET route); a generic renderer will draw them, so onboarding a rApp will make its page appear with no GUI build (the directory, the renderer and the GUI backend that reaches the routes come in the next steps; this change is the format, the check and the author's helper). Onboarding now validates the declaration and refuses, with the place and the rule, an unknown panel kind, a source that is not a GET, a route with .., a {row.x} that names no column, a declaration over its limits (64 KiB, 20 panels, 20 columns) and duplicate ids; the package ends FAILED and the message is returned as failureReason in the 202 answer of POST /packages (and logged). A package without operatorUi, the four sample packages included, onboards exactly as before. smo_sdk.operator_ui writes and checks a declaration. The decision, the limits and a worked example are docs/adr/0004-operator-ui-declaration.md; the JSON Schema is docs/schemas/operator-ui-1.schema.json; the manifest key is in docs/RAPP_PACKAGING.md. No migration and no change to the OpenAPI documents.
• The O1 conformance kit checks the emitting side too, and the RAN O1 stub emits (SB-9.3 to 9.5, SB-9.8; conformance/o1, mock-o1-adaptor). python -m conformance.o1 --adaptor URL --oam-url http://ran-nf-oam:8000 now also runs 12 checks (FM-1 to 4, PM-1 to 3, SW-1 to 3, HB-1 and 2): the adaptor is asked, through a trigger API (--emit-url, default --adaptor), to raise an alarm, report a PM report and file, report the DOWNLOAD / INSTALL / ACTIVATE phases of a software-management job and send a heartbeat, and the kit reads RAN NF OAM back (the alarm listed with the fields sent and the severity normalised, the file listed and stored, the job COMPLETED, the endpoint ACTIVE). It prepares the element, endpoint and PM subscription itself (or uses --element). Without --oam-url, and for a service the adaptor does not declare in /capabilities, a group is skipped and the report says so. mock-o1-adaptor implements the trigger API: POST /emit/alarm, /emit/pm-report, /emit/pm-file, /emit/heartbeat, /emit/software-phase, towards MOCK_O1_OAM_URL (new; compose and the chart set http://ran-nf-oam:8000), so the stub is a source of FM, PM, software and heartbeat for development. Plain HTTP: not usable towards a RAN NF OAM that requires mutual TLS. The compose end-to-end CI job runs the kit against the stack. This replaces 0.5.0's known limitation that the stub is the configuration-side mock only.
• Retention periods ready to use, and a warning when a table you chose to keep grows large (DB-3.10). Nothing changes by default: every retention setting is still 0 (keep), so an upgrade deletes nothing. The periods proposed in docs/RETENTION.md (alarms cleared 90 days, PM files 14, safeguard refusals 90, MDAF reports 30, GUI audit log 365 with export first) are now in deploy/helm/smo/values-production.yaml (Helm) and the prod GitOps overlay, and, commented out, in .env.example (compose, which now also passes GUI_AUDIT_RETENTION_DAYS and GUI_AUDIT_EXPORT_DIR to gui-bff). The GUI audit log is purged with kubectl -n smo exec deploy/gui-bff -- python -m app.retention (the chart has no CronJob: the BFF's SQLite volume belongs to one pod). New SMO_RETENTION_WARN_ROWS (default 1000000, 0 never warns): the workers' purge tasks (and python -m app.retention) log one WARNING per table per day when a table whose retention is 0 is larger, and the workers export smo_retention_off_rows{table}. The chart now sets SAFEGUARD_REFUSAL_RETENTION_DAYS: "0" explicitly on ran-nf-oam and its worker (as compose does), which restarts those two pods once on upgrade.
• Multi-factor sign-in for the GUI (SEC-7): a one-time code for local accounts, an OIDC-only mode, and a break-glass account. Nothing changes until you set something. (1) GUI_LOGIN_MODE=oidc closes the password form to every account except a break-glass one, so your identity provider's MFA applies (it needs GUI_OIDC_ENABLED=true, or the backend does not start); the sign-in page then shows only "Sign in with " and a small "Break-glass sign-in" link. local hides OIDC even when configured. The default both is today's behaviour. (2) With GUI_TOTP_KEY (or GUI_TOTP_KEY_FILE; at least 32 characters, openssl rand -base64 32; in the c...

SMO 0.5.0

SMO 0.5.0

Prepares the road to 1.0.0, and most of it is proof. Every claim made for 0.4.0, and the new ones, is now tried by a lane: mutation testing of the shared library, a contract test and a breaking-change check against the previous release, an upgrade from the previous two releases (with data, under load, and with a migration that cannot finish), a rollback of the whole chain of revisions over data, a three-node Kubernetes cluster with a node drain, pod kills, a planned Postgres switchover and the autoscaler under load, a restore from a CloudNativePG WAL archive, the database at volume, stress and soak runs of the gateway, browser checks of the GUI, and authenticated and unauthenticated scans. What they found is under Fixed. New: mutual TLS between the services, OIDC login for the GUI, a shared rate limiter, distributed traces and log shipping, alerts and SLOs with runbooks, disaster recovery (RPO 15 minutes, RTO 1 hour), retention of the high-volume tables (off until set), a configuration reference, measured sizing, an O1 adaptor conformance kit and the standards documents. Removed: the compatibility views of the per-module schemas, and A1, the Near-RT RIC and E2 (out of scope). Not done, and needing a host of our own: a disaster-recovery drill at real size, a hard node loss, a network partition, PgBouncer failover, zones. The criteria for 1.0.0 are in docs/RELEASES.md.

Removed

• The compatibility views of the per-module schemas are dropped (revision 0029, DB-2). When 0.4.0 moved each module's tables into a schema of its own it left a view of the same name in public for the previous release (0.3.0), which names its tables with no schema; those views were promised for one release and this is the revision after it. 0.4.0's modules connect as their own roles, with the module's schema first on the search path, and never read them (scripts/check_previous_release_code.sh has run on that basis since 0.4.0), so a rolling upgrade from 0.4.0 and a helm rollback to it are not affected. What stops working: naming a module table with no schema as the database owner: psql as smo, a script of your own, the code of 0.3.0 (upgrading from 0.3.0 is still supported, rolling back to it is not and never was past one release). Name the schema (onboarding.application_package) or connect as the module's role (docs/SECRETS.md). A release with databaseRoles.enabled=false (every module connects as the owner) keeps working: scripts/db_roles.py, run by the migration Job, puts the module schemas on the owner's search path for this database (the Helm lane found it failing without). The downgrade of 0029 makes the views again, as they were. The retired A1 tables are not dropped here: 0.4.0's a1-related still uses them, so they go in the revision after this release.
• A1, the Near-RT RIC and E2 are out of scope at every stage, and what implemented them is gone. Removed: the a1-related module (A1 policy management, the A1-P service registry and A1 EI types) and mock-near-rt-ric; the /a1-related route of the gateway (now 404 NO_ROUTE) and its role-policy and GUI-permission entries; their compose services, secret db_password_a1-related and network a1_mock_net, their Helm modules and the chart's networkPolicy (its only rule isolated the mock RIC); A1_RELATED_URL; their specs (docs/openapi/a1-related.json, mock-near-rt-ric.json), call flow 05 and demo runbook steps 11 and 22 (the other steps keep their numbers); the SO SMOS dispatch of POLICY / A1_RELATED steps (such a step now fails as having no dispatcher); in the GUI the A1 policies, policy status subscriptions, A1 services and A1 EI types tabs, call flow 05 and the A1 tile of the dashboard (the page "Policy & Intents" is now "Intents", at the same address); and the OI-5-a1-* open items. The code is in the tag smo-v0.4.0; OPEN_ITEMS.md keeps A1 as a future work item.
  Upgrading: nothing consumed A1 outside this repository (no sample rApp used it). The four tables (a1_ei_type, a1_policy, a1_service_registration, policy_status_subscription, in the schema a1_related) and the role smo_a1_related stay in an existing database, unused: the release that removed the code cannot drop what the previous release still uses (a rolling upgrade runs both), so a later revision drops them, and until then they are listed under _retired in migrations/table_owners.json. An operator who does not need the previous release any more can drop them, the schema and the role by hand. The Helm values modules.a1-related, modules.mock-near-rt-ric and networkPolicy are gone: delete them from your values file.

Fixed

• A new chart version no longer restarts every pod, the database included (found by the upgrade lane once its load was gated on a release PR, V-10). The chart put its own version (helm.sh/chart: smo-<version>) in the labels of every pod template, and a changed pod template restarts the pod: so the release that raised the chart version restarted the bundled Postgres on helm upgrade, and again on helm rollback, and for the seconds it was gone every route answered 401 (the gateway could not check the token), 500 or 502. The lane showed it: postgres-0 went Terminating one second after the migration finished, and the first error was 401 one second later. Pod templates now carry no chart version (smo.podLabels); the objects still do. The Postgres pod keeps the label release 0.4.0 gave it (smo-0.4.0, frozen, with a comment and a test), so the upgrade from 0.4.0 does not restart the bundled database and neither does any later chart release. The modules still restart on an upgrade because their image tag changes, one pod at a time behind a readiness probe. An external or managed Postgres was never affected.
• python -m smo_shared.audit verify no longer reports a broken chain when a writer commits while it reads (found by the load through the upgrade lane, V-10). It read the rows, then the head in a second statement; a row added in between left the head ahead of the rows and was reported as "rows removed from the end?" (the lane failed once in three runs with the head says the chain ends at row 290; it ends at 288). It now reads the head before and after the rows and accepts a chain that ends between the two; rows missing from the end, or rows past the head, still break it. The chain was never damaged: only the check was wrong, so a running stack could fail its own verification under load.
• podDisruptionBudget.enabled=true no longer blocks a node drain on a module with one pod (found by the node-drain lane, V-12). The chart gave every service module without a volume a budget of minAvailable: 1, including one that runs a single replica (mock-o1-adaptor): evicting its only pod could never satisfy the budget, so kubectl drain of the node it was on waited for ever. A budget is now rendered only for a module with more than one replica, or when the autoscaler's minimum is more than one.
• The shared rate limiter (R1_RATE_STORE=postgres) let more calls through a burst than its budget, and no longer does (found by the three-replica burst, V-9). Each replica stamps its statement with its own clock before the statement waits for the caller's row, so statements issued together are applied in any order; one stamped earlier than the row's clock moved the bucket's clock back, and the next statement was refilled for the same interval again. In the burst (120 calls at once through three replicas, a budget of 20 and 5 a second) 43 calls got through against about 30. The bucket's clock now only moves forward (GREATEST/max in the upsert); a replica whose clock stays behind adds no refill of its own. Nothing to change; the default memory store is not affected. Upgrading: take effect with the new image, no migration.
• A request to MDAF with a timeOut that is not a date-time is refused (422) when it is made. Such a value used to be stored and then made every later POST /mda-reports answer 500 while the request was matched; a timeOut without a zone is read as UTC. Found by the nightly contract fuzzing (mdaf/app/ts28104.py, mdaf/app/mda.py).
• Deleting a GUI user now also removes its failed-login counter, and a user created again under the same name no longer accepts the old person's tokens (found while writing the erasure procedure, STD-4.3). DELETE /api/admin/users/{username} removed the account and left the gui_login_failure row kept under that name; both go in one transaction now. A user created after a deletion started at session-token version 0 and so accepted an unexpired token the previous holder of the name had been issued; a new user now starts at a random version. Existing users and sessions are unaffected. The audit rows that name a deleted user are kept (append-only), as are the names the GUI wrote into module records: docs/PRIVACY.md says which and what the options are.
• A destination that does not answer no longer holds up the other notifications (found while writing the subscriber stress case, V-9). smo_shared.outbox.drain sent the due rows one after the other, so every row behind a dead or hanging subscriber waited out that subscriber's two-second timeout (a sweep over 100 rows to a dead destination took over three minutes, during which healthy subscribers got nothing). The claimed rows are now sent concurrently (SMO_OUTBOX_SEND_CONCURRENCY, default 8); the claiming, the retry/backoff and the lease are unchanged. A test with eight hanging destinations and one healthy one shows the healthy one delivered at once.
• Logging out of the GUI now ends the session itself; a copied cookie or token stops working (found while designing the browser session checks, V-13). POST /auth/logout only deleted the browser's cookies, so a session token that had been copied stayed valid until it expired (8 hours by default). Each session now carries...

SMO v0.4.0

SMO 0.4.0

Runs highly available, and what that took. Two replicas of every module that can have them are proved on Kubernetes and in compose (the whole runbook replayed against them, a rolling restart under a health probe, the delivery worker and the job runner killed mid-work), Postgres runs as a three-instance cloud-native cluster behind postgres.external.* and survives losing its primary, PgBouncer is available in compose, every module has its own schema and database role, and a staged CM job that had been failing on Postgres since 0.2.0 now works.

Upgrading from 0.3.0

• Schema: revision 0026 replaces two CHECK constraints (the status of CM jobs and their sub-changes) with ones that allow more; no data changes, and the previous release runs on it. Revisions 0023 (Onboarding), 0024 (MLMR, RAN Analytics, SO SMOS, SA SMOS, Intent Service, MDAF and the four reference rApps) and 0025 (SME, DME, NFO, RApp Management, A1 Related, FOCOM, AIMgF, RAN NF OAM) move each module's tables into a schema of its own (the module's name, _ for -); 0023 moves Onboarding's three tables (application_package, artifact, package_usage_registration) into the schema onboarding and leaves an updatable view of each name in public, so 0.3.0 code, including connections opened before the migration, reads and writes them as before. The views are dropped one release later.
• Compose: run scripts/init_secrets.sh (it makes secrets/db_password_<module> for each module with a role), then docker compose up -d: the migrate service now also runs scripts/db_roles.py, which makes the role smo_onboarding and sets its password, and Onboarding connects as that role. Without the new secret file docker compose refuses to start (a missing secret), which is the signal to run the script.
• Helm: helm upgrade makes the Secret smo-role-secrets (one password per role) before the migration runs, the migrate Job makes the role, and Onboarding connects as it; nothing to do. With postgres.external.* the owner needs CREATEROLE (or set databaseRoles.enabled=false, or make the roles by hand). A Secret you supply for the passwords goes in databaseRoles.existingSecret.
• Nothing is required for the new options. SMO_DB_POOLER / SMO_DB_PREPARE_THRESHOLD (process settings), SMO_DB_HOST / SMO_DB_PORT (compose variables) and postgres.external.targetSessionAttrs (a chart value, host may now be a comma-separated list) default to the behaviour of 0.3.0. The compose profile pooler and the file docker-compose.replicas.yml are opt-in.
• Compatibility views: each table that moved into a module schema left an updatable view of the same name in public, so the 0.3.0 code runs on this schema. They are dropped in the release after this one; code that names a table without its schema (your own scripts, reports) should use <schema>.<table> or set search_path before then.
• Mixed versions: a 0.3.0 module and a 0.4.0 module can run side by side during a rolling upgrade (CI runs the previous release's runbook on this schema).

Changed

• Every module that uses the database has a schema and a database role of its own (PR-DB-2.5 to 2.7). Onboarding first, then MLMR, RAN Analytics, SO SMOS, SA SMOS, Intent Service, MDAF and the four reference rApps, then SME, DME, NFO, RApp Management, A1 Related, FOCOM, AIMgF and RAN NF OAM (its worker uses the same role) and R1 Termination (a role, no schema: it owns no table). MLLF, the two mocks and the GUI backend (its own SQLite file) do not use the shared database. R1 Termination's one read of another module's table is explicit: the gateway's kill switch reads ran_nf_oam.rapp_kill, a SELECT-only grant listed in migrations/db_roles.json; it names the schema through R1_KILL_SWITCH_SCHEMA (set by compose and the chart), and the previous release's gateway still finds the table through the compatibility view. What the pilot established, which holds for each: Its tables are in onboarding; smo_onboarding can use them, the shared module_identity, and alembic_version, and nothing else (it cannot read any other module's table, the audit chain, the outbox or the idempotency keys, or create anything in public). The role's default search path is onboarding, public, so no code changed. CI proves it twice: a Postgres test connects as the role and is refused every other module's table, and the compose job checks that Onboarding's live connections are the role's during the runbook replay. docs/SECRETS.md has rotation and restore. This is the pattern for the other modules, one PR each. The Helm chart does the same (databaseRoles): a module with a databaseRole mounts its own role's password and not the owner's; smo-role-secrets is a hook resource so an upgrade from a chart without roles finds it before the migration needs it; CI installs, turns the roles off and on again on the running release, and checks the role is refused other tables.
• Foreign keys between modules are gone (PR-DB-2.4, revision 0022). 24 constraints crossed a module boundary (AIMGF → MLMR, NFO and DME; Onboarding ↔ NFO; RApp Management → Onboarding; SA SMOS → SO SMOS, MLMR and MDAF; A1 Related → DME). They are dropped; the columns and the data stay, and the ORM already declared them as plain ids. What that means: the database no longer refuses a row that names a package, model or order that does not exist (the owning module's API checks, as before), and deleting an MLMR model no longer deletes AIMGF's rows about it (they stay and name a model that is gone; before, Postgres cascaded, which the SQLite tests never did). No application code relied on the constraints (none catches their errors). downgrade puts them back and fails if a row now names something that does not exist. The previous release's code runs on the new schema.
• Every table has an owner (PR-DB-2.1/2.2). migrations/table_owners.json maps the 134 tables to the module that declares them; a unit test and the Postgres schema check fail on a table with no owner or a foreign key across modules. This is the map per-module schemas and roles are built from (next).

Fixed

• A staged CM job failed on Postgres the moment it paused after its first wave (since 0.2.0). A job with waveSize goes to HALTED between waves, and one reverted automatically marks its applied sub-changes REVERTED; the status CHECK constraints written with the baseline allowed neither, so the request got a 500 (violates check constraint "write_config_job_status_check") and the job stayed at wave 1. The unit tests run on SQLite, which does not carry the constraints, and the runbook uses single-wave jobs, so nothing exercised it until the failover test of PR-HA-4.2. Revision 0026 replaces both constraints; tests_integration/test_status_checks.py checks them against the states of the code on Postgres, and the migration round-trip test now compares constraint definitions across every schema (it looked only at public, so a revision that changed a module's table was invisible to it). If you ran staged jobs on 0.2.0 or 0.3.0 and saw 500s from POST /config-jobs, this is the cause; the job rows from those requests were rolled back, so there is nothing to clean up.
• A stopped rApp's config job left no record. Since 0.3.0 the gateway refuses every change by a stopped rApp, config jobs included, with 403 RAPP_KILLED, before RAN NF OAM sees it. RAN NF OAM is what records a refusal (GET /safeguard-refusals) and tells those subscribed to refusal events, so a config job refused at the gateway was neither recorded nor announced. The gateway now leaves POST /ran-nf-oam/config-jobs to RAN NF OAM, which refuses it the same way (403 RAPP_KILLED) and records it; every other change by a stopped rApp is still refused at the gateway. (It showed on the CI cluster once the gateway ran two replicas: with one, the gateway's 3 s cache of the switch was usually stale at that moment, so RAN NF OAM refused first.)
• A notification whose first send failed was never retried. The outbox (0.1.0 onward) leaves a row PENDING when the destination does not answer or answers 5xx, to be retried by a sweep, and the sweep had not been built: only the send made right after the commit ever ran, so a subscriber that was down for a moment missed that notification for good, and so did one whose sender died mid-send. Every worker (ran-nf-oam-worker in compose and in the chart) now runs the sweep (outbox-sweep, every 5 s, SMO_OUTBOX_SWEEP_SECONDS; SMO_OUTBOX_SWEEP=false turns it off): due rows are sent with the existing back-off (5 s, 30 s, 2 min, 10 min) and are DEAD after five attempts; a row claimed by a sender that died is sent again when its 60 s lease runs out. Delivery is at least once. If you ran no worker, run one.

Added

• Calls to a Service are spread over its replicas, checked on Kubernetes. The CI job helm sends 80 calls (to a path that does not exist, which the service counts and answers 404; /bootstrap for the gateway) to each of eight replicated modules from inside the cluster and reads each replica's own request counter (scripts/k8s_route_count.py); every replica must have served at least 10. This is the kube-proxy spread (per connection); a client that keeps one connection open stays on one replica until it reconnects.
• A staged CM job survives its job runner, proved on Kubernetes (PR-HA-4.2). The CI job helm starts a three-wave job with a 20 s pause between waves, scales the worker to zero after wave 1, waits out the pause and checks the job has not moved (only the worker advances it), scales the worker back, and requires the job to complete with every sub-change applied once (scripts/k8s_staged_job_failover.py). The state is in the database, so nothing was needed in the product.
• The runbook replayed against two replicas of every module (PR-HA-1.2). `docker-compose.rep...

SMO 0.3.0

SMO 0.3.0

Safer rApps and a second way to run it. An rApp that writes through DME is held by its own safeguards (this was not the case in 0.2.0), the gateway lets an rApp change only what it uses and refuses everything from a stopped one, every change is in a tamper-evident audit chain, a Helm chart installs the stack on Kubernetes, and the operator GUI manages change jobs, KPIs and safeguards. Docker Compose stays the way to run it on one machine.

Upgrading from 0.2.0

• Schema: revisions 0019 (KPI schedules), 0020 (KPI guard) and 0021 (audit chain) are additive; the migration runs as before (the migrate service, or the Helm Job). The 0.2.0 code runs on the new schema.
• R1 Termination now needs the database (SMO_DATABASE_URL, db_password), for the audit chain and the kill switch. Compose does this; a deployment of your own must give it the same settings as a module.
• An rApp may now change only what it uses. A third-party rApp that calls a route outside the list gets 403 ROLE_NOT_PERMITTED (see Changed). Set SMO_ROLE_ENFORCEMENT=audit on SME and R1 to log instead while you move it, or add the route to RAPP_MAY_CHANGE in shared/smo_shared/roles.py.
• The worker: compose has a new service, ran-nf-oam-worker; the periodic work (staged job advance, KPI schedules and guards) runs only while it does.
• Mixed versions: during a rolling upgrade a module that is still 0.2.0 does not forward X-R1-On-Behalf-Of, so the DME safeguard fix applies once every module is on 0.3.0.

Security

• An rApp that wrote through DME was not held by its own safeguards. DME mediates an rApp's action (POST /dme/actions, the way the SDK and every sample rApp write configuration) and posts the write to RAN NF OAM itself, so RAN NF OAM saw DME as the caller: the kill switch did not stop such an rApp and its limits (config jobs per hour, elements per job, change per write) did not apply, and its jobs were counted against DME. A module now says whom it acts for: X-R1-On-Behalf-Of, which R1Client adds by itself to every onward call made while handling an rApp's request (so it holds across several modules). R1 Termination forwards it only from an SMO module and drops an rApp's own value, so an rApp cannot pose as another to escape its stop or spend another's allowance. RAN NF OAM applies the safeguards to the rApp, records refusals against it, and counts its jobs. If you run rApps that write through DME and had set limits or a stop, they take effect now. 0.2.0 had this gap; apply this release's gateway and modules together (a mixed-version stack keeps the old behaviour for a call that crosses an old module).

Changed

• An rApp may now change only what it uses (PR-SEC-14, v0.3). The gateway's role policy was a short deny-list; it is now also an allow-list for changes (POST, PUT, PATCH, DELETE) by a caller with the rapp role. An rApp may change what smo_sdk calls (the SDK's own tests fail if a call is off the list), the consumer-facing request routes of the AI/ML services, the config-jobs and rollback routes of RAN NF OAM (its safeguards apply), and the rApp-facing modules a1-related, ran-analytics, dme-push and dme-pull. Everywhere else a change is refused with 403 ROLE_NOT_PERMITTED: onboarding, instance management, NFO, FOCOM, the orchestrators, the sample rApps' own APIs, other invokers' registrations, trusted invokers, function/repository/storage administration, sweeps and runtime scaling. Reads are unchanged (open to every valid token, apart from the deny-list). SMO modules and the GUI are not affected. An rApp that calls a route outside the list gets a 403 after upgrading: SMO_ROLE_ENFORCEMENT=audit counts and logs instead while it is moved, and the route can be added to RAPP_MAY_CHANGE in shared/smo_shared/roles.py.
• A stopped rApp changes nothing through the gateway (AI-10.4, extended). The per-rApp kill switch refused a stopped rApp's config jobs; everything else it could still do (publish data, start training jobs, post intents). R1 Termination now reads the switch (rapp_kill) and refuses every POST, PUT and PATCH by a stopped rApp, and by an SMO module acting for it (X-R1-On-Behalf-Of), with 403 RAPP_KILLED before any backend is called, recorded in the audit chain as REFUSED:RAPP_KILLED and counted in smo_role_refusals_total{action="killed"}. It can still read, DELETE (withdraw what it made), fetch a token, and roll back its own config job. The gateway looks the switch up at most every 3 s per rApp (R1_KILL_CACHE_SECONDS), so a switch thrown at RAN NF OAM bites at the gateway within that time (RAN NF OAM's own check for config jobs is immediate). If the database cannot be read the last answer is used for up to a minute, and with none the change is refused (503 KILL_SWITCH_UNAVAILABLE). R1_KILL_SWITCH=off turns the gateway's part off. The GUI's stop dialog says what a stop now does.

Added

• An rApp instance's credentials are delivered to its workload (PR-SEC-14, completing it). SME keeps only a hash of an invoker's secret, so the secret existed for one moment and had to be fetched by a person and put into the workload by hand. With RAPP_CREDENTIAL_DELIVERY=kubernetes (Helm: rappCredentials.delivery=kubernetes, default off) rApp Management writes it, when the instance's invoker is made, to a Kubernetes Secret rapp-<instanceId>-credentials (SMO_INVOKER_ID, SMO_INVOKER_SECRET, SMO_IDENTITY_KIND=rapp) for the workload to take with envFrom; rotating (POST /instances/{id}/credentials) replaces it and answers with the Secret's name instead of the secret, and terminating deletes it. If it cannot be written, creating the instance fails with 503 and the invoker is withdrawn. In the chart the rApp Management pod, and only it, has a service account (token projected, not auto-mounted) whose Role can create, update and delete Secrets in the namespace and read none; CI checks this on kind. Compose has no equivalent (its sample rApps are static services with module identities). scripts/k8s_credential_delivery_check.py is the in-cluster check.
• Tamper-evident audit of changes (PR-SEC-11). The gateway records every authenticated POST, PUT, PATCH and DELETE in a hash chain (audit_log, audit_head, revision 0021): who (the invoker id and role; for an SMO module acting for an rApp, the module with the rApp in detail), what (method and path, never the query or the body), the result (the status, or REFUSED:ROLE_NOT_PERMITTED) and the correlation id. A row that is edited, deleted, inserted or cut from the end is found by python -m smo_shared.audit verify (exit 1 and the first broken row); export [--since SEQ] [--format jsonl|syslog] writes the rows and a closing head line to keep outside the database. Reads, unauthenticated calls and rate-limited calls are not recorded. R1 Termination now needs the database (SMO_DATABASE_URL; compose gives it the password secret and starts it after the schema): a database that is down fails no call, it is counted in smo_audit_writes_total{outcome="failed"} and logged; R1_AUDIT=off turns recording off. Not done: the BFF/GUI view (SEC-11.6), and the chain does not protect against someone who can write the database and recompute every hash (keep the exported head elsewhere).
• A Helm chart (smo/deploy/helm/smo, PR-OPS-2): the same stack as docker-compose.yml on Kubernetes, which stays the way to run it on one machine. One template over a modules map gives each module a Deployment and a Service (named as in compose, so one release per namespace), probes from /live and /ready (a heartbeat file for the worker), the hardened pod settings of PR-SEC-13, secrets as files under /run/secrets (an rApp never gets the enrollment secret), and optional Ingress, NetworkPolicy (the mock Near-RT RIC isolation), PodDisruptionBudget and HorizontalPodAutoscaler. The database: a bundled one-pod Postgres by default, or postgres.enabled=false with postgres.external.* for a managed one. Migrations (PR-OPS-3): a Job runs scripts/migrate.py on install and as a pre-upgrade hook, so a failed migration aborts the upgrade with the old pods still serving; every module waits for the schema in an init container (migrate.py --wait SECONDS, new) and never serves on an older one. CI installs the chart on kind from images built from the checkout, runs the compose smoke scripts inside the cluster, upgrades it and uninstalls it. tests_integration/test_helm_chart.py keeps chart and compose in step. Not in the chart: netconf-lab, edge-tls (use the Ingress). Multiple replicas of a module, load balancing and HA remain the next release's work.
• Change management in the operator GUI. The config-job drawer (Infrastructure, O1 endpoints & jobs) now shows a staged job's waves and why it is halted, and an operator can Continue, Halt or Abort a halted job; Roll back… previews what would be written and which values were changed after the job wrote them, and overwrites those only when the operator chooses "Roll back anyway"; a job's KPI guard and what it found are shown. The "New CM write" form takes an optional staged rollout (wave size, pause, alarm gate, halt or undo on a failed gate) and an optional KPI guard. The KPIs page has a KPI definitions tab: define, edit and delete KPIs, add the standard set, and manage the schedules that publish a KPI to DME with what the last run did. Operators may drive and roll back jobs (attributed to them; an admin holds the MSAC tier, as for a new write); KPI definitions and schedules are admin-only. The GUI backend deliberately does not expose kpi-check, publish or advance-due.
• Safeguards page in the operator GUI: per rApp instance, whether it is stopped (and by whom and why) and its limits with the hourly use; Stop (an operator may; the reason is kept and the action is attributed to t...

SMO 0.2.0

SMO 0.2.0

Rollback, staged rollouts and KPIs, and the safeguards an rApp is held to: who may call what, how much it may change, and a record of every refusal. The schema moves from revision 0001 to 0018, all additive.

Known limitations

• The gateway's caller-role policy is a short deny-list (per-rApp limits, KPI definitions, history purge, the kill switch, safeguard events and refusals, KPI standard/publish); a route not on it is open to every valid token.
• The compose sample rApps are static services with module identities. They are not deployed through POST /instances, so manifest limits do not reach them.
• Delivering an instance's credentials to its workload is not built; the credential is issued once through POST /instances/{id}/credentials.
• The kill switch stops an rApp's writes through RAN NF OAM only. Safeguard refusals are not purged.
• The seeded KPIs are over this build's own counters, not the TS 28.554 definitions, and nothing publishes a KPI on a schedule.
• The O-RAN WG4 YANG models are not bundled (SB-4).

Added

• Standard KPIs and KPIs in DME (MGT-11.6, 11.7): POST /ran-nf-oam/kpi-definitions/standard defines six KPIs that are not defined yet (dl_prb_utilization, rrc_connected_ues_mean, dl_ue_throughput, handover_failure_rate, handover_success_rate, handover_ping_pong_rate) and keeps one an operator has edited; GET lists them without writing. They are over the counters this build carries, not the TS 28.554 definitions, which are not reproduced. POST /kpis/{name}/publish computes a KPI as GET /kpis/{name} does and delivers one record per group to every DME data job open on the type RAN.KPI.<name> (registered as RAN NF OAM's production capability), so an rApp reads a KPI as it reads any other data. Nothing runs it on a schedule. Both are internal-only at the gateway. A KPI cannot be named standard.
• Events for refusals (AI-10.6): every time the platform refuses an rApp (the kill switch, the rate, the blast radius or the magnitude limit) the refusal is recorded (GET /ran-nf-oam/safeguard-refusals, filters invoker_id, code, since) and announced through the outbox to each subscriber of POST /ran-nf-oam/safeguard-subscriptions (callbackUri, optional refusals to narrow it): eventType RAPP_SAFEGUARD_REFUSAL with refusal, invokerId, requestedBy, detail, occurredAt. The same refusal of the same rApp is announced once per SAFEGUARD_EVENT_MIN_INTERVAL_SECONDS (default 60; 0 announces every one) but every one is recorded, so a looping rApp cannot flood its watchers. Subscriptions and the record are internal-only at the gateway (they name other rApps). Schema revision 0018 (safeguard_subscription, safeguard_refusal). Refusals are not purged yet.
• Blast radius and magnitude limits (AI-10.3): the manifest's limits also takes maxElementsPerJob (managed elements one job may touch) and maxChangePercent (how far a numeric value may move in one write, in percent of its current value), pushed to RAN NF OAM with configJobsPerHour at bootstrap. PUT /rapp-limits/{id} now takes any of the three (at least one; it replaces the set). POST /config-jobs from an rApp with such a limit answers 403 RAPP_BLAST_RADIUS_EXCEEDED or RAPP_MAGNITUDE_EXCEEDED before anything is recorded or sent, dry runs included; the current value is read from the NF, and a value that cannot be read or is not a number cannot be measured and is refused too. Rollbacks and reverts are not limited. Schema revision 0017 (rapp_limit).
• Per-rApp kill switch (AI-10.4): PUT /ran-nf-oam/rapp-kill/{invokerId} (requestedBy, reason) stops an rApp: its POST /config-jobs (dry runs too) answer 403 RAPP_KILLED, and a job of its that waits between waves does not continue. Rollback, revert, halt and abort still work, because undoing must never be refused. DELETE lifts it, GET /rapp-kill[/{id}] reads it. rApp Management offers it per instance (PUT/DELETE /instances/{id}/kill, which uses the instance's oauthClientId; 503 when RAN NF OAM cannot be told). All of these are internal-only at the gateway (an rApp cannot stop or free another rApp; it may read whether it is stopped itself). Schema revision 0016 (rapp_kill).
• Caller roles at the gateway (PR-SEC-14). Before, any valid token reached every route behind R1, and an rApp's token was indistinguishable from an SMO module's (both asked for the same scope). Now SME records every invoker as internal (it presented the enrollment secret, X-SMO-Enrollment: a new compose secret, secrets/enrollment_secret, mounted by SME and every SMO module, never an rApp) or rapp; the smo-internal and smo-gui scopes go to internal invokers only (an rApp asks for smo-rapp); introspection reports the role; R1 Termination forwards it as X-R1-Role (a caller-supplied value is dropped) and refuses an rApp (403 ROLE_NOT_PERMITTED) on a deny-list of internal-only routes: PUT/DELETE /ran-nf-oam/rapp-limits/{id}, PUT/DELETE /ran-nf-oam/kpi-definitions/{name}, POST /ran-nf-oam/config-history/purge. Every other route is open to every valid token, as before; the list is INTERNAL_ONLY in smo_shared/roles.py, and a test fails if an entry names a route that no longer exists. SMO_ROLE_ENFORCEMENT is enforce by default and audit for a rolling upgrade (counted and logged, then allowed). SME refuses a wrong enrollment secret (403) and, with none configured, registration (503 ENROLLMENT_NOT_CONFIGURED) unless SME_ALLOW_OPEN_ENROLLMENT is set. Sample rApps run as SMO_IDENTITY_KIND=rapp. Schema revision 0015 (invoker_registration.kind; existing invokers stay internal).
• rApp Management registers an SME invoker for every rApp instance at create (no enrollment secret: an rApp's role), and the id is the instance's oauthClientId (before, a random UUID that matched no token). POST /instances/{id}/credentials issues the client id and secret once, in that answer only (Cache-Control: no-store), and rotates them on a repeat while the instance is DEPLOYING; terminating the instance deregisters the invoker. This is what makes the per-rApp limit of AI-10.2 apply to the rApp that holds the credentials: with the random UUID it keyed on, it never could.
• CI: the job "Upgrade from the previous release keeps its data" (scripts/check_upgrade_from_previous_release.sh) installs the newest final smo-v* release from its own source, puts data into it with its own demo runbook, stops it and starts this commit's stack on the same database, so this commit's migrate service upgrades real rows. It then checks that every table kept its rows (but the outbox and idempotency keys, which come and go), that the packages and instances are the same by id, and that the upgraded stack answers. Its control seeds a migration that passes on an empty database and fails on one with packages in it; the upgrade must refuse it and leave the schema revision as it was. Runs when a pull request changes smo/migrations or the check, and on every push to main (OPS-10.5).
• Link types in the topology: GET /topology/links lists the neighbour relations declared in the cell guards, each as INTRA_ELEMENT (both cells on one managed element), INTER_ELEMENT, AMBIGUOUS (several elements claim the neighbour's cell id; not guessed) or EXTERNAL (no managed element has it), with reciprocal (the other side declares it back), sameSectorGroup and sameIncidentZone; filters managed_element_ref (either end) and link_type. GET /topology/relation?a=<DN>&b=<DN> says how two managed objects stand in the containment tree: SAME, ANCESTOR, DESCENDANT, SIBLING, SAME_ELEMENT or DIFFERENT_ELEMENT (404 MANAGED_OBJECT_NOT_FOUND for a DN that is not in the tree). The names are this build's own, not the TEIV RAN domain model (MGT-10.2).
• Per-rApp limits: the manifest may declare limits: {configJobsPerHour: N} (under rappManifest or at the top level; a whole number from 1 to 100000; an unknown limit name or a bad value fails onboarding). When an instance finishes bootstrapping, rApp Management hands the limit to RAN NF OAM under the instance's OAuth client id (PUT /rapp-limits/{invokerId}; GET also says how many jobs the rApp started in the last hour; DELETE), and POST /config-jobs answers 429 RAPP_RATE_LIMITED with Retry-After once that caller has started N jobs in the last hour. The push fails closed: a package that declares a limit that RAN NF OAM does not accept stays DEPLOYING and bootstrap-complete answers 503 until it can be retried. Rollbacks, automatic reverts and dry-runs are not counted or refused. A caller cannot change its own limit (403 RAPP_LIMIT_SELF_CHANGE); the limit is removed when the instance is terminated. Schema revision 0014 (rapp_limit, write_config_job.invoker_id and created_at, rapp_instance.rapp_limits_set) (AI-10.1, AI-10.2).
• Revert on a KPI regression: POST /config-jobs/{jobId}/kpi-check (kpi, baselineMinutes, observationMinutes, maxRegressionPercent, direction higher/lower, minSamples, revert, force) compares a KPI over the window before the job with the window from the job on, for each element the job changed. A change worse than maxRegressionPercent is REGRESSED; with revert the regressed elements (only those) are rolled back with a rollback job (access check and changed-since guard as for /rollback). Too few samples, an undefined KPI or a zero baseline is INSUFFICIENT_DATA and nothing is reverted. It is a check to call, not a watcher: nothing runs it on a schedule (AI-10.5).
• KPIs: PUT /kpi-definitions/{name} defines a KPI as a formula over PM counters (formula, and counters: which counter feeds which variable and how its samples are combined, sum avg min max last or count), GET /kpi-definitions[/{name}], DELETE. The formula language is arithmetic, comparisons, min max abs sqrt `roun...

SMO 0.1.0

SMO 0.1.0

The first release: source and the docker compose stack. Images are not published yet (OPS-4.2); build them with docker compose up -d --build.

Known limitations

• A CM job writes each sub-change in its own NETCONF transaction, so a job whose second change fails leaves the first one applied (SB-1.10 will make it one transaction per element).
• The O-RAN WG4 YANG models are not bundled; model-based endpoints use the smo-lab profile (SB-4).
• No rollback endpoint yet: GET /managed-entities/{ref}/config-history records the values, restoring them is manual (MGT-1.6, planned for 0.2.0).
• The compose replay does not yet run over SSH (SB-1.11), and the rotation runbooks in docs/SECRETS.md other than the database password's are untried (SEC-4.8).

Added

• Filling the containment tree from a server: POST /managed-entities/{ref}/managed-objects/refresh reads a model-based ssh or tls endpoint (registered with ?model=) with a whole-container get-config and makes the tree match (new objects source=walk, objects the server no longer reports removed, registry objects kept) (SB-6.2). GET /topology exports the tree in the TEIV adapter wire shape FOCOM already uses (one ManagedObject entity per node, a MANAGEDOBJECT_CHILD_OF_MANAGEDOBJECT relationship per parent link; optional managed_element_ref filter) (SB-6.7). RAN_NF_OAM_ENFORCE_MO_TREE=true (off by default) rejects a CM sub-change whose target DN is not in the tree with MANAGED_OBJECT_NOT_FOUND, before anything is sent, in dry runs too (SB-6.5).
• A managed-object containment tree in RAN NF OAM: every registered element has its root ManagedElement=<ref> (and the function it was registered with, with its ancestors) in the new managed_object table (schema revision 0010, which also fills it from the elements already registered). Read it with GET /managed-objects/{dn}, GET /managed-objects/{dn}/children (paged) and GET /managed-objects/{dn}/subtree?depth= (nested, at most 1000 nodes). An unknown DN is 404 MANAGED_OBJECT_NOT_FOUND (SB-6.1, 6.3, 6.4, 6.6).
• NETCONF over TLS (RFC 7589) to an O1 adaptor: register it with transport: tls and adaptorUri tls://host[:port] (default 6513; the same ?model= and ?datastore= options as ssh). The service verifies the server certificate against a CA file (TLS 1.2 or later, hostname checked, no switch to skip either) and presents a client certificate: with the endpoint's credentialRef, NETCONF_CRED_<NAME>_CERT_FILE, _KEY_FILE and _CA_FILE; without one NETCONF_TLS_CERT_FILE, NETCONF_TLS_KEY_FILE, NETCONF_TLS_CA_FILE. Schema revision 0009 allows tls in o1_adaptor_endpoint.transport (SB-2.4).
• Pinned SSH host keys per endpoint: PUT /o1-adaptor-endpoints/{id}/host-keys (keyType, publicKey as in a known_hosts line, pinnedBy), GET and DELETE .../host-keys/{keyType}. The service trusts the keys pinned for an endpoint and, as before, NETCONF_SSH_KNOWN_HOSTS; a server key that is not among them, or differs from the pinned one of its type, is refused, and the only way to accept a changed key is for a named operator to pin the new one (the response says replaced: true). The key is never learned by connecting. New table o1_adaptor_host_key, schema revision 0008 (SB-2.3).
• Per-endpoint credentials for NETCONF over SSH: register the adaptor with credentialRef: "<name>" and the service connects with NETCONF_CRED_<NAME>_PASSWORD (or _PASSWORD_FILE) / NETCONF_CRED_<NAME>_KEY_FILE from its own environment or mounted secrets; the database holds only the name (new column o1_adaptor_endpoint.credential_ref, schema revision 0007). A reference that names no configured credential, or a pasted secret, is refused at registration without being echoed. Endpoints without a reference keep using NETCONF_SSH_PASSWORD (SB-2.1, 2.2, SEC-4.6).
• ?datastore=candidate on an ssh adaptor URI: each CM write is one transaction on the candidate datastore (lock, edit-config, commit, unlock); a refused edit or commit is followed by discard-changes, a refused lock changes nothing, and the step that failed is named in rejectionDetail. A server without :candidate:1.0 is refused before anything is sent. Per write, not per job (SB-1.8).
• Writes to a model-based NETCONF server: with ?model=smo-lab on the ssh adaptor URI, POST /config-jobs sends an edit-config on the model's list entry (operation as nc:operation, key leaf, kebab-case leaves) instead of the <managed-object> shape (SB-1.6). A refused change now carries the server's own <rpc-error> as rejectionDetail on GET /config-jobs/{id} (tag, path, message; rejectionReason stays NETCONF_RPC_FAILED); new column write_config_sub_change.rejection_detail, schema revision 0006 (SB-1.7).
• /metrics now also counts outbound calls: smo_outbound_calls_total{client,target,method,outcome} and smo_outbound_call_duration_seconds{client,target}. client=r1 is a call through R1 Termination (target is the module, outcome is 2xx..5xx, timeout or error); client=webhook is a callback to a registered destination (target=callback, plus outcome=blocked when the SSRF guard refused it). Callback hosts are deliberately not a label (OBS-2.6).
• A model-based NETCONF server can be read: register the ssh adaptor with ?model=smo-lab on its URI (ssh://netconf@host:830?model=smo-lab) and GET /managed-entities/{ref}/config sends a subtree get-config on that YANG model and returns its leaves as camelCase attributes; without ?model= the <managed-object> shape is sent as before. Writes to such an endpoint are refused until SB-1.6 (SB-1.5).
• R1_PUBLIC_BASE_URL: with the TLS profile, set it to the HTTPS door (for example https://localhost:8443) and /bootstrap advertises https://.../sme/... addresses, with the OAuth2 token endpoint forwarded by the edge, instead of http://sme:8000; unset, nothing changes (SEC-1.6).
• /metrics now also carries smo_fsm_transitions_total and smo_fsm_illegal_transitions_total (every lifecycle state change, by machine, state and event) and the database pool gauges smo_db_pool_connections{state} and smo_db_pool_capacity (OBS-2.4, 2.5).
• docker compose --profile netconf-lab up -d netconf-lab: a real NETCONF-over-SSH server (Netopeer2) with a small model, on 127.0.0.1:8830, for trying the SSH transport (netconf-lab/README.md; development only) (SB-1.4).
• CM change history: every dispatched write records the values it replaced and the values it wrote (cm_snapshot, schema revision 0004); GET /managed-entities/{ref}/config-history lists them. Each write now makes one extra read of the NF first (up to 30 s on an unresponsive one); RAN_NF_OAM_CM_SNAPSHOTS=false turns it off (MGT-1.1–1.4).
• dryRun: true on POST /config-jobs: every check runs, nothing is sent or stored, and the response says per change whether it would pass (MGT-3).
• NETCONF over SSH to an O1 adaptor: register it with transport: ssh and adaptorUri ssh://user@host[:port]; CM writes and GET /managed-entities/{ref}/config then use RFC 6242 sessions. Set NETCONF_SSH_KNOWN_HOSTS and NETCONF_SSH_PASSWORD (or _FILE) / NETCONF_SSH_KEY_FILE for RAN NF OAM. Schema revision 0003 adds o1_adaptor_endpoint.transport (default http-mock) (SB-1.1–1.3).
• Structured JSON logs on stdout in every service, one access line per request with the route template, status and duration, secrets scrubbed; LOG_LEVEL (#207).
• GET /metrics in every service (Prometheus): request count and latency by method, route template and status. Container network only: R1 Termination and the TLS edge do not forward it (#208).
• Alembic schema migrations: baseline revision 0001 is migrations/001_init.sql; python scripts/migrate.py upgrades, and stamps a database that was created from the file (#209).
• Optional TLS edge (docker compose --profile tls up, nginx on 3443 and 8443, development certificates from scripts/make_dev_certs.sh) (#206).
• Secrets inventory (docs/SECRETS.md), *_FILE variants for secrets, and the database password as a compose secret (scripts/init_secrets.sh) (#205).
• Request body cap (R1_MAX_BODY_BYTES) and a per-caller rate limit (R1_RATE_PER_SECOND, R1_RATE_BURST) at R1 Termination (#204).
• Database backup and restore scripts (scripts/db_backup.sh, db_restore.sh) (#200); slow-statement log in compose (POSTGRES_SLOW_QUERY_MS) (#201).
• Route-table authorisation walk: every backend route is refused without a valid token (#202).
• Liveness and readiness probes (/live, /ready; /health kept), single-runner guard for periodic work, idempotency keys on command routes, optimistic concurrency on lifecycle rows, one SME identity per module across replicas, connection pool limits and graceful shutdown (#190–#198).

Changed

• Every service's root filesystem is read-only, with /tmp as a memory-backed scratch directory (SEC-13.2). The runbook copies rApp CSARs into the new smo_scratch volume at /srv/scratch on r1-termination instead of /tmp; if you had your own docker compose cp ... :/tmp/... step, point it there. A service that wrote anywhere else would now fail loudly.
• DME's stop-job request to each producer (when a data job is deleted) is written to the transactional outbox in the same transaction as the deletion and sent after it commits, so a crash no longer leaves a producer running a deleted job; at-least-once, so a producer may see the DELETE twice. Schema revision 0005 adds notification_outbox.method (default POST) (MSG-1.10).
• PATCH /alarms/{id}/ack and /clear return 404 ALARM_NOT_FOUND for an unknown alarm (they failed with a 500), and new_state must be ACKNOWLEDGED or UNACKNOWLEDGED (422 otherwise) (MGT-8.1).
• CM writes are checked against each leaf's type and constraints before anything is sent. A write with an out-of-range number (localPortNumber 70000), the wron...