Skip to content

Release v3.11.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Nov 05:12
· 4 commits to main since this release
v3.11.0
This tag was signed with the committer’s verified signature.
tyronechrisharris
SSH Key Fingerprint: liRyKMwW51OJnMtng6FeXr+1hgg0Uu4yi/euvQ8llec
Verified
Learn about vigilant mode.
e69efaa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release overview

OSCAR 3.11.0 is the first official release published under Oak Ridge National Laboratory's github.com/ORNL organization (ORNL/oscar), re-architected for multi-year, 24/7 operation across up to 50 concurrent radiation portal monitor (RPM) and RTSP camera lanes on Ubuntu Server 24.04 x86-64 and Windows 11 x86-64 (with Docker Desktop and WSL 2). Because v3.8.0 (2026-09-19) was the last general-public release prior to the interim 3.8.1–3.10.0 builds, the New features since public release v3.8.0 section below summarizes every capability added across 3.8.1 through 3.11.0 in one place.

Release artifacts — which file to download

Release Asset Target Audience & Use Case
oscar-3.11.0.zip (+ .sha256) Connected Ubuntu Server 24.04 or Windows 11 installations and upgrades. Includes Docker Compose definitions, Linux/Windows administration CLIs (oscar.sh / oscar.bat), configure-windows-host.ps1, application build context, all Markdown documentation, and all four localized PDF operator manuals.
oscar-3.11.0-windows-x86_64-offline-full.zip (+ .sha256) Air-gapped / offline Windows 11 installations where Docker Desktop or WSL 2 is not yet installed. Bundles pinned Docker Desktop 4.87.0 and WSL 2.7.3 installers, pre-built container images (oscar:3.11.0, oscar-postgis:3.11.0, nginxinc/nginx-unprivileged:1.28.1-alpine), administration scripts, and all Markdown/PDF documentation.
oscar-3.11.0-windows-x86_64-offline-images.zip (+ .sha256) Air-gapped / offline Windows 11 installations or upgrades where compatible Docker Desktop and WSL 2 are already running. Contains the pre-built container image archive, administration scripts, and documentation without the Windows prerequisite installers.
OSCAR-3.11.0-Administrator-Operator-Manual-{en,es,fr,el}.pdf Standalone localized Administrator and Operator Manuals in English (en), Español (es), Français (fr), and Ελληνικά (el).
oscar-v3.11.0-source.zip (+ .sha256) Complete tracked source archive including all submodules (osh-core, osh-addons, osh-oakridge-modules, and oscar-viewer; GitHub's auto-generated source archive omits submodules).
OSCAR-3.11.0-SHA256SUMS.txt Master SHA-256 checksum manifest covering all release ZIPs and localized PDF manuals.

Start here — documentation for new and current users

New features since public release v3.8.0 (v3.8.0 → v3.11.0)

For users upgrading or evaluating OSCAR from v3.8.0 (2026-09-19), the following user-facing and operational capabilities were introduced across 3.8.1, 3.8.2, 3.9.0, 3.9.1, 3.10.0, and 3.11.0:

  • Alarm Adjudication, EML Non-Threat Automation & Event Log Filtering:
    • Admin-Uploaded EML Runtime & Automated Code 9999: EML Non-Threat Disposition (3.11.0): Hot-loadable external EML runtime plugin upload (Services -> OSCAR Service -> EML Plugin Path, stored privately in /var/lib/oscar/eml-plugin/ with zero third-party EML artifacts redistributed in public repositories or release bundles). Automatically evaluates Rapiscan VM-250 GX spectra prior to publication, suppresses verified EML RELEASE non-threat alarms from the Dashboard and browser notifications at t = 0 while keeping INVESTIGATE alarms prominent, persists durable Code 9999: EML Non-Threat adjudications with EML_METADATA JSON, and renders localized EML Automated Analysis cards in Event Details. Fresh installs without an uploaded EML JAR require no extra files and run with full core portal monitoring capabilities.
    • Custom Disposition Codes (3.10.0) & Full-Database Adjudication Code Filtering (3.11.0): Configure site-specific custom disposition codes in SensorHub Admin, display an optional Adjudication Code column on the Events page, and filter the full hot PostgreSQL + cold Parquet history by Adjudication Code (equals, does not equal, is any of, is empty, is not empty) across built-in Codes 1–11, Code 9999: EML Non-Threat, and custom codes (backed by indexed "code:<n>" markers and an asynchronous zero-blocking startup backfill for upgraded systems).
    • Priority Alarm Ordering (3.10.0), Nested AND/OR Filters & Bulk Adjudication (3.9.0, 3.9.1): Automatic alarm priority sorting (Gamma-Neutron > Neutron > Gamma), full-database server-side nested AND/OR filter groups (with inclusive numeric Occupancy ID, Max Gamma, and Max Neutron comparisons), cross-page and all-filtered selection with bounded-concurrency Bulk Adjudication, direct vehicle ID entry during Dashboard adjudication (3.9.0), and deduplicated live + historical adjudication logs (3.11.0).
    • Event UUID Deep Links & Air-Gapped QR / File Alarm Transfer (3.9.1, 3.10.0): Exact Event UUID lookup from the Events page, shareable node-qualified Event Details URLs (Share Occupancy), visible occupancy observation IDs in Event Details and PDF exports, single-code compressed Base45 + SHA-256 QR code and portable alarm package export (preserving chart endpoints, extrema, and threshold crossings), and an offline Alarm Transfer page with live camera QR scanning and file import.
  • Multi-Gate Operations, Status of Health, Individual Detector Charts & Map Resilience:
    • URL-Scoped Operational Views (3.9.1, 3.11.0): Workstation-specific URLs (/view/<slug>) driven by backward-compatible lane CSV keywords (OperationalViews) scope lane discovery, Dashboard queues, maps, Event Log queries, National Statistics, reports, Status of Health, and browser alarm notifications to assigned portal groups, with fail-closed access checks on out-of-scope events.
    • Status of Health Page & Compact Dashboard Fault Indicators (3.8.2, 3.9.1, 3.11.0): Dedicated localized Status of Health page and compact per-lane Dashboard badges tracking independent RPM and camera connectivity plus Gamma High, Gamma Low, Neutron High, Tamper (with deterministic Clear baseline on connect), and configurable Extended Occupancy faults across Rapiscan, Aspect, and RS350 lanes using bounded 30-second ConSys lateral/DuckDB health batches and independent fault-isolated lane loading (3.8.2).
    • Individual Detector Charting, Live/Historical Daily File Views & Site Diagram Fixes (3.8.1, 3.10.0, 3.11.0): Switch between summed and individual gamma/neutron detector traces in both live Lane View and historical Event Details (including first-point Rapiscan neutron capture), inspect raw RPM Daily File messages on historical events and the latest 15 live messages in Lane View, and reliably render OpenStreetMap + uploaded site-diagram bounds (3.8.1).
  • Multi-Camera Live RTSP/HLS Video & Five-Frame Retained Alarm Media:
    • Loopback MediaMTX RTSP Fan-Out, Multi-Camera Lanes & Multi-Viewer HLS Leases (3.9.1, 3.10.0, 3.11.0): Embedded loopback mediamtx (1.12.2) deduplicates identical camera streams to strictly one upstream :554 TCP socket per camera per OSCAR server, pairs with infinite (-1) camera reconnects (3.10.0), dedicated bounded video executor pools, keyframe-gated HLS startup with localized retry controls, multi-camera lane feed switching, and per-viewer HLS session leases (X-OSCAR-HLS-Viewer) so multiple operators can watch the same camera concurrently.
    • Five-Frame JPEG Evidence Gallery with 24-Hour MP4 Grace (3.9.1, 3.11.0): Generation-safe keyframe-verified MP4 alarm recording followed by asynchronous extraction of five deterministic sRGB JPEG frames (FIRST, QUARTER, MIDDLE, THREE_QUARTERS, LAST) per camera, an authenticated Viewer five-frame evidence gallery (with legacy MP4 playback fallback), a 24-hour MP4 rollback grace window, and 30-day secondary-frame pruning retaining frame-03.jpg (MIDDLE) long-term.
  • Multi-Year 50-Lane Storage (PARQUET_PRIMARY), Daily-File Journaling, Backup & Legal Holds:
    • PARQUET_PRIMARY Tiered Storage Engine (3.11.0): Stages immutable high-volume 1 Hz detector streams (gammaCounts, gammaThreshold, neutronCounts, setupGamma, setupNeutron) into 30-minute LOGGED PostgreSQL partitions (osg_*), seals parity-verified windows into Zstandard Apache Parquet files (~10–15x smaller on disk), retires stage partitions via O(1) NO INHERIT + DROP TABLE, and federates hot PostgreSQL + cold Parquet queries transparently through embedded DuckDB with CQL2 != and JSONB array pushdown.
    • Pre-Parse Daily-File Journal & Dual UTC/Local Timestamps (3.10.0, 3.11.0): DailyFileJournalService captures CRC32C-framed raw Rapiscan, RS350, and Aspect messages on disk before parsing, publishes immutable legacy-compatible daily CSVs with UTC and wizard-configured local timestamps plus SHA-256 manifests, and bounds sealed journals after 30 days.
    • Opt-In Single-Snapshot Backup, Restore, Legal Holds & Serialized Statistics (3.11.0): Defaults to OSCAR_BACKUP_MODE=disabled (archive_mode=off to prevent unattended WAL disk exhaustion) with opt-in single mode (pg_basebackup, pg_verifybackup, OSCAR state, cold Parquet archive, AES-256-CBC encrypted credentials, and HMAC-SHA-256 CURRENT verification), transactional observation/generation legal holds (oscar_retention_hold), audited two-step cold deletion, portable Parquet export packages, and serialized National Statistics refreshes.
  • Complete 4-Language Localization (en, es, fr, el) & Published PDF Manuals (3.9.0–3.11.0):
    • Extends v3.8.0's initial i18n foundation across all 530 Viewer UI keys, /login authentication, OpenSensorHub Vaadin Admin ConfigMessages, live/historical alarm statuses, custom disposition codes, and four screenshot-complete localized Administrator & Operator Manual PDFs published with every GitHub Release (0 px horizontal overflow at 1920x1080 and 1280x800).
  • Ubuntu 24.04 & Unattended Windows 11 Deployment, Air-Gapped Bundles & Security Hardening (3.9.0–3.11.0):
    • Connected (oscar-3.11.0.zip) and air-gapped Windows 11 (offline-full and offline-images) bundles, configure-windows-host.ps1 (prepare-host / host-check across 20 automated checks including Modern Standby VIDEOIDLE = 0, pre-login S4U Docker Desktop boot startup, .wslconfig memory caps, and bugcheck auto-reboot), LAN/Tailscale SAN TLS auto-generation, Sentry.io removal (3.9.0), Next.js 16 + React 19 upgrade with zero npm audit advisories (3.9.1), browser autoplay resilience (3.9.0), and automated 50-lane + 6-physical-camera end-to-end certification.

Why the major architectural changes in 3.11.0 were implemented

  • Why PARQUET_PRIMARY Tiered Storage Replaced Single-Table PostgreSQL Detector Retention: At a 50-lane site emitting 1 Hz gamma/neutron telemetry, single-table PostgreSQL JSONB retention generated millions of rows per day and severe MVCC dead-tuple bloat, autovacuum I/O contention, WAL amplification, and multi-gigabyte B-tree/GIN index fragmentation during row-by-row retention deletes. Staging immutable detector streams into 30-minute LOGGED PostgreSQL partitions (osg_*), sealing parity-verified windows into Zstandard-compressed Apache Parquet files (~10–15x smaller on disk), retiring whole stage partitions via O(1) NO INHERIT + DROP TABLE, and federating hot + cold reads through embedded DuckDB eliminates DELETE bloat, keeps PostgreSQL compact (~9–12 GiB), speeds up multi-day historical CQL/report scans ~19x, and preserves 100% OpenSensorHub IObsSystemDatabase / ConSys / CQL2 compatibility.
  • Why Five-Frame JPEG Media Replaced Long-Term Raw MP4 Storage: Retaining full H.264 MP4 clips across 50 cameras consumes terabytes rapidly, while video re-encoding was CPU-heavy and brittle on 1-second occupancies. Asynchronous extraction of five deterministic baseline sRGB JPEGs (FIRST, QUARTER, MIDDLE, THREE_QUARTERS, LAST) with a 24-hour MP4 rollback grace and 30-day secondary-frame pruning (keeping MIDDLE / frame-03.jpg long-term) preserves visual evidence at a small fraction of MP4 disk usage.
  • Why Direct Pre-Parse Daily-File Journaling Was Added: Reconstructing vendor daily CSVs from parsed database rows risked losing raw messages across restarts or parser errors and caused startup spikes on 50-lane systems. DailyFileJournalService records CRC32C-framed vendor messages on disk before parsing and decouples historical day rendering from live ingestion.
  • Why Schema-v2 Single-Snapshot Backup (archive_mode=off) Replaced Continuous External WAL Archiving: Continuous external WAL archiving risked silent disk exhaustion on unattended servers if backup storage disconnected, while multiple local cold-tier copies multiplied storage cost. Opt-in single mode uses streamed pg_basebackup, pg_verifybackup, HMAC-SHA-256 inventory verification, and an atomic CURRENT pointer for self-contained same-host or cross-host recovery.
  • Why Operational Views, Batched Status of Health, Event UUID/QR & Serialized Statistics Were Built: Multi-gate sites needed URL-scoped workstation views (/view/<slug>) so gate consoles subscribe only to assigned lanes; bounded 30-second health batching and keyframe-gated HLS recovery so 50-lane dashboards and cameras never stall browsers; single-code QR export/import for air-gapped alarm handoff; and serialized National Statistics refreshes so archive scans cannot overlap.
  • Why Windows 11 Modern Standby (S0 Low Power Idle) & Pre-Login Docker Hardening Were Added: Endurance testing on Windows 11 Modern Standby hardware proved that AC display blanking (VIDEOIDLE > 0) triggers Kernel-Power 506 idle suspension even when STANDBYIDLE = 0, and that Docker Desktop does not start after an unattended power-loss reboot until interactive login. configure-windows-host.ps1 (prepare-host / host-check) now enforces VIDEOIDLE = 0 on Modern Standby systems, registers OSCAR-DockerDesktop-BootStart (LogonType=S4U) for pre-login container startup, caps WSL 2 memory via .wslconfig, verifies NIC IPv4 stability, and enables bugcheck auto-reboot across 20 automated checks.
  • Why External EML Runtime Plugin Upload, Automatic EML Non-Threat Adjudication & Adjudication Code Filtering Were Added: Separately licensed EML libraries and models cannot be redistributed in public open-source repositories, while benign NORM alarms on high-throughput Rapiscan VM-250 portals otherwise flood the Dashboard queue. A hot-loadable EmlRuntimeRegistry (emlPluginPath in /var/lib/oscar/eml-plugin/) suppresses verified EML RELEASE non-threat alarms from the Dashboard and notifications at t = 0, records durable Code 9999: EML Non-Threat adjudications with EML_METADATA JSON, and enables indexed full-database Adjudication Code filtering across built-in Codes 1–11, Code 9999, and custom disposition codes (with an asynchronous startup backfill for existing deployments).

Storage-profile changes and rollout

  • Introduce the OscarTieredObsSystemDatabase storage profile with sequential observation IDs, batched writes, compact short-lived stage ownership, and a versioned PostgreSQL archive catalog
  • Initialize new PostgreSQL clusters with data checksums, WAL compression, I/O timing, and pg_stat_statements, while explicitly keeping archive_mode=off and omitting any external WAL bind; an existing cluster is not recreated by upgrade
  • Preserve an upgraded deployment's saved config.json and database, so a pre-3.11.0 deployment is not silently converted to the complete 3.11.0 storage profile
  • Require a fresh 3.11.0 database and standard configuration for the current test systems and new production deployments; preserve any required pre-3.11.0 production history through a separately validated migration
  • Make fresh 3.11.0 deployments explicitly use PARQUET_PRIMARY for the five allowlisted immutable high-volume outputs: route each accepted value into a LOGGED, UTC-aligned 30-minute PostgreSQL staging partition, reject detector timestamps more than five minutes ahead of the database receipt clock, and run the Parquet-primary worker every minute with a five-minute closed-window safety delay
  • Publish Parquet, schema, and manifest objects before manifest-last catalog activation; record exact ownership in a compact short-lived ledger, re-hash the active Parquet object and require exact row/UUID parity, then transactionally detach and drop the complete staging relation instead of retaining raw detector rows for a three-day overlap or deleting them row by row; retire at most eight staging partitions per cycle
  • Keep Parquet and published daily files as the durable long-term representations of allowlisted detector data while retaining mutable occupancy/alarm workflows, adjudications, reports, small OSH metadata, archive control records, overlays, and tombstones in PostgreSQL; the normal healthy staging budget is at most 60 minutes but is an operational SLO rather than a hard deadline during backlog or failure
  • Omit the generic observation-payload GIN index and heavy event indexes from fresh PARQUET_PRIMARY staging children, while keeping lean routing/parity indexes on those short-lived relations and purpose-built event indexes on the PostgreSQL parent for mutable workflows; ordinary PostGIS, HOT_RETENTION, and existing schemas are unchanged
  • Replace aging-video keyframe reduction with an additive exact-five JPEG retained-media format; keep the legacy occupancy videoPaths field and Viewer playback fallback for old/unconverted events
  • Make completed alarm video immediately eligible for asynchronous five-JPEG conversion on the next retention scan, preserve the source MP4 for 24 hours after verified publication, and keep every failed/fewer-than-five-frame source intact
  • Make new self-signed deployments LAN/Tailscale-ready by adding active non-container LAN IPv4, stable Tailscale IPv4/IPv6, and discoverable MagicDNS identity to certificate SANs, accepting explicit additional DNS/IP SANs, validating every requested identity, and printing covered client URLs plus a verifiable public-certificate fingerprint

Statistics reliability and resource control

  • Serialize scheduled and operator-requested full National Statistics publications, use fixed-delay scheduling, and retain cancellable datastore shutdown so long archive-backed scans cannot overlap and multiply CPU, memory, or scratch-space pressure
  • Return a terminal COMPLETED command only after a newly persisted statistics observation is available, preserve legacy statistics timestamp decoding without double-counting history, and use strictly increasing millisecond cutoffs with open-lower incremental ranges
  • Keep the Viewer busy until the durable command status is terminal, fetch the exact returned observation instead of an older latest result, fail visibly on terminal command errors, and cancel stale UI work when the node or operational-view scope changes
  • Execute operational-view range scans sequentially, disable duplicate refresh controls while initial or manual work is active, and regression-test the exact command, observation, rendered counts, and resource-safe lifecycle

Storage and data lifecycle

  • Add a crash-recoverable, per-lane direct daily-file journal that captures Rapiscan and RS350 input before parsing and Aspect register snapshots before observation publication
  • Make the daily journal a required driver dependency by default so startup or append failures cannot silently bypass source-record capture
  • Frame journal records with timestamp, length, and CRC32C; bound unforced data by time, record count, and byte count; recover only incomplete tails and fail visibly on complete-frame corruption
  • Decouple journal readiness from historical backlog recovery: recover the current lane before its first append, process older lane/days in bounded background batches without holding live writer locks, and skip re-rendering complete durable publications so a restart cannot time out every RPM or repeat a large CPU/I/O spike
  • Durably create every journal and retained-media directory component with parent-directory synchronization; surface permission and filesystem failures instead of treating them as platform fallbacks
  • Make daily-journal shutdown bounded and fail-visible: service stop first closes admission, drains active sensor writer handles while allowing their final records and keeping periodic fsync active, then stops maintenance and forces/closes every lane; a timed close fails module shutdown instead of silently abandoning an unclosed writer
  • Mark Rapiscan, RS350, and Aspect readers as stopping before transport teardown and join source workers before releasing their journal handles, so expected disconnects do not produce false errors and a final source record already in durable I/O cannot race sensor shutdown; RS350 releases its source writer independently of deferred WebID cleanup
  • Serialize supported stop, restart, upgrade preparation, and restore shutdown as gateway, then OSCAR, then PostgreSQL on both Ubuntu and Windows; PostgreSQL remains available through OSCAR's bounded writer/module drain instead of receiving a concurrent Compose stop signal
  • Publish immutable legacy daily CSVs with exact time-zone windows, SHA-256 manifests, atomic commit ordering, and a manifest-gated compatibility-observation cleanup path
  • Retry the bounded daily-file cleanup on every hourly maintenance pass after the midnight attempt, isolate it from unrelated purge failures, and retain all unsealed current-day compatibility rows until manifest-validated publication
  • Bound private daily-journal storage even when managed backup is disabled: after 30 days from the later of day close or publication, freeze and delete a sealed day only after immutable CSV/manifest/ledger validation and exact journal-to-CSV replay; retain an opt-in accepted-backup gate for sites that require it
  • Replace startup and once-per-minute all-history journal enumeration with durable bounded lifecycle work queues, and recover interrupted staged deletion without discarding a late or unverified segment
  • Batch observation inserts with immediate DataBlock snapshots, prepared statements, PostgreSQL batch rewriting, and one statement-level datastream-range update per batch
  • Correct observation natural-key uniqueness for null fields, make exact replays idempotent and same-key corrections last-write-wins under the stable OSH ID, route archived corrections through generation-owned overlays even after primary-stage retirement, fail index creation visibly when a legacy database contains ambiguous duplicates, and use the full expression index for natural-key conflict lookups even under PostgreSQL generic prepared plans
  • Preserve the sequential observation-ID high-water from every cataloged archive state so a restart cannot reuse public IDs after a cold copy is cataloged and the corresponding primary stage is retired
  • Export configured high-volume detector streams from a read-only repeatable-read snapshot into Hive-partitioned, sorted, ZSTD Parquet generations with gzip OSH datastream-schema snapshots
  • Prevent dense-window Parquet retries from exhausting DuckDB memory by clearing bounded 500-row insertion batches and publishing memory-derived deterministic same-window prefixes; retries reuse the failed job (including exact legacy sub-hour windows), empty reclaimed attempts terminate as audited SUPERSEDED no-ops, later prefixes become ordinary revisions, and no unverified stage child is dropped
  • Verify Parquet count/ID range before activation, publish manifests last, and atomically record exact ownership in a compact temporary ledger so late arrivals remain eligible for later generations
  • Serialize archive activation/sealing against ingestion, updates, removals, overlays, tombstones, and stage detach/drop with transaction-scoped store and ordered observation locks, while keeping file export outside the critical section
  • Merge PostgreSQL and catalog-authorized Parquet reads behind the OSH observation-store contract, preferring the staged overlap and applying archived overlays and tombstones without rewriting immutable files
  • Push mutation-free explicit-stream resultTime=latest queries into indexed PostgreSQL lateral probes and DuckDB latest-per-stream reduction before merging the two tiers, so health refreshes read at most one staged and one cold candidate per stream instead of paging the complete detector history
  • Implement PostgreSQL feature-parent lookup through the standard OSH datastore contract, so lane-scoped retention and purge code resolves the authoritative lane system instead of silently falling back to a child RPM identity
  • Preserve open and closed temporal bounds through archive candidate pruning and DuckDB pushdown so cold selection, count, limit, and filtered removal match hot PostgreSQL semantics
  • Add exact hot/cold row-digest parity receipts and whole-child primary-stage retirement; retain configurable rollback grace, bounded exact-ID deletion, and deferred sealing only for HOT_RETENTION compatibility, while generation/observation holds protect authoritative cold data
  • Add versioned archive jobs, generations, objects, ID blocks, schemas, locators, overlays, tombstones, holds, verifications, deletion plans/ledger, media records, and backup epochs
  • Compact processed-observation and deletion metadata behind restart-safe cursors so long-running systems do not accumulate unbounded bookkeeping rows
  • Add embedded DuckDB query pushdown over explicit catalog-authorized file lists and portable generation export for independent Parquet inspection
  • Bound every embedded DuckDB export, parity, deletion, query, metadata, and count connection to production-configurable memory, thread, and per-operation spill limits; add fair process-wide session admission with a bounded fail-busy wait; confine sensitive scratch to private disjoint directories and remove it on normal close
  • Keep general-purpose container temporary storage noexec while routing JVM native-library extraction to a bounded, owner-only executable tmpfs that is verified before startup
  • Preserve 64-bit Admin configuration values with type-correct Long, Integer, Short, and range converters, including the 2 GiB archive-generation limit, instead of narrowing every whole number through a 32-bit converter
  • Bound Viewer health snapshots to ten stream IDs per request with two global workers and a ten-second per-batch deadline, preventing large accumulated databases from leaving every RPM and camera indicator stuck at Waiting
  • Persist the successfully uploaded lane-CSV bucket key in the OSCAR Service configuration, so Apply Changes plus the global Save action recreates live Lane, RPM, and camera modules after a supported restart instead of leaving only historical system records in PostgreSQL
  • Serialize the legacy ERNIE VM250 XML-schema initialization across Rapiscan lanes, preventing 50-lane Windows restarts from racing the process-global registry and leaving apparently Started RPM modules with an unusable analyzer
  • Replace bundled EML libraries and model configs with an administrator-uploaded runtime plugin (Services -> OSCAR Service -> EML Plugin Path, stored privately in /var/lib/oscar/eml-plugin/, extracted onto /tmp/oscar-eml-runtime, and hot-loaded via an isolated URLClassLoader), enforce fail-closed packaging checks against redistributing EML artifacts, suppress EML RELEASE non-threat alarms from the Dashboard and browser notifications, and automatically record Code 9999 (EML Non-Threat) adjudications with structured EML_METADATA JSON queryable via CQL2 (adjudicatedIds='eml-non-threat') across PostgreSQL GIN and DuckDB Parquet and inspectable in Event Details across all four locales (en, es, fr, el)
  • Accept provisional Java Instant.MAX (+1000000000-12-31T23:59:59.999999999Z) rollbackDeleteAfter timestamps in Viewer retained-media validation and downgrade benign browser/MQTT client disconnect EOF exceptions in RestApiServlet and WebSocketProxy from ERROR/WARN to DEBUG

Media retention

  • Serialize recording commands in a bounded per-camera FIFO and use generation-qualified filenames so reconnects and back-to-back occupancies cannot finalize or reuse another recording's clip
  • Acknowledge occupancy MP4 OPEN only after the current recording generation has successfully muxed its first keyframe; bound readiness to five seconds, wake on stop/failure, reject stale generations, and remove the hidden partial instead of producing a zero-packet alarm clip
  • Bind every published recording to its exact lane, camera, and alarm; reject blank, duplicate, cross-camera, stale-generation, or otherwise unowned paths instead of attaching them heuristically
  • Cancel non-alarm recordings and fail closed on close timeouts, invalid output, or publication failure, while retaining authoritative late-close cleanup so delayed camera work cannot leave an orphaned clip
  • Make camera, occupancy, and state-listener delivery race-safe so lifecycle transitions cannot lose, duplicate, or leak recording callbacks
  • Retain exactly five distinct decoded positions (FIRST, QUARTER, MIDDLE, THREE_QUARTERS, and LAST) from each eligible alarm camera clip
  • Give short saved alarm recordings a bounded post-roll until at least five video packets have been muxed, so one-second occupancies remain eligible for five-frame retention
  • Encode each retained asset as an independently decodable opaque baseline sRGB JPEG with configurable quality (85 by default), SHA-256, dimensions, decoded ordinal, source PTS, and lifecycle metadata
  • Publish frames before a versioned manifest commit marker, use a durable identity-keyed job ledger for idempotent retry, and preserve the source MP4 on every failure or fewer-than-five-frame result
  • Fail closed unless retained media can resolve the authoritative urn:osh:system:lane:* parent, preventing a camera set from being committed under an RPM UID
  • Bind each retention scan to a cancellable PostgreSQL operation and honor interruption between bounded maintenance items, allowing the supported OSCAR-before-PostgreSQL restart sequence to drain a historical backlog without leaving a database query running
  • Preserve a successfully converted source for a configurable 24-hour rollback grace and delete it only on a later verified, unheld lifecycle pass
  • Set the production profile to zero-day five-JPEG eligibility, scan asynchronous work in bounded five-minute windows, cache occupancy datastream IDs for 15 minutes, and persist bounded late-arrival plus repeating historical reconciliation cursors so restarts, delayed commits, and clock corrections do not skip eligible media
  • Add an authenticated read-only event media API and Viewer five-frame gallery, with 404 fallback to unchanged legacy video playback
  • Enforce streaming catalog/asset transfer limits plus bounded retained-media camera, frame-byte, dimension, decoded-pixel, and event-wide budgets in the Viewer; fetch only each camera's selected frame and limit event-wide frame-request concurrency
  • Restrict storage-pressure cleanup to complete cataloged media sets, honor holds and source grace, publish DELETING before removal, and leave legacy video, daily files, adjudications, reports, and archives untouched

Backup, deletion, and operations

  • Default fresh deployments to OSCAR_BACKUP_MODE=disabled; require explicit single opt-in and an operator-selected safe absolute non-overlapping backup path before backup, verification, restore, or the retired prune command can run; recommend, but do not require, a separate physical drive for failure isolation
  • Recover a unique fully accepted first-backup snapshot by reconstructing authenticated CURRENT; preserve ambiguous accepted sets plus schema-v1, failed, partial, unauthenticated, unaccepted, and incomplete deletion candidates for operator review, and fail closed on unsafe or acceptance-invalid CURRENT
  • Add coordinated schema-v2 snapshot commands for a streamed PostgreSQL base backup, OSCAR state, primary Parquet cold tier, release metadata, and AES-256-CBC/PBKDF2-encrypted deployment credentials; omit cumulative wal-archive.tar.gz
  • Automatically verify each new snapshot with SHA-256 and pg_verifybackup, retain versioned backup-epoch manifests and VERIFIED markers, and commit exactly one logical current snapshot through a four-line HMAC-authenticated CURRENT pointer
  • Preserve explicit schema-v1 verification/restore compatibility, authenticating its required historical WAL payload but ignoring that external payload during restore because streamed postgres/pg_wal is sufficient, without treating legacy media as schema v2 or exposing a PITR workflow
  • Authenticate each backup inventory with HMAC-SHA-256, stream the bounded canonical backup-time archive-generation UUID inventory, bind the catalog receipt to its count, byte length, and SHA-256 without duplicating a large UUID array in PostgreSQL, require the exact 3.11.0 epoch and image identities, and issue daily-journal confirmation receipts only after the recovery point has passed structural and cryptographic verification
  • Authenticate the exact top-level backup payload inventory, require an explicit member allowlist for the release-configuration archive, reject duplicate names, path aliases, prefix collisions, links, reparse points, traversal, and out-of-root paths in every archive, and serialize every storage-administration mutation with a cross-command lock
  • Publish backup and portable-export directories through a compiled Linux renameat2(RENAME_NOREPLACE) helper after durability barriers; stamp the helper's packaged-source SHA-256 into the image and regression-test collision preservation
  • Keep coordinated snapshots outside every long-running service mount, give a network-scoped pg_basebackup --wal-method=stream client only its fresh staging directory, and exclude only verified image-owned state links from the mutable-state snapshot
  • Preserve legacy WAL roots/data without mounting them into PostgreSQL or deleting them during setup/upgrade mode changes
  • Resolve the Ubuntu release directory physically, reject symlinked launchers, and reject Windows launcher/release reparse points while canonicalizing storage-root ancestors, so an alias cannot bypass destructive-operation containment checks
  • Restore PostgreSQL only through the selected online base backup's consistency endpoint, promote onto a fresh timeline, reconstruct the exact VERIFIED epoch receipt, and prove a post-promotion write across checkpoint and clean restart with archive_mode=off
  • Preserve the existing Docker volume and bind-mount root metadata while restoring OSCAR state and cold archives by skipping each archive's root member, preventing otherwise valid restores from failing when the container runtime rejects timestamp or mode changes on a mounted root
  • Reconcile the entire cold-archive root against the catalog and approved datasets/ shape, including an otherwise empty/missing datasets tree, so unexpected top-level objects fail status, verification, and backup before publication
  • Re-verify every payload against the bundle's exact SHA-256 inventory and always load the checksum-verified offline-images.tar during offline initialization or upgrade, even when same-version tags already exist locally, so stale development images cannot shadow the selected bundle; the co-packaged unsigned manifest provides integrity but not publisher/origin authentication
  • Create persistent Docker volumes through setup, label new volumes with a deployment UUID, reject missing, foreign, or mixed ownership, and require an explicit audited adoption receipt before reusing unlabelled legacy volumes; declare the exact names external to Compose so upgrades never offer to recreate adopted data while reconciling immutable labels
  • Force-recreate all service containers during a clean-directory upgrade so unchanged services cannot retain release-local secret/config mounts or Compose provenance from the previous directory, while preserving external storage roots and named data volumes
  • Add explicit-confirmation restore and preservation of pre-restore deployment credentials
  • Add an external primary-cold-tier archive bind, an optional administrator-owned single-snapshot root exposed only to scoped one-shot helpers, and size-rotated Docker logs
  • Add archive status and full object-checksum verification commands that require exactly one active Parquet, manifest, and schema object for every query-visible generation, reject stale generation Parquet path/checksum or manifest-path pointers, and fail/retry if the canonical catalog snapshot changes during live verification
  • Qualify archive-generation status fields across the object join so an object's identically named lifecycle columns cannot make the operator status query ambiguous
  • Parse Docker image and volume inspection metadata as bounded, identity-validated JSON on Windows PowerShell 5.1, avoiding native-argument quote loss in setup, backup, and restore label checks
  • Bind Windows durability-barrier flags with explicit unsigned Win32 types so PowerShell 5.1 can initialize protected storage roots without narrowing the high-bit CreateFile option
  • Revalidate and extract the exact authenticated root-only OSCAR-state archive through a confined one-shot privilege bridge, then run daily-journal confirmation as the non-root OSCAR user before committing CURRENT
  • Give every capability-free root helper an explicit neutral working directory on Linux and Windows, preventing the image's owner-only OSCAR state directory from breaking validators, extractors, HMAC helpers, TLS setup, or restore tools after capabilities are dropped
  • Stream bounded UTF-8 bootstrap SQL over standard input on Windows PowerShell 5.1 instead of forwarding a quoted shell argument, preserving each statement exactly and retaining ON_ERROR_STOP failure behavior
  • Stream the multiline PostgreSQL checksum-marker and saved OSCAR configuration acceptance checks over standard input on Windows PowerShell 5.1, so native quote rewriting cannot truncate a fresh initialization after healthy containers start
  • Route every Windows setup/storage shell body and direct-psql statement through bounded UTF-8 standard input, require the direct docker run adapter to attach stdin itself, and relay setup-to-storage options through an allowlisted Base64/JSON environment envelope plus a fixed encoded child command so PowerShell 5.1 cannot rewrite operator values in native argv
  • Run pg_verifybackup in a networkless, read-only helper with every capability dropped except DAC_READ_SEARCH, so both administrator-normalized Linux backups and UID-999 Docker Desktop backups remain verifiable without granting write access or weakening mode-0600/0700 host permissions
  • Recheck the exact image inventory after connected OSCAR/PostGIS builds and pull the gateway only when it is still absent, avoiding an unnecessary registry/credential-helper dependency when the pinned gateway image is already cached
  • Build connected Windows OSCAR/PostGIS images from explicit validated release contexts, avoiding the Docker Compose 5.6/Buildx 0.37 in-tree Dockerfile entitlement regression without disabling filesystem checks or granting wildcard host access
  • Add supported generation/observation hold list/create/release commands with persistent actor, reason, and release audit fields
  • Resolve observation-hold membership from an exact PostgreSQL row before consulting coarse archive ID ranges, so sparse/interleaved retired generations cannot reject unrelated extant rows while tombstoned and cold-only ambiguous targets remain fail-closed
  • Persist each new hold's GENERATION_EXACT, HOT_EXACT, or COLD_RANGE_CONSERVATIVE resolution mode, expose it in both administration CLIs, preserve unresolvable pre-3.11.0 observation-hold provenance as protective LEGACY_UNKNOWN, and document that neither a cold range nor unknown legacy provenance proves a sparse observation ID exists
  • Enforce active observation and exact-generation holds transactionally across single and filtered hot/cold datastore removals and tombstones; single removal fails visibly, bulk removal skips protected rows, and set-based candidate lookup avoids per-row connection borrows or store-wide hold materialization
  • Fail closed for generic tiered datastore/datastream clear, drop, table-clone backup, and table-replacing restore APIs; prevent datastream removal while any held observation remains, and reject opening a catalog created by a newer storage-schema version
  • Add a two-step cold-deletion workflow that freezes eligible sealed generations in a 24-hour plan, emits a one-time token, rechecks holds and each object hash, validates relative paths, persists restartable DELETING intent, and retains generation tombstones plus catalog/deletion-ledger audit records
  • Add atomic portable archive generation packages with catalog-relative data/schema/manifest objects, catalog JSON, README, ready-to-run DuckDB query, and package checksums
  • Add complete architecture, lifecycle, archive-format, backup/restore, operations, monitoring, failure-recovery, privacy, and staged-rollout documentation
  • Fail closed on macOS for 3.11.0 deployment/storage operations because Docker Desktop for Mac cannot meet the protected bind-root ownership and recovery-media isolation contract; production hosts are Ubuntu 24.04 or Windows 11/WSL 2
  • Add (datastreamid, phenomenontime, id) composite indexes and window-pruned partition routing on attached PARQUET_PRIMARY stage partitions to eliminate multi-partition sequential scans during live and historical chart queries
  • Migrate legacy dailyFile datastream schemas (rapiscanMessage/aspectMessage to rpmMessage) in-place during store initialization, preserve backward-compatible JSON key deserialization, and harden PostGIS datastream/command-stream schema-evolution and member-system queries
  • Refresh Event Details adjudication history deterministically after submission and deduplicate historical vs. live MQTT adjudication entries by observation ID and timestamp
  • Add operational views based on configured keywords per lane
  • Modify daily file UI to show latest 15 messages on live view
  • Add individual detector charting to historical events
  • Fix Rapiscan neutron chart not showing first point in occupancy

Windows 11 unattended server readiness, ORNL upstream transition, and release CI

  • Transition official upstream repositories, submodule pins, osh-js dependency lock, localized manual asset links, and governance files (CODEOWNERS, CONTRIBUTING.md, SECURITY.md) to Oak Ridge National Laboratory (github.com/ORNL)
  • Add configure-windows-host.ps1 (.\oscar.bat prepare-host and .\oscar.bat host-check) with 20 automated checks covering AC power/hibernate/USB/NIC power-saving policy, S0 Low Power Idle (Modern Standby) detection (powrprof.dll!GetPwrCapabilities) with VIDEOIDLE = 0 (Never) enforcement, pre-login Docker Desktop startup (OSCAR-DockerDesktop-BootStart AtStartup +10s, LogonType=S4U with ACL-locked %ProgramData%\OSCAR\start-docker-prelogin.ps1), bounded .wslconfig memory/swap caps (wsl2-resource-cap), static vs. DHCP IPv4 stability (network-address-stability / -AllowDhcpAddress), bugcheck auto-reboot (AutoReboot=1, CrashDumpEnabled=7), Win32 long paths, W32Time sync, and optional LAN HTTPS 443 firewall rules
  • Stage Confirm-DailyJournalBackup inside an isolated native Linux ext4 Docker named volume instead of NTFS over WSL 2 9P, and use Win32 \\?\ long-path cleanup (Remove-DirectoryTreeLongPath) on Windows
  • Set absolute_redirect off; in dist/release/docker/gateway/default.conf so /view/<name> workstation redirects preserve the external HTTPS host and port instead of exposing internal container port :8443
  • Normalize /opt/oscar permissions in a dedicated Docker build stage to eliminate duplicate copy-up layers in oscar:3.11.0 and offline Windows image bundles
  • Add end-to-end live Docker Compose, ConSys API, headless Chromium Playwright UI/Admin, and single-snapshot backup regression verification (tools/test-live-deployment-regression.sh against ORNL/oscar-regression) to both continuous integration and tagged release builds, and publish SHA-256 manifests for all release ZIPs and localized PDF manuals
  • Push down exact PARQUET_PRIMARY stream/CQL counts without DataBlock decoding, prune cold archive generations by bounded resultTime, eliminate obs self-joins in scanHotMetadata, replace in-place mutable stream counter updates with lock-free append-only delta compaction (oscar_mutable_stream_counts_delta), add generic-plan-compatible covering partial event indexes (obs_non_alarm_ptime_desc_idx, obs_unadjudicated_alarm_idx), and coalesce WSL 2 9P Parquet SHA-256 verification with 16 KiB buffers
  • Deduplicate initial Viewer lane-discovery snapshots, cancel superseded EventTable count/page requests via AbortController, deprioritize background pagination count sweeps (priority: 'low'), resolve cold-tier gammaThreshold lookups, and eliminate BroadcastChannel wallboard listener leaks
  • Embed and automatically supervise a digest-pinned loopback mediamtx (1.12.2) RTSP/RTSPS proxy inside oscar:3.11.0 (MediaMtxSidecarManager, 127.0.0.1 only, keeping real camera URLs unchanged across config.json, Admin UI, and CSV imports/exports), deduplicate identical camera streams to one upstream TCP socket per node with immediate TEARDOWN on stop/edit, collapse accidental leading // RTSP paths, unthrottle >24 fps live network cameras, normalize packet DTS/PTS discontinuities, reconnect dropped streams in place without restarting the SensorHub module, and coordinate per-viewer live HLS leases so multiple operators can watch the same lane concurrently
  • Complete English (en), Spanish (es), French (fr), and Greek (el) localization and container-fit alignment across the React Viewer, Vaadin SensorHub Admin UI, /login session authentication page, and operator manuals: correct RPM domain terminology (including Greek Ανιχνευτής, διέλευση, and αξιολόγηση, French Adjugé, and Spanish fallos), localize live/historical occupancy alarm statuses and custom disposition codes, shorten verbose column/input/toolbar captions (such as compact Λωρίδα, Διέλευση, Απομακρυσμένος κόμβος, and Τερματισμός) and widen constrained DataGrid headers, open navigation drawer, and Admin split-pane widths so translated labels fit without horizontal clipping, localize /login via osh-language cookie, ?lang=, and Accept-Language without inline scripts, and add 4-locale ConfigMessages bundles for core, PostGIS/Parquet, MQTT, FileServer, Bucket, DailyFileJournal, occupancy-process, and session-security modules
  • Remove all non-redistributable EML JARs, models, and XML files from OSCAR repositories and release bundles; add hot-loadable admin EML plugin upload (emlPluginPath stored privately in /var/lib/oscar/eml-plugin/, backed up in oscar_state, and extracted to /tmp/oscar-eml-runtime), synchronous pre-publication GX evaluation (EmlRuntimeRegistry), zero-flash t = 0 Dashboard/notification suppression for verified EML RELEASE non-threat alarms, durable automated Code 9999: EML Non-Threat adjudication records with EML_METADATA JSON, localized EmlAnalysisCard + AdjudicationLog views, and pinned TeX Live cache hardening in release PDF CI
  • Add indexed full-database Adjudication Code filtering (equals, does not equal, is any of, is empty, is not empty) and an optional Adjudication Code column on the Events page across built-in Codes 1–11, automated Code 9999: EML Non-Threat, and site-configured custom disposition codes ("code:<n>" synthetic markers on occupancy.adjudicatedIds), backed by an asynchronous, idempotent startup backfill (AdjudicationCodeMarkerBackfill) for historical adjudications on upgraded deployments

Contributors

Thanks to everyone who contributed across ORNL/oscar and its submodules since public release v3.8.0:

  • Alex Almanza (@earocorn) — ORNL/oscar, ORNL/oscar-viewer, ORNL/osh-oakridge-modules
  • Drew Botts (@drewbotts) — ORNL/osh-oakridge-modules
  • Kalyn Stallings (@kalynstricklin) — ORNL/oscar, ORNL/oscar-viewer, ORNL/osh-oakridge-modules
  • Kyle Fitzpatrick (@kyle-fitzp) — ORNL/osh-oakridge-modules
  • Tyrone Harris (@tyronechrisharris) — ORNL/oscar, ORNL/oscar-viewer, ORNL/osh-addons, ORNL/osh-core, ORNL/osh-js, ORNL/osh-oakridge-modules

Full changelog across repository and submodules

Contributors

drewbotts, earocorn, and 3 other contributors
Assets 15
Loading