Skip to content

v1.37.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 21 Oct 15:39
v1.37.0
8459646
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

To learn how to install and configure OpenCue, see our Getting Started guide.

This release focuses on Maestro performance and correctness at production scale: shorter scheduling ticks, accurate farm and show gauges, and fixes for stalls and lost completions. It also makes CueGUI much more responsive, moves frame_history writes off the frame update path, makes frame retry limits configurable, and publishes the OpenCueWeb and REST Gateway Docker images for the first time.

Upgrade Notes

  • Database migrations V50 and V51.
    Cuebot applies them automatically at startup through Flyway. Back up your database before upgrading. V50 adds a partial index on frame. On a large frame table, build it out-of-band first with CREATE INDEX CONCURRENTLY i_frame_layer_dispatch_waiting ON frame (pk_layer, int_dispatch_order, int_layer_order) WHERE str_state = 'WAITING' so writes to frame aren't blocked. The migration then has nothing left to do.
    (#2603)

  • frame_history is now written in batches and can lag by up to 5 minutes.
    Frame state changes are queued in a new frame_history_queue table and drained by a background task. Tune this with history.frame_drain_interval_ms (default 300000) and history.frame_drain_batch_size (default 5000). Reports that read frame_history directly will see recent frames late.
    (#2579)

  • Frame retry limits are configurable, and a job's max_retries is respected.
    Previously, any job max_retries above 1 was silently forced to 1. Jobs now get the requested value, clamped to job.frame_retries_min and job.frame_retries_max (defaults 0 and 16, with job.frame_retries_default=1). Jobs that asked for more than one retry will now actually retry more.
    (#2613)

  • The orphaned-proc age is now a property, with a longer default.
    maintenance.orphaned_proc_age_seconds defaults to 600 (it was a hard-coded 300 s), so finished frames whose completion is still queued are no longer reset to WAITING and re-rendered.
    (#2608)

  • OpenCueWeb and REST Gateway Docker images are now published.
    opencue/cueweb and opencue/rest-gateway are available on Docker Hub starting with this release.

Major Features

  • Faster Maestro scheduling ticks
    Each tick now reads its candidate layers once and splits them per host group in memory, reads each planned layer's next waiting frames once through a new ordered index, and resolves host pins in memory instead of with a layers × hosts query. Ticks are also timed by phase so slow phases show up in metrics.
    (#2603), (#2606), (#2610), (#2620)

  • CueGUI responsiveness
    Job actions run on a background worker instead of the GUI thread, post-action refreshes show the new state right away, and the job tree no longer re-expands, recomputes cell data, or re-sorts on every paint. This also fixes a deadlock in setShow.
    (#2619)

  • Batched frame_history writes
    Frame starts and completions no longer pay for an inline insert or update on the heavily indexed frame_history table. Short frames whose start and end land in the same batch are written as a single row.
    (#2579)

Cuebot Improvements

  • Maestro layer sizing: layers are sized from the larger of their memory request and their largest reported RSS, and cores follow memory. The size persists and is visible in CueGUI. Layers tagged manual keep their request (#2604)
  • Maestro folder limits: folder core usage is counted from folder_resource, matching what the database enforces, and a commit chunk that fails is split and retried so one folder over its limit no longer blocks every booking in the tick (#2611)
  • Maestro metrics: cue_maestro_show_cores and cue_maestro_running_frames are read from the proc table each tick, so they no longer reset on restart or drift upward. A new cue_maestro_farm_cores_busy gauge reports the CPU that running frames actually use, next to core utilisation (#2600), (#2584)
  • Lost completions fixed: the orphaned-proc sweep no longer resets frames whose completion is queued but not yet applied. Frame-before-stats lock ordering removes a deadlock with the completion drain, and new gauges expose the completion queue (#2608)
  • Host lock waits: locking or unlocking a host no longer holds the host row lock while waiting on the RQD RPC (#2612)
  • Maestro configuration: new maestro.* properties for placement priority, commit and drain chunk sizes, leader checks, slow-tick warnings, and shutdown waits. The per-tick orphan sweep was removed, and its setting with it. Cleanup is left to the maintenance task (#2615), (#2620)
  • Seed data: pre- and post-process layers now request 1 core instead of 0.1, so they book on the default test shows (#2613)

Documentation and Ecosystem

  • CI: GitHub workflow actions updated to their latest versions, runners moved to Ubuntu 24.04, and the simulation workflow moved to PostgreSQL 16 (#2575), (#2609)
  • Dependencies: OpenCueWeb (next, sharp, @sentry/nextjs, OpenTelemetry instrumentations, source-map-js, brace-expansion) and Rust (rustls) dependency bumps (#2601), (#2592), (#2595), (#2596), (#2588), (#2602), (#2587)
  • Docs: release notes and docs version moved to 1.34.22 (#2599)

Changes

  • 5b7c6cb [docs] Move release notes and docs version to 1.34.22 (#2599)
  • 20f439b [cuebot] Maestro: read show_cores and running_frames from the proc table (#2600)
  • 27d21a2 Bump @opentelemetry/instrumentation-knex and @sentry/nextjs in /cueweb (#2596)
  • a27f22b Bump @opentelemetry/instrumentation-tedious and @sentry/nextjs in /cueweb (#2595)
  • d12eea3 Bump rustls from 0.23.42 to 0.23.45 in /rust (#2587)
  • aa060d8 Bump source-map-js from 1.2.1 to 1.2.2 in /cueweb (#2588)
  • a0b5717 Bump next from 16.3.4 to 16.4.0 in /cueweb (#2601)
  • 404746c Bump brace-expansion from 1.1.18 to 1.1.21 in /cueweb (#2602)
  • bca6e0e Bump sharp from 0.35.4 to 0.35.5 in /cueweb (#2592)
  • 93cc8c9 Maestro: export CPU utilisation beside core utilisation (#2584)
  • 6d15380 [cuebot] Maestro: read each planned layer once per tick and time the tick by phase (#2603)
  • 16bd204 Maestro: size layers from their largest rss, drop the live ledger (#2604)
  • 81bcd93 [cuebot] Maestro: read the tick's candidates once and cut them per group in memory (#2606)
  • 166618a [cuebot] Batch frame_history writes through a queue (#2579)
  • 1e72818 [cicd] Fix migration numbers and version up (#2609)
  • 3dbb11f [cuebot] Orphaned-proc age as a property, frame-before-stats lock order, completion-queue gauges (#2608)
  • 1c2b927 [cuebot] Maestro: resolve pins in memory from the tick's candidate read (#2610)
  • d6726f2 [cuebot] Don't hold the host row lock across the RQD lock/unlock RPC (#2612)
  • 7c6628c [cuebot] Maestro: count folders from folder_resource and split a failed commit chunk (#2611)
  • 5a9eb73 Job retry limit fix (#2613)
  • 0150b86 [cicd] Update GitHub workflow actions to latest versions (#2575)
  • 5f8d106 [Cuebot] Refactor Maestro.java (#2615)
  • 6027b75 Maestro: drop the per-tick orphan sweep (#2620)
  • 8459646 [cuegui] Fix performance issues reported on #2617 (#2619)
Assets 12
Loading