Conversation
Port the Slint renderer from PR #237 into a standalone native plugin using the NativeSourceNode trait from PR #238. This keeps slint and slint-interpreter as plugin-only dependencies, avoiding committing them to the core workspace. The plugin renders .slint files to RGBA8 video frames at configurable resolution and frame rate. All Slint operations run on a shared dedicated thread with channel-based message passing to handle the non-Send constraint of Slint types. Includes: - Plugin crate with config, shared thread, and NativeSourceNode impl - justfile build/lint/fix/copy targets - CI format check and clippy in lint-simple job - Marketplace metadata (plugin.yml) and regenerated official-plugins.json - Docs reference page and updated plugin index (9 -> 10) - Sample .slint files (watermark, scoreboard, lower_third) - Sample oneshot pipeline using plugin::native::slint Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Replace simplified steps-based oneshot pipeline with the full compositor pipeline from PR #237 (colorbars + watermark overlay → compositor → VP9 → WebM → http_output). Add the missing dynamic pipeline (scoreboard + lower-third overlays composited onto colorbars, streamed via MoQ). Update docs example to match the correct node-based format. All pipelines use plugin::native::slint instead of video::slint. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
![staging-devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/870844?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
🐛 1 issue in files not directly in the diff
🐛 Missing libfontconfig1-dev in marketplace-build.yml breaks plugin CI build (.github/workflows/marketplace-build.yml:45)
The Slint plugin depends on yeslogic-fontconfig-sys (confirmed in plugins/native/slint/Cargo.lock:1280), which requires the libfontconfig1-dev system package to compile. The PR correctly adds this dependency to .github/workflows/plugins.yml:72 for lint CI, but does not add it to .github/workflows/marketplace-build.yml:45 which installs system dependencies for building all official plugins. The build_official_plugins.sh script auto-discovers plugins from marketplace/official-plugins.json and builds them all, so it will attempt to build the Slint plugin and fail with a missing fontconfig header error. This violates the AGENTS.md checklist item: "Build prerequisites in .github/workflows/release.yml if new system deps are required to compile or package the plugin."
View 6 additional findings in Devin Review.
Debug
- Rename plugin output pin from 'video' to 'out' to match the convention used by all built-in nodes. The pipeline compiler's 'needs' syntax defaults from_pin to 'out', so using a different name broke graph wiring. - Add libfontconfig1-dev to marketplace-build.yml system deps so the Slint plugin can be compiled by build_official_plugins.sh. - Update docs to reflect the pin name change. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
The host probes source plugins by calling create_instance with no params to read source_config(). The Slint plugin requires slint_file which fails validation with default config. Return a lightweight probe-only instance when params is None so the host correctly detects is_source=true and uses run_source() instead of run_processor(). Without this fix the watermark node was treated as a processor, crashed immediately (no input pins), and the compositor produced output with only colorbars (no overlay). Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
There was a problem hiding this comment.
🚩 PORTABILITY_REVIEW.md referenced in AGENTS.md but deleted from repo
AGENTS.md requires updating marketplace/PORTABILITY_REVIEW.md when adding an official plugin, but the file was deleted in commit 9548c98 and no longer exists in the repository. This is a pre-existing gap — this PR is consistent with other recent plugin additions (supertonic, helsinki) that also didn't update it. The AGENTS.md checklist should either be updated to remove this requirement or the file should be restored.
Was this helpful? React with 👍 or 👎 to provide feedback.
Debug
Slint Native Plugin — E2E Test ReportBuilt the plugin locally, loaded it into a running StreamKit server, and tested the oneshot watermark pipeline end-to-end via curl + ffprobe + pixel comparison. All 4 tests passed. Test Results
Bugs Found & Fixed (prior to this run)
Watermark Overlay Evidence (Test 3)Extracted frame 5 from both colorbars-only and watermark pipelines, cropped the overlay region (x:1080, y:20, w:180, h:44), and compared pixel data.
Notes
|
…ng, validation - Add ClearWindow scope guard so CURRENT_WINDOW thread-local is cleared even if definition.create() or component.show() fails via early return - Remove unnecessary data.clone() in static-UI render path — store first, then clone from cache (saves ~3.7 MB allocation per cache-miss render) - Rename lib from slint_plugin to slint, matching the naming convention of all other native plugins (whisper, kokoro, vad, etc.) - Add MAX_DIMENSION (7680 / 8K) upper bound on width/height to guard against config typos that would attempt multi-GB buffer allocations - Reject absolute paths in validate_slint_asset_path() as defense-in-depth - Document update_timers_and_animations() idempotency for multi-instance usage on the shared Slint thread Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…-instance max_ticks The load-time probe creates a default instance (null params) to detect source plugins, so max_ticks is always 0 (infinite). Per-instance params like frame_count: 300 were never applied to the tick loop limit. Now run_source() re-queries get_source_config() on the live instance after it's created with actual params, so the tick loop respects the configured frame_count. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…ch_else Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… None Split the tuple pattern into chained and_then/map so begin_call() is only invoked when get_source_config is Some, preventing an in_flight_calls counter leak in the else branch. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(client): add declarative overlay controls in client section
Add a `controls` array to `ClientSection` so pipeline authors can
declare interactive widgets (toggle, text, number, button) targeting
specific node properties. The StreamView renders these controls
automatically when a session is active, sending `TuneNodeAsync` /
`UpdateParams` messages on interaction.
Rust changes:
- New `ControlType` enum and `ControlConfig` struct in `yaml.rs`
- Extend `ClientSection` with `controls: Option<Vec<ControlConfig>>`
- Add `name` field to `NodeInfo` for node-name validation
- Lint rules: `control-unknown-node`, `control-number-no-bounds`
- Register new types for TypeScript generation
UI changes:
- New `OverlayControls` component with toggle/text/number/button widgets
- New `buildParamUpdate()` utility for dot-notation → nested JSON
- Integrate `OverlayControls` into `StreamView`
Sample:
- Add controls to `video_moq_slint_scoreboard.yml` for scoreboard
and lower-third properties
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(controls): stabilize debounce/throttle with ref pattern, add useTuneNode hook
Address Devin Review feedback:
1. TextControl: store onSend in a ref so the debounce closure is stable
across re-renders. Pending timers now always call the latest callback
instead of leaking stale intermediate values.
2. NumberControl: apply the same ref pattern for onSend in the throttle
closure for consistency, preventing similar issues.
3. Extract useTuneNode hook from useSession — a lightweight hook that
only provides tuneNodeConfig without subscribing to pipeline or
connection state, avoiding unnecessary re-renders in OverlayControls.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(controls): address review feedback — dead code, guards, stale state, docs
1. Remove dead isDraggingRef in NumberControl (written but never read).
2. Guard buildParamUpdate against empty/malformed paths — filter empty
segments, throw on zero valid segments. Add unit tests covering
single/multi-segment, empty, dot-only, double-dot, and leading/
trailing dot paths.
3. Fix ToggleControl stale checked on rapid double-click — use
functional updater (setChecked(prev => ...)) with onSendRef
pattern matching TextControl/NumberControl.
4. Update misleading 'stable' comment on makeSend to accurately
describe that a new closure is created per render but child
controls absorb this via onSendRef.
5. Move getWebSocketService() to module scope in useTuneNode since
it returns a singleton — avoids a new reference on every render
and keeps tuneNodeConfig deps minimal (only sessionId).
6. Document that ControlConfig.default is a UI-only hint — it seeds
local widget state but is not sent to the server on mount.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(controls): deep-merge partial nested updates in useTuneNode
useTuneNode now reads the current nodeParamsAtom state and deep-merges
the partial config before writing, so sibling nested properties are
preserved. Previously, two controls targeting the same node but
different nested paths (e.g. properties.home_score and
properties.away_score) would clobber each other because writeNodeParams
does a shallow top-level merge.
Add deepMerge utility to controlProps.ts — recursively merges plain
objects, replaces arrays and primitives wholesale. Includes 8 unit
tests covering nested merge, array replacement, type transitions,
successive merges, and immutability.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* test(e2e): add Playwright test for overlay controls
Add overlay-controls.spec.ts exercising all four control types (toggle,
text, number, button). The test:
1. Selects the new 'Test: Overlay Controls' sample pipeline (colorbars →
sink, no plugins/MoQ required).
2. Creates a session and verifies the Pipeline Controls section renders
with correct labels and group headings.
3. Exercises each control type and asserts the correct TuneNodeAsync /
UpdateParams WebSocket payload is sent:
- Toggle: sends { draw_time: false }
- Text: sends { label: "World" } after 300ms debounce
- Slider: sends { properties: { width: 800 } } (dot-notation path)
- Button: sends { reset: true }
4. Asserts no unexpected console errors.
5. Destroys the session and cleans up.
Also adds data-testid='overlay-controls' to the OverlayControls
component for scoped locators, and fixes a TS6 useRef() arity error.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
| pub fn lint_client_against_nodes( | ||
| client: &ClientSection, | ||
| _mode: EngineMode, |
There was a problem hiding this comment.
🚩 lint_client_against_nodes has no production callers yet
The lint_client_against_nodes function (including the new Rules 21-22 for controls) is only called from unit tests — no production code in the server crate invokes it. The function and NodeInfo.name field are pub, suggesting integration is planned. Until then, the control-unknown-node and control-number-no-bounds lint rules provide no runtime protection. This isn't a bug — just worth noting that the lint coverage exists only at the unit-test level.
(Refers to lines 1157-1161)
Was this helpful? React with 👍 or 👎 to provide feedback.
Debug
The `controls` field is dynamic-only but was not included in the `has_dynamic_fields` check in `lint_client_section`, so a oneshot pipeline with a `controls` section would not trigger the `mode-mismatch-oneshot` warning. Add it to the check and update the warning message and doc comment accordingly. Adds a unit test verifying the warning fires for controls-only oneshot pipelines. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
| // Resolved once at module level — getWebSocketService returns a singleton, | ||
| // so hoisting it avoids a new reference on every render and keeps | ||
| // tuneNodeConfig's useCallback deps minimal. | ||
| const wsService = getWebSocketService(); |
There was a problem hiding this comment.
🚩 Module-level getWebSocketService() call executes at import time
const wsService = getWebSocketService() at line 16 runs when the module is first imported, which could be during app bootstrap before the WebSocket singleton is fully configured. The comment says it returns a singleton, and this pattern is used to keep useCallback deps minimal. If getWebSocketService uses lazy initialization internally (create on first send, not on call), this is safe. If it eagerly connects, it could fail or connect prematurely. Worth verifying the singleton's initialization strategy if this module is imported in test environments or SSR contexts.
Was this helpful? React with 👍 or 👎 to provide feedback.
Debug
There was a problem hiding this comment.
Safe in this case — getWebSocketService() is a lazy singleton factory that only constructs a WebSocketService(url), which stores the URL string without connecting. The actual WebSocket connection is established later via an explicit connect() call (triggered by the app's connection lifecycle, not at import time). So module-level instantiation just creates a lightweight object with no side effects.
This matches the existing useSession pattern where getWebSocketService() is also called inline (though per-render there). The module-level hoist was intentional to keep useCallback deps stable.
2 participants
Summary
Native Slint UI plugin for StreamKit — renders
.slintUI definitions as RGBA video frames for overlay compositing. Built on the native plugin SDK (PR #238) with source-node tick-driven rendering.Core plugin (
plugins/native/slint/)!Sendthread managing Slint instances via channels; supports multi-instance rendering with shared event loop..traversal and absolute paths), dimensions (1–7680 MAX_DIMENSION cap), FPS, keyframe intervals, andframe_countfor finite pipelinesstatic_ui: true, caches rendered frames and only re-renders on keyframe property changesHost-side fixes
compositor/mod.rs): Two-pass dequeue in oneshot mode ensures all active slots have pending frames before dequeueing any — prevents fast sources (colorbars) draining ahead of slower ones (slint)wrapper.rs): Re-queriesget_source_config()on the live instance after creation with actual params, soframe_count: 300correctly limits the tick loop instead of always using the load-time probe default (0 = infinite)wrapper.rs): Refactored from tuple pattern to chainedand_then/mapsobegin_call()is only invoked whenget_source_configisSome, preventing anin_flight_callscounter leakLint fix
mode-mismatch-oneshotnow includescontrols: Thehas_dynamic_fieldscheck was missingcontrols, so oneshot pipelines with acontrolssection would silently pass. Fixed with test coverage.Review findings addressed
data.clone()in static-UI path (saves ~3.7 MB per cache-miss)slint_plugin→slint(matches convention)is_absolute()check on path validation (defense-in-depth)update_timers_and_animations()idempotencybegin_call()leak prevention viaand_then/mapchainingcontrolsadded tohas_dynamic_fieldsformode-mismatch-oneshotlintReview & Testing Checklist for Human
curl -X POST http://localhost:4545/api/v1/process -F "config=@samples/pipelines/oneshot/video_slint_watermark.yml" -o /tmp/output.webm— should terminate in ~10s with 300 frames/tmp/output.webm— should show colorbars background with Slint watermark overlay, no trailing black+overlay framescompositor/mod.rsdoesn't regress existing oneshot tests (just testpasses all compositor tests)libslint.sois produced (notlibslint_plugin.so) and all references are consistent acrossplugin.yml,official-plugins.json,justfile, docscontrolssection and verify themode-mismatch-oneshotwarning firesNotes
84364c6) since it wasn't included in the merge to mainmax_ticksfix is in the host-side wrapper (crates/plugin-native/), not the plugin itself — it affects all source plugins that setmax_ticksbased on runtime paramsPORTABILITY_REVIEW.mdis referenced in AGENTS.md but was deleted in commit9548c98— this is a pre-existing gap, consistent with other recent plugin additionslint_client_against_nodes(Rules 21-22 for controls) is only called from unit tests currently — no production caller yet; integration is plannedgetWebSocketService()inuseTuneNode.tsis safe: the constructor only stores the URL, no connection is established until explicitconnect()callLink to Devin session: https://staging.itsdev.in/sessions/87676781c3744d4e8dbd36f39422b350
Requested by: @streamer45