Skip to content

Add a durable named image proxy - #1730

Open
Aaronontheweb wants to merge 8 commits into
devfrom
feat/1728-named-image-proxy
Open

Add a durable named image proxy#1730
Aaronontheweb wants to merge 8 commits into
devfrom
feat/1728-named-image-proxy

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

  • Add a reusable named-model runtime registry for role and proxy model resolution.
  • Add Models.Proxies.Image with schema, CLI, and TUI support.
  • Persist image descriptions before a text-only main model call.
  • Preserve original images for image-capable main models.
  • Process old, unprocessed session images on the next turn.
  • Route approved Slack, Discord, and Mattermost images through the shared proxy path.

Safety

  • Fail startup for unknown or incompatible proxy definitions.
  • Stop the main and fallback calls when the proxy fails.
  • Send no session history or tools to the proxy.
  • Mark proxy output as untrusted text and neutralize wrapper delimiters.

Validation

  • dotnet test Netclaw.slnx --no-restore
  • dotnet slopwatch analyze
  • pwsh ./scripts/Add-FileHeaders.ps1 -Verify
  • openspec validate named-image-modality-proxy --strict
  • ./scripts/smoke/run-smoke.sh model-manager

Eval note

  • Add skill_image_proxy_configuration to the behavioral eval suite.
  • The available qwen2 model cannot call tools.
  • The available qwen2.5 model did not load the required operations skill.
  • Run the case with the normal tool-capable eval target before merge.

Depends on #1729.

Closes #1728.

: ValueOverride<ModelModality>.Set(ModelModality.Text);

if (!inputOverride.Value!.Value.HasFlag(ModelModality.Image)
|| !inputOverride.Value.Value.HasFlag(ModelModality.Text)
.WithChild(_roleList)
.WithChild(new TextNode("").Height(1))
.WithChild(new TextNode(" [Enter] Assign model [D] Discover models [C] Clear optional role")
.WithChild(new TextNode(" [Enter] Assign model [D] Discover models [C] Clear optional role or proxy")
Comment on lines +2831 to +2834
catch (Exception ex)
{
return new ImageProxyPreparationFailed(ex);
}
Comment on lines +48 to +50
var mediaDirectory = Path.GetFullPath(Path.Combine(
GetSessionDirectory(sessionId, basePath),
MediaSubdirectory));
var mediaDirectory = Path.GetFullPath(Path.Combine(
GetSessionDirectory(sessionId, basePath),
MediaSubdirectory));
var fullPath = Path.GetFullPath(Path.Combine(mediaDirectory, relativePath));
configuration,
new StubRegistry(runtime),
new FakeTimeProvider(DateTimeOffset.FromUnixTimeMilliseconds(1234)));
var basePath = Path.Combine(Path.GetTempPath(), $"netclaw-image-proxy-{Guid.NewGuid():N}");
client,
ModelCapabilityResolution.ResolveModelCapabilities(model, detected: null));
var analyzer = CreateAnalyzer(model, runtime);
var basePath = Path.Combine(Path.GetTempPath(), $"netclaw-image-proxy-empty-{Guid.NewGuid():N}");
Copilot AI lite review requested due to automatic review settings August 4, 2026 03:50
@Aaronontheweb
Aaronontheweb force-pushed the feat/1728-named-image-proxy branch from 162a101 to 2add740 Compare August 4, 2026 03:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

When a session contains historical media the current model cannot
accept (e.g. images from a multimodal model in a text-only session),
degrade gracefully:

- ChatMessageConverter strips incompatible DataContent at assembly
  time without touching persisted MediaReferences.
- SessionMessageAssembler injects a volatile [system: media-filtered]
  notice so the user knows content was omitted.
- FireLlmCall logs a warning instead of failing the turn.
- TryRejectIncompatibleInput only hard-rejects new user-supplied
  media (defense-in-depth; the adapter already handles this).

The session stays usable. The assembler boundary is the single correct
place to filter — after history reconstruction, before the provider.

Tests: 11/11 pass (compatibility unit tests + modality gate tests +
integration tests for recovered history and buffered drain paths).
Without the gate, the first FakeChatClient response completes instantly
and the second turn's drain + model call races against the
FishForMessageAsync timeout. Gating the first response ensures both
TurnCompleted events arrive deterministically.
This guard was removed in the original PR but should have been kept.
It catches media that slipped past the adapter's capability gate (a
contract violation in netclaw-input-adapters), strips the offending
references, logs an operator-visible error, and appends a system
notice so the user knows something went wrong.

This is the new-message counterpart to the assembly-time history filter
added in the previous commit — both degrade instead of rejecting.
Copilot AI review requested due to automatic review settings August 4, 2026 04:49
@Aaronontheweb
Aaronontheweb force-pushed the feat/1728-named-image-proxy branch from 2add740 to 17f86c1 Compare August 4, 2026 04:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Base automatically changed from fix/1727-session-modality-compatibility to dev August 4, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants