Skip to content

feat: five of the seven Neural Filters run a network - #65

Merged
IAmJSD merged 1 commit into
mainfrom
t3code/implement-neural-filters
Aug 29, 2026
Merged

feat: five of the seven Neural Filters run a network#65
IAmJSD merged 1 commit into
mainfrom
t3code/implement-neural-filters

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Aug 29, 2026

Copy link
Copy Markdown
Member

The README said two of them did. Now five do, and the two that do not say why.

What is new

Trained here, shipped inside the binary (tools/train/, and each script's defaults reproduce the weights that ship):

params what it does
dejpeg.onnx 66k / 256 KB +1.0 to +1.2 dB over the JPEG at every quality from 10 to 60, on held-out images. Trained on Kodak compressed across that whole range, with the patches cut at unaligned offsets so it has to find the 8-pixel grid rather than assume where it is — a filter runs on a selection that starts anywhere.
colorize.onnx 466k / 1.8 MB Predicts a distribution over 249 binned chromas rather than a chroma. Asked for one number, a network that cannot tell whether the car is red or blue answers brown, which is how colourisers end up sepia; class rebalancing and an annealed mean make it pick a side. Trained on 20,000 CC BY 2.0 photographs from Open Images. 1.1× the colourfulness of the real thing on held-out photographs, at 0.43 chroma correlation.

Downloaded on demand, hash checked, both running in tract unmodified: MiDaS v2.1 small (66 MB, MIT) for Depth Blur, UltraFace RFB-320 (1.2 MB, MIT) for Skin Smoothing. Depth Blur is now a real defocus — subject sharp and street gone, or the reverse — instead of a guess from local sharpness. Skin Smoothing smooths skin that is on a face, and scales its radius to the face, instead of anything skin-coloured anywhere.

Super Zoom is retrained, because finding out why the deblocker would not train explained why Super Zoom underperformed as well. PyTorch's default convolution initialisation shrinks activations by about a third a layer, so a residual network with a zero-initialised tail can train for thousands of steps and sit at exactly the identity — scoring precisely as well as doing nothing, which is a very quiet way to fail. Kaiming initialisation and normalisation fix it: +0.40 → +0.58 dB at the same 39k parameters.

Colour Transfer keeps no model and is not missing one. Moving one image's colour distribution onto another is arithmetic; Photoshop's network there is doing the segmentation around it, which is Object Selection's problem.

Supporting changes

  • schist-neural gains an Input enum separating models that change an image (tiles at full resolution) from models that answer a question about one (one resampled frame, letterboxed for the detector), per-channel input normalisation, and depth/faces/colour modules that decode each network's answer.
  • Chroma is resampled up guided by the luminance the model saw, so colour stops at the edges of the subject instead of bleeding over them.
  • Each filter memoises its network's answer against a fingerprint of the pixels — where the faces are does not change while somebody drags Strength.
  • Manage Models lists the new entries with sizes and licences, and a download now reads Downloading… 12.4 of 63.7 MB rather than sitting on an ellipsis for a minute.

Checked

Full workspace suite green, clippy and rustfmt clean. New tests cover the deblocker against a real JPEG fixture, the colouriser by correlation rather than error (a timid colouriser scores better on error and looks broken), and depth and faces where those models are installed — skipped, not failed, where they are not. The app was also driven headlessly: the 66 MB depth model removed and re-fetched through the dialog, and a Depth Blur preview reporting Using Depth (Depth Blur) · MiDaS v2.1 small, Intel ISL, MIT.

Known limits, stated in the README

The colouriser is small enough to be confidently wrong: it painted a red patch into a river in one test image. It knows sky, foliage, wood and skin, and hedges on anything whose colour is a choice. UltraFace has opinions about dogs.

🤖 Generated with Claude Code

Two did: Super Zoom and Style Transfer. JPEG Artifact Removal, Colorize
and Depth Blur were signal processing, and Skin Smoothing did frequency
separation on anything skin-coloured whether or not it was on a face.

Trained here and shipped in the binary:

* dejpeg.onnx, 66k parameters. Kodak compressed at every quality from 10
  to 60, cropped at unaligned offsets so it has to find the block grid
  rather than assume it is at a multiple of eight. +1.0 to +1.2 dB on
  held-out images across the whole quality range.
* colorize.onnx, 466k. Predicts a distribution over 249 binned chromas
  rather than a chroma: asked for one number a network that cannot tell
  red from blue answers brown, which is how colourisers end up sepia.
  Class rebalancing and an annealed mean pick a side. Trained on 20,000
  CC BY photographs from Open Images; 1.1x the colourfulness of the real
  thing on held-out photographs, at 0.43 chroma correlation.

Downloaded on demand, hash checked, both running in tract unmodified:
MiDaS v2.1 small for depth and UltraFace RFB-320 for faces.

detail.onnx is retrained, because finding out why dejpeg would not train
explained why Super Zoom underperformed too: PyTorch's default
convolution initialisation shrinks activations by about a third a layer,
so a residual network with a zero-initialised tail can train for
thousands of steps and sit at exactly the identity -- scoring precisely
as well as doing nothing. Kaiming initialisation and normalisation fix
it, and Super Zoom goes from +0.40 to +0.58 dB at the same 39k
parameters.

schist-neural grows the shapes this needs: an Input enum separating
models that change an image (tiles at full resolution) from models that
answer a question about one (a single resampled frame, letterboxed for
the detector), per-channel input normalisation, and depth/faces/chroma
modules that decode each network's answer. Chroma is resampled up guided
by luminance, so colour stops at the edges of the subject rather than
bleeding across them. Each filter memoises its network's answer against a
fingerprint of the pixels: where the faces are does not change while
somebody drags Strength.

Colour Transfer keeps no model and is not missing one -- moving one
colour distribution onto another is arithmetic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@IAmJSD IAmJSD changed the title Five of the seven Neural Filters run a network feat: five of the seven Neural Filters run a network Aug 29, 2026
@IAmJSD
IAmJSD merged commit b340b0e into main Aug 29, 2026
3 checks passed
@IAmJSD
IAmJSD deleted the t3code/implement-neural-filters branch August 29, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant