feat: five of the seven Neural Filters run a network - #65
Merged
Conversation
Two did: Super Zoom and Style Transfer. JPEG Artifact Removal, Colorize and Depth Blur were signal processing, and Skin Smoothing did frequency separation on anything skin-coloured whether or not it was on a face. Trained here and shipped in the binary: * dejpeg.onnx, 66k parameters. Kodak compressed at every quality from 10 to 60, cropped at unaligned offsets so it has to find the block grid rather than assume it is at a multiple of eight. +1.0 to +1.2 dB on held-out images across the whole quality range. * colorize.onnx, 466k. Predicts a distribution over 249 binned chromas rather than a chroma: asked for one number a network that cannot tell red from blue answers brown, which is how colourisers end up sepia. Class rebalancing and an annealed mean pick a side. Trained on 20,000 CC BY photographs from Open Images; 1.1x the colourfulness of the real thing on held-out photographs, at 0.43 chroma correlation. Downloaded on demand, hash checked, both running in tract unmodified: MiDaS v2.1 small for depth and UltraFace RFB-320 for faces. detail.onnx is retrained, because finding out why dejpeg would not train explained why Super Zoom underperformed too: PyTorch's default convolution initialisation shrinks activations by about a third a layer, so a residual network with a zero-initialised tail can train for thousands of steps and sit at exactly the identity -- scoring precisely as well as doing nothing. Kaiming initialisation and normalisation fix it, and Super Zoom goes from +0.40 to +0.58 dB at the same 39k parameters. schist-neural grows the shapes this needs: an Input enum separating models that change an image (tiles at full resolution) from models that answer a question about one (a single resampled frame, letterboxed for the detector), per-channel input normalisation, and depth/faces/chroma modules that decode each network's answer. Chroma is resampled up guided by luminance, so colour stops at the edges of the subject rather than bleeding across them. Each filter memoises its network's answer against a fingerprint of the pixels: where the faces are does not change while somebody drags Strength. Colour Transfer keeps no model and is not missing one -- moving one colour distribution onto another is arithmetic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README said two of them did. Now five do, and the two that do not say why.
What is new
Trained here, shipped inside the binary (
tools/train/, and each script's defaults reproduce the weights that ship):dejpeg.onnxcolorize.onnxDownloaded on demand, hash checked, both running in
tractunmodified: MiDaS v2.1 small (66 MB, MIT) for Depth Blur, UltraFace RFB-320 (1.2 MB, MIT) for Skin Smoothing. Depth Blur is now a real defocus — subject sharp and street gone, or the reverse — instead of a guess from local sharpness. Skin Smoothing smooths skin that is on a face, and scales its radius to the face, instead of anything skin-coloured anywhere.Super Zoom is retrained, because finding out why the deblocker would not train explained why Super Zoom underperformed as well. PyTorch's default convolution initialisation shrinks activations by about a third a layer, so a residual network with a zero-initialised tail can train for thousands of steps and sit at exactly the identity — scoring precisely as well as doing nothing, which is a very quiet way to fail. Kaiming initialisation and normalisation fix it: +0.40 → +0.58 dB at the same 39k parameters.
Colour Transfer keeps no model and is not missing one. Moving one image's colour distribution onto another is arithmetic; Photoshop's network there is doing the segmentation around it, which is Object Selection's problem.
Supporting changes
schist-neuralgains anInputenum separating models that change an image (tiles at full resolution) from models that answer a question about one (one resampled frame, letterboxed for the detector), per-channel input normalisation, anddepth/faces/colourmodules that decode each network's answer.Downloading… 12.4 of 63.7 MBrather than sitting on an ellipsis for a minute.Checked
Full workspace suite green, clippy and rustfmt clean. New tests cover the deblocker against a real JPEG fixture, the colouriser by correlation rather than error (a timid colouriser scores better on error and looks broken), and depth and faces where those models are installed — skipped, not failed, where they are not. The app was also driven headlessly: the 66 MB depth model removed and re-fetched through the dialog, and a Depth Blur preview reporting
Using Depth (Depth Blur) · MiDaS v2.1 small, Intel ISL, MIT.Known limits, stated in the README
The colouriser is small enough to be confidently wrong: it painted a red patch into a river in one test image. It knows sky, foliage, wood and skin, and hedges on anything whose colour is a choice. UltraFace has opinions about dogs.
🤖 Generated with Claude Code