AI Features

VirtualPaper includes three AI-powered image processing features, all running fully offline via ONNX Runtime. No data leaves your machine.

All models are bundled under Plugins/ML/ and loaded on demand — the session is initialized once and reused across calls.

---

Image Style Transfer

Model: AdaIN (Adaptive Instance Normalization) File: Plugins/ML/StyleTransfer/adain_style_transfer.onnx Implementation: StyleTransfer/AdaIn.cs

How it works

AdaIN is an encoder-decoder architecture for arbitrary neural style transfer. It works in three steps:

1. Encode — both the content image and the style image are independently passed through a shared VGG encoder to extract their feature representations.
2. Align statistics — the content features are normalized so that their per-channel mean and variance match those of the style features. This is the "adaptive instance normalization" step: it transfers the style's color and texture statistics onto the content structure without retraining.
3. Decode — the aligned features are passed through a decoder to reconstruct a stylized image.

An alpha parameter (0.0–1.0) controls the blend strength between the original content and the stylized result. At alpha = 1.0 (default), the full style is applied.

Pipeline in code

content image ──► LoadAndResizeImage (short-side → 512px)
                         │
style image ──────► LoadAndResizeImage (short-side → 512px)
                         │
              ┌──────────┴──────────┐
              │                     │
         ImageToTensor          ImageToTensor
         [1, 3, H, W]           [1, 3, H, W]
              │                     │
              └──────┬──────────────┘
                     │  + alpha scalar [1]
                     ▼
              ONNX Session.Run()
              inputs: "content", "style", "alpha"
                     │
                     ▼
              output tensor [1, 3, H, W]
                     │
              TensorToImageAndSave
              (resize back to original content dimensions)

Pixel values are normalized to [0, 1] before inference and clamped back to [0, 1] after. Images are converted between BGR (OpenCV native) and RGB before and after processing.

Cancellation

A CancellationToken is bound to RunOptions.Terminate. When the token fires, ONNX Runtime aborts inference at the next operator boundary, so cancellation is near-instant rather than waiting for the full forward pass to finish.

---

Super Resolution

Model: Real-ESRGAN (Generative Adversarial Network for super-resolution) File: Plugins/ML/SuperResolution/realesrgan_x4plus_dynamic.onnx Implementation: SuperResolution/Realesrgan.cs

How it works

Real-ESRGAN is a residual dense network trained with a discriminator to produce photorealistic upscaled images. The x4plus variant is designed for general natural images and upscales by a factor of 4×.

Two modes are exposed through the UI:

Mode	What it does
Clarity Restoration	Runs the model at scale ×1 (input = output resolution). The network still denoises and sharpens, but the final image is resized back to the original dimensions. File size may actually decrease due to higher compressibility after denoising.
Lossless Upscaling	Runs the model at scale ×2 or ×4. Resolution increases while detail is preserved.

Tiled inference

Full-image inference at 4× on large images would exceed memory budgets. The implementation splits the image into overlapping tiles and processes them in parallel:

Parameter	Value	Purpose
TileSize	512 px	Fixed input tile size. Keeps ONNX execution plan constant across tiles.
TileOverlap	16 px	Overlap between adjacent tiles. Strips are discarded after stitching to eliminate seam artifacts.
MaxParallelTiles	2	Maximum concurrent Session.Run() calls. IntraOpNumThreads is set to ProcessorCount / MaxParallelTiles so all tiles together saturate CPU cores without contention.

Edge tiles that are smaller than TileSize use edge-replication padding (repeating the border pixel outward) rather than zero-padding to prevent black-border artifacts in the upscaled output.

Stitching

Each tile's output region (after stripping the overlap margins) maps to a non-overlapping destination rectangle in the final image. Tiles write directly to the shared output Mat — because their destination rectangles are disjoint, no locking is needed.

input image
  ├── tile (0,0) ──► ONNX ──► strip overlap ──► write to output[0,   0  ]
  ├── tile (1,0) ──► ONNX ──► strip overlap ──► write to output[496, 0  ]
  ├── tile (0,1) ──► ONNX ──► strip overlap ──► write to output[0,   496]
  └── ...
final resize to targetWidth × targetHeight

Cancellation

Same strategy as Style Transfer: RunOptions.Terminate is set on cancellation. Each tile catches the resulting OnnxRuntimeException and returns silently; Parallel.ForEach's CancellationToken then surfaces a single OperationCanceledException to the caller.

---

Depth Estimation (3D Parallax)

Model: MiDaS v2 Small File: Plugins/ML/DepthEstimate/model-small.onnx Implementation: DepthEstimate/MiDaS.cs

This model is used internally to generate the depth map required for the 3D parallax effect on still images. It is not a user-facing feature in the AI+ panel.

How it works

MiDaS (Mixed Dataset approach for monocular depth estimation) takes a single RGB image and predicts a per-pixel relative depth map.

1. The input image is resized to the model's expected resolution (read from InputMetadata at load time).
2. Pixel values are normalized to [0, 1].
3. The model outputs a float array of relative depth values.
4. The output is min-max normalized to [0, 1] and then scaled to 8-bit grayscale (0–255).
5. The depth map is resized back to the original image dimensions and saved as a grayscale PNG alongside the wallpaper.

The depth map is consumed by the parallax renderer to shift image layers proportionally to cursor position, creating an illusion of depth.

---

Runtime & Performance

Item	Detail
Runtime	Microsoft.ML.OnnxRuntime
Image processing	OpenCvSharp4
Execution provider	CPU (default). CUDA / DirectML providers are commented out in code and can be enabled for GPU acceleration.
Memory	ArrayPool is used for tensor buffers to avoid per-inference GC pressure.
Thread safety	InferenceSession is created once and shared across calls. All per-call mutable state (output Mat, RunOptions) is local to each RunAndSave invocation.