Repository navigation
Astro render mode design notes
Notes on a possible astrophotography render mode in the RAW inspector. Nothing here is implemented; this is a record of the options considered and which ones look worth building.
The constraint throughout is that the app has a good performance balance today and should not be inflated. Anything that cannot run inside the existing tiled renderer, at render scale for the fit view and full resolution only when zoomed or saving, is not a candidate.
Operate on the Bayer grid before demosaicing: apply a generalised Anscombe transform so the mixed Poisson–Gaussian sensor noise becomes additive Gaussian, denoise in that space, then invert.
Verdict: the stabilisation itself is cheap — a per-pixel square root after black level subtraction, plus the exact unbiased inverse. The naive inverse biases the background, which in an astro frame appears as a false gradient, so the unbiased form is not optional. The generalised form needs sensor gain and read noise, but both can be estimated from the frame itself in a single pass (per-tile variance against per-tile mean), so no camera database is required.
The cost lies entirely in the denoiser placed between the transforms. Split the frame into its four Bayer sub-planes, filter each independently and recombine before demosaicing: each plane is spatially coherent on its own, so there is no colour crosstalk and no demosaic artefact baked in ahead of the filter. NLM or BM3D is out of reach in Kotlin at 17.8 MP; an edge-aware separable filter is not.
Verdict: dual conversion gain is a sensor readout feature. Unless the camera wrote both gain reads into the file — most stills ARWs do not — there is nothing to exploit after the fact.
The rest of the proposal does not need DCG at all. A piecewise response curve with highlight rolloff and a shadow lift is another LUT in a pipeline that already builds a gamma LUT. Paired with per-channel highlight reconstruction — when red clips but green and blue do not, infer the clipped channel from the unclipped ratio rather than letting the pixel go white — star cores keep their colour instead of blowing out into white discs. This is the largest visible improvement of anything considered here and also the cheapest.
The original suggestion was to compare a rotated copy of the embedded JPEG against the unrotated one: stellar objects do not coincide after rotation, lens falloff is symmetric and does, so what survives is the vignetting.
The reasoning holds. Two practical refinements:
- Rotation is one instance of the general trick, and annulus statistics are the cheaper instance. Walking radially out from the optical centre and taking a median or low percentile of each thin annulus sees straight past the stars, since they are a tiny fraction of any annulus, and produces the falloff curve in one pass instead of one resample per rotation angle. Fit a low-order radial polynomial, or the classic cos⁴ model, to that curve to get the flat.
- The embedded JPEG is the wrong surface to measure. It is gamma encoded, while vignetting is multiplicative in linear light, so a fit made directly on it is wrong by the gamma exponent. It can also carry whatever shading compensation the camera chose to apply, in which case it shows less falloff than the sensor data actually has and a flat derived from it under-corrects — though with adapted or manual lenses the camera has no profile to apply and the two agree. The green plane of the raw is the honest estimator and is already in hand at that point in the pipeline.
Related and worth separating: light pollution and airglow produce a broadly linear gradient across the frame, while vignetting is radial. Fitting both terms lets sky gradient be subtracted and falloff divided out independently. The sky gradient correction is wanted on nearly every wide-field frame; the flat is not, so they deserve separate toggles.
Both must be toggles regardless. A lunar frame, or any bright central subject, is a radial brightness excess, and the fit will happily identify the subject as vignetting and flatten it. A per-annulus median resists this better than a mean, but is not immune.
In-camera shading compensation only applies when the lens identifies itself and the option is enabled. With adapted glass, a manual lens, or a telescope, nothing is applied — which is exactly the case where falloff is worst, because the image circle was never designed for the sensor. This matches what the project's own a6000 and a7R IIIa produce in practice. A single-frame shooter who does not stack and does not shoot flats currently has no good fix for this, in this app or elsewhere; a flat synthesised from the light frame itself is that fix, and it needs nothing but the pixels already in hand.
- Subtract black level first. A pedestal left in place flattens the measured curve and makes the division wrong.
- Fit each CFA plane separately — R, G1, G2, B — rather than a single luminance curve. Vignetting is wavelength dependent, so corners usually go cool or magenta as well as dark, and per-plane fitting corrects the colour cast at the same time. This is free, since the data is still on the Bayer grid at that point, and the colour cast is the part hardest to fix in post.
- Take a per-annulus median (or an upper percentile, see below) walking out from the optical centre, normalise the centre to 1, fit a low-order even polynomial in r² of the cos⁴ family, then divide.
- Fit the centre position as two extra parameters. Adapters and tilted telescope trains give falloff that is not concentric with the sensor.
- Corner noise. Correcting two stops in the corner multiplies the noise there fourfold. A 0–100% strength control takes most of the falloff out without turning the corners to mush, and pairs naturally with the denoise step if that lands.
- Foreground in frame. A treeline or horizon breaks the assumption that every annulus sees statistically the same sky: the dark foreground drags those annuli down and the fit over-brightens that side. A per-annulus upper percentile handles the common case where sky is the bright part, but a landscape astro frame will need the slider pulled back or the toggle off.
Comparing the raw's radial profile with the embedded JPEG's is not useful as a filter, but is a cheap diagnostic: it shows directly whether a given body and lens combination received any in-camera correction, rather than leaving it to impression.
A Bayer quad carries two green photosites sampling almost the same light. That redundancy is worth more in astro than anywhere else.
Subtracting the two greens cancels most of the scene and leaves noise. Divided by √2 it gives a per-frame measurement of noise against signal level — which is exactly the gain and read noise the generalised Anscombe transform needs. The VST option therefore requires no sensor database and no user input: the frame calibrates itself from its own green split. Sites where the difference is far out of family are hot pixels or cosmic ray hits, identified in the same pass.
Collapse each 2×2 quad to one output pixel: R, (G1+G2)/2, B. Half the linear resolution, but no interpolation of any kind, so no false colour on stars, no maze artefacts, √2 less noise in green, and a quarter of the pixels to process.
For undersampled star fields this is arguably more correct than demosaicing rather than a compromise. A star landing on a single photosite is the case bilinear interpolation handles worst: it smears the point and tints it. Astrophotographers choose superpixel output deliberately for this reason. It is also the cheapest renderer available here, which suits the app.
Fast, adapted or non-telecentric lenses produce steep ray angles at the frame edges, and microlens and crosstalk asymmetry then make G1 and G2 diverge systematically — the source of labyrinth texture after demosaic. Measuring that divergence radially both detects it and allows the two green planes to be equalised before demosaic. It reuses the annulus machinery of the flat-field fit, so it costs almost nothing once that exists.
The green split supplies the noise model, the noise model drives the VST denoise, and superpixel mode sidesteps demosaic artefacts altogether. The result is an astro path that never runs the normal demosaicer and is faster than the current developed view rather than slower.
Everything that must happen on the CFA grid — black level, flat field, green equalisation, VST denoise — runs first. Then exactly one reconstruction step turns the mosaic into RGB. Then the colour matrix and curve run on RGB. Superpixel does not precede a demosaic; it is that one reconstruction step, at half linear resolution, and an interpolating demosaic is the alternative choice for the same slot.
Interpolators, worst to best for star fields:
- AHD is actively wrong here despite being a good general default. It is edge directed, and an undersampled star resembles an edge from every direction at once, so it produces coloured cross artefacts on exactly the objects of interest.
- Bilinear, the current developed view's choice, smears and tints point sources but at least does so predictably.
- VNG is the traditional astro recommendation: it selects among gradient directions rather than averaging blindly, handling smooth sky and isolated points far better. Soft, which matters little here.
- LMMSE is the best practical option on noisy data specifically, which a single night-sky frame always is. Heavy; likely too heavy at 17.8 MP in Kotlin outside a tile.
- Superpixel performs no interpolation at all, so there is nothing to produce false colour or maze texture, at the cost of half the linear resolution. Fastest and highest fidelity per output pixel.
Two ordering and post-reconstruction details are worth more than the choice of interpolator:
- Chroma smoothing after reconstruction. Most visible star fringing is chroma error rather than luma. Blurring only the colour difference channels over a couple of pixels removes fringes while leaving star cores sharp, in one O(n) pass. Highest value per unit cost in this stage.
- White balance before reconstruction, not after. Interpolation assumes neighbouring channels have comparable magnitudes. Night-sky frames are heavily unbalanced, so that assumption fails and colour error is introduced before white balance can correct it. The current developed pipeline reconstructs first and balances after, which is fine for daylight and should be reversed on the astro path.
Suggested settings: superpixel as the astro default, VNG plus chroma smoothing as the full-resolution option, AHD avoided.
Using the embedded JPEG to shape a spatial filter is guided filtering, a real technique that runs at box-filter cost. It is the wrong guide for astro specifically: the camera has already crushed everything near the noise floor to black and sharpened what remains, so as a guide it asks for smoothing over exactly the faint nebulosity and dust that the mode exists to preserve, and asks for detail preservation only at star cores, which need no help. Self-guided filtering — a smoothed green plane guiding its own denoise — keeps the same O(n) structure with a guide that still contains the faint signal.
- Per-plane vignetting correction fitted from the sensor data, with a strength slider, off by default. Promoted to first because it is the only item here that nothing else in a single-frame workflow does.
- An Astro toggle beside Tune doing black-level-accurate subtraction, per-channel highlight reconstruction and the piecewise shadow-lift/highlight-rolloff curve. No new pipeline, no performance risk.
- Superpixel reconstruction as the astro path's default, replacing bilinear interpolation, with white balance moved ahead of reconstruction and a chroma smoothing pass after it. Cheaper than what exists, and removes star fringing rather than trading it for something. VNG as the full-resolution alternative if wanted.
- Sky gradient removal as a separate toggle, fitting the planar term alongside the radial one.
- Anscombe VST with per-plane edge-aware denoise, its noise model estimated from G1 − G2, at render scale for the fit view and full resolution only when zoomed or saving.
If steps 1 and 2 already look right on real frames, the rest becomes optional rather than load-bearing.