Sensor

Sensor Scripts and Tutorials

---

Overview

• The first part of ISETCam uses computer graphics to model scene radiance.
• The second part of ISETCam uses optics to model the transformation of scene radiance to sensor irradiance.
• The third stage of ISETCam uses device physics to model how irradiance creates the sensor response.

With the development of CMOS image sensors in the 1990s, sensor structures and electronics have become increasingly sophisticated. Modern sensors comprise multiple components with a wide range of geometric and electronic properties. Simulation requires accounting for these properties.

Sensors are an array of photodetectors which convert the photons into electrons. The photodetectors, which spatially sample the irradiance, are usually behind a microlenses and a color filter. The pixel itself may contain one or multiple photodetectors. The photodetector signals may be processed in various ways, including for autofocus, for high dynamic range imaging, or for increasing low light sensitivty.

ISETCam groups the IR filter, color filters, and photodetector parameters as part of the sensor model. The parameters accounts for both geometric and electrical properties of the sensor. Sophisticated microlens properties, however, are part of the optics calculations in ISET3d. See the section on Lightfields.

Like other fundamental ISETCam structures, the sensor is managed using several functions (sensorCreate, sensorCompute, sensorSet/Get, sensorWindow, sensorPlot). A large number of sensor<TAB> functions are implemented for analyzing sensor properties. There are multiple sensor tutorials and scripts that illustrate various simulations.

Sensor overview

This code creates a simple scene and renders it through wavefront optics into a Sony IMX363 sensor model.

scene = sceneCreate; 
scene = sceneSet(scene,'fov',10);
oi = oiCreate('wvf'); 
oi = oiCompute(oi,scene,'crop',true);
sensor = sensorCreate('imx363'); 
sensor = sensorSet(sensor,'fov',10,oi);
sensor = sensorCompute(sensor,oi);

We can visualize the sensor data this way

sensorWindow(sensor);

The left side of the window shows the properties of the pixel, including its size, fill factor, electrical noise, and so forth. The right side summarizes the properties of the sensor as a whole, including the number of pixels, color filter pattern, dark signal nonuniformity (DSNU), photoresponse nonuniformity (PRNU), and so forth.

Sensor model

To dive deeper into the sensor structure and its computations, visit the Sensor model page.

Human visual system

For many years, we also modeled human visual encoding using portions and specialization in ISETCam. Around 2015, Dave Brainard, Joyce Farrell, and Brian Wandell decided that the many specializations of the human encoding needed their own implementation. Until about 2023, we maintained parallel repositories for ISETCam and ISETBio.

In 2023-2024 we refactored the code in the two repositories, making ISETCam the base and ISETBio a specialization that built on ISETCam. These days the way we calculate for the human specializations is by using ISETBio and including both ISETCam and ISETBio on the Matlab path.