Sensor

Sensor Scripts and Tutorials

---

Overview

• The first part of ISETCam uses computer graphics to model scene radiance.
• The second part of ISETCam uses optics to model the transformation of scene radiance to sensor irradiance.
• The third stage of ISETCam simulates the encoding of sensor irradiance by the image sensor.

Since the development of CMOS image sensors in the 1990s, the sensors structures and electronics have become increasingly sophisticated. Modern sensors comprise multiple components with a range of geometric and electronic properties. To simulate the response requires accounting for these factors.

For example, many sensors include a filter that limits the waveband of light that will be recorded (e.g., IR filters). Sensors, which are an array of pixels, have microlenses in front of the pixels. They also have color filters in front of each pixel. The pixel itself may contain one or multiple photodetectors. The photodetector signals may be processed in various ways, including for autofocus, for high dynamic range imaging, or for increasing low light sensitivty.

ISETCam groups the IR filter, color filters, and photodetector parameters as part of the sensor model. The parameters accounts for both geometric and electrical properties of the sensor. Sophisticated microlens properties, however, are part of the optics calculations in ISET3d. See the section on Lightfields.

Like other fundamental ISETCam structures, the sensor is managed using several functions (sensorCreate, sensorCompute, sensorSet/Get, sensorWindow, sensorPlot). A large number of sensor<TAB> functions are implemented for analyzing sensor properties. There are multiple sensor tutorials and scripts that illustrate various simulations.

Sensor overview

This code creates a simple scene and renders it through wavefront optics into a Sony IMX363 sensor model.

scene = sceneCreate; 
scene = sceneSet(scene,'fov',10);
oi = oiCreate('wvf'); 
oi = oiCompute(oi,scene,'crop',true);
sensor = sensorCreate('imx363'); 
sensor = sensorSet(sensor,'fov',10,oi);
sensor = sensorCompute(sensor,oi);

We can visualize the sensor data this way

sensorWindow(sensor);

Sensor model

To dive deeper into the sensor structure and its computations, visit the Sensor model page.

Human visual system

For many years, we also modeled human visual encoding using portions and specialization in ISETCam. Around 2015, Dave Brainard, Joyce Farrell, and Brian Wandell decided that the many specializations of the human encoding needed their own implementation. Until about 2023, we maintained parallel repositories for ISETCam and ISETBio.

In 2023-2024 we refactored the code in the two repositories, making ISETCam the base and ISETBio a specialization that built on ISETCam. These days the way we calculate for the human specializations is by using ISETBio and including both ISETCam and ISETBio on the Matlab path.