Sensor

Sensor Scripts and Tutorials

---

Overview

• The first part of ISETCam uses computer graphics to model scene radiance.
• The second part of ISETCam uses optics to model the transformation of scene radiance to sensor irradiance.
• The third stage of ISETCam uses device physics to model how irradiance creates the sensor response.

With the development of CMOS image sensors in the 1990s, sensor structures and electronics have become increasingly sophisticated. Modern sensors comprise multiple components with a wide range of geometric and electronic properties. Simulation requires accounting for these properties.

Image sensors are an array of pixels. Each pixel contains one (or more) photodetectors which convert the photons into electrons. The pixels, which spatially sample the irradiance, are almost always behind a microlenses and a color filter. The photodetector signals are processed both on the sensor and computers to manage focus, exposure duration, and high dynamic range imaging, and color processing.

ISETCam groups the IR filter, color filters, and photodetector parameters as part of the sensor model. The parameters accounts for both geometric and electrical properties of the sensor. Sophisticated microlens properties, however, are part of the optics calculations in ISET3d. See the section on Lightfields.

Like other fundamental ISETCam structures, the sensor is managed using several functions (sensorCreate, sensorCompute, sensorSet/Get, sensorWindow, sensorPlot). A large number of sensor<TAB> functions are implemented for analyzing sensor properties. There are multiple sensor tutorials and scripts that illustrate various simulations.

Sensor overview

This code creates a simple scene, renders it through wavefront optics, and then captures it using a Sony IMX363 sensor model.

scene = sceneCreate;    scene = sceneSet(scene,'fov',10);
oi = oiCreate('wvf');   oi = oiCompute(oi,scene,'crop',true);
sensor = sensorCreate('imx363');  sensor = sensorSet(sensor,'fov',10,oi);
sensor = sensorCompute(sensor,oi);

We can visualize the sensor data this way

sensorWindow(sensor);

There are many different sensor parameters stored in the sensor structure. The data, meaning the voltage at each pixel, is stored inside the sensor.data slot.

>> sensor

sensor = 

  struct with fields:

               name: 'imx363'
               type: 'sensor'
              pixel: [1×1 struct]
               rows: 362
               cols: 484
           spectrum: [1×1 struct]
               data: [1×1 struct]
       sigmaGainFPN: 0.7000
       gainFPNimage: []
     sigmaOffsetFPN: 0
     offsetFPNimage: []
         analogGain: 0.2037
       analogOffset: 0.0059
       quantization: [1×1 struct]
                cfa: [1×1 struct]
              color: [1×1 struct]
    integrationTime: 0.0134
                 AE: 1
                CDS: 0
             chartP: []
          noiseFlag: 2
            etendue: [362×484 double]
          noiseSeed: [1×1 struct]
           metadata: [1×1 struct]

The left side of the window shows the properties of the pixel, including its size, fill factor, electrical noise, and so forth. The right side summarizes the properties of the sensor as a whole, including the number of pixels, color filter pattern, dark signal nonuniformity (DSNU), photoresponse nonuniformity (PRNU), and so forth.

Sensor model

To dive deeper into the sensor structure and its computations, visit the Sensor model page.

Human visual system

For many years, we also modeled human visual encoding using portions and specialization in ISETCam. Around 2015, Dave Brainard, Joyce Farrell, and Brian Wandell decided that the many specializations of the human encoding needed their own implementation. Until about 2023, we maintained parallel repositories for ISETCam and ISETBio.

In 2023-2024 we refactored the code in the two repositories, making ISETCam the base and ISETBio a specialization that built on ISETCam. These days the way we calculate for the human specializations is by using ISETBio and including both ISETCam and ISETBio on the Matlab path.