Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve blink documentation #463

Merged
merged 2 commits into from Oct 15, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 22 additions & 2 deletions src/core/best-practices.md
Expand Up @@ -109,9 +109,29 @@ consecutive detections that should have been grouped to a single fixation (type
if the gaze points are dispersed over 2 degrees due to bigger eye rotations, filtering with a maximum
dispersion of 1 degree could classify two separate fixations instead of one. Setting the maximum
dispersion too high might group fixations together that should have been considered separate fixations.
Ensure the maximum dispersion accommodates your viewing task.
Ensure the maximum dispersion accommodates your viewing task.

## Sychronization
## Blink Detector Thresholds
In general, the default values for the [Blink Detector](/core/software/pupil-player/#blink-detector) should work reasonably
well on good quality eye data with robust pupil detection. However, it is often necessary to adjust the thresholds in the
event that [blinks](/core/terminology/#blinks) are not accurately classified. Therefore, it is important to understand
N-M-T marked this conversation as resolved.
Show resolved Hide resolved
the types of errors that can occur and to be able to spot them when they occur, thereby enabling you to make appropriate
adjustments.

### Errors
**A.** False negatives - Blinks are not being detected, e.g. due to the onset threshold being too high
**B.** False positives - The onset threshold is set too low and blinks are classified even though they did not occur
**C.** The end of a blink is not detected due to the offset threshold being too low. This can lead to erroneous blinks
papr marked this conversation as resolved.
Show resolved Hide resolved
that have unreasonable durations

### Pupil Detection and Blinks
It is worth noting that poor pupil detection in general can lead to false negatives. In such instances, adjusting the
thresholds can make it easier to detect blinks, but also increases the chance of false positives. It is worth taking the
time to ensure an optimal setup with regards to [eye camera positioning](/core/#_3-check-pupil-detection) and
[2d detector settings](/core/software/pupil-capture/#fine-tuning-pupil-detection) so that the pupils are well-detected
when the eyes are open.

## Synchronization
Pupil Core is often used concurrently with third-party devices and software (e.g. physiological sensors,
motion capture, stimuli presentation). In order to correlate the data between these, temporal alignment is of great importance.

Expand Down
17 changes: 5 additions & 12 deletions src/core/software/pupil-capture.md
Expand Up @@ -449,19 +449,12 @@ For all new projects we strongly recommend using Apriltags!



### Blink Detection
The pupil detection algorithm assigns a `confidence` value to each pupil datum. It represents the quality of the detection result. While the eye is closed the assigned confidence is very low. The `Blink Detection` plugin makes use of this fact by defining a `blink onset` as a significant confidence drop - or a `blink offset` as a significant confidence gain - within a short period of time. It always operates on 2D pupil confidence data. The plugin creates a `blink` event for each event loop iteration in the following format:

```python
{ # blink datum
'topic': 'blink',
'confidence': <float>, # blink confidence
'timestamp': <timestamp float>,
'base_data': [<pupil positions>, ...]
'type': 'onset' or 'offset'}
```
### Blink Detector
The online blink detector classifies [blinks](/core/terminology/#blinks) according to onset and offset thresholds
associated with [2D pupil confidence](/core/terminology/#confidence). See the
[Blink Detector documentation](/core/software/pupil-player/#blink-detector) for more information.

The `Filter length` is the time window's length in which the plugin tries to find such confidence drops and gains. The plugin fires the above events if the blink confidence within the current time window exceeds the `onset` or `offset` confidence threshold. Setting both thresholds to `0` will always trigger blink events, even if the confidence is very low. This means that onsets and offsets do not appear necessarily as pairs but in waves.
Read more about accessing blink detection results in real-time in the [developer documentation](/developer/core/network-api/#blink-messages).

### Annotations
The `Annotation Capture` plugin allows you to mark timestamps with a label -- sometimes
Expand Down
51 changes: 51 additions & 0 deletions src/core/software/pupil-player.md
Expand Up @@ -217,6 +217,57 @@ Toggle `Show fixations` to show a visualization of fixations. The blue number is

You can find more information in our [dedicated fixation detection section](/core/terminology/#fixations "Pupil Core terminology - fixations").

#### Blink Detector
Pupil Core's Blink Detector leverages the fact that [2D pupil confidence](/core/terminology/#confidence) drops rapidly
in both eyes during a [blink](/core/terminology/#blinks) as the pupil becomes obscured by the eyelid, followed by a rapid
rise in confidence as the pupil becomes visible again.

The Blink Detector processes [2D pupil confidence](/core/terminology/#confidence) values by convolving them with a
[filter](https://github.com/pupil-labs/pupil/blob/eb8c2324f3fd558858ce33f3816972d93e02fcc6/pupil_src/shared_modules/blink_detection.py#L360).
The filter response – called 'activity' – spikes the sharper the confidence drop is and *vice versa* for confidence
increases.

Blinks are subsequently detected based on onset and offset confidence thresholds and a filter length in seconds.

- **Onset Confidence Threshold:** The threshold that the filter response ('Activity') must rise above to classify the
onset of a blink, corresponding to a sudden **drop** in 2D pupil detection confidence
- **Offset Confidence Threshold:** The threshold that the filter response ('Activity') must fall below to classify the
end of a blink, corresponding to a **rise** in 2D pupil detection confidence
- **Filter Length:** The time window's length in which the detector tries to find confidence drops and gains

In Pupil Player, the green data series labelled 'Activity' is the filter response, and the yellow bars that denote
onset and offset confidence are above and below this, respectively. The blink detection result is shown by the red
line, characterised by a step function where blinks have been classified.

<div class="pb-4">
<img src="../../media/core/imgs/pp-blinks.jpg" style="display:flex;margin:0 auto;">
</div>

:::tip
<v-icon large color="info">info_outline</v-icon>
See our [Best Practices](/core/best-practices/#blink-detector-thresholds) for tips on choosing
appropriate Blink Detector thresholds
:::

Results are exported in `blinks.csv` with the following columns:
| Key | Description |
|:--------------------|:-----------------------------------------------------------|
| `id` | Numerical ID of blink |
| `start_timestamp` | Blink end timestamp (ms) |
| `duration` | Blink duration (s) |
papr marked this conversation as resolved.
Show resolved Hide resolved
| `end_timestamp` | Blink start timestamp (ms) |
papr marked this conversation as resolved.
Show resolved Hide resolved
| `start_frame_index` | Blink start world frame index |
| `index` | Blink median world frame index |
| `end_frame_index` | Blink end world frame index |
| `confidence` | Mean of absolute filter response during blink clamped at 1 |
| `filter_response` | Filter response ('Activity') during blink |
| `base_data` | Timestamps of data associated with blink |

:::tip
<v-icon large color="info">info_outline</v-icon>
Blink count is included in the `blink_detection_report.csv`
:::

#### Head Pose Tracking
This plugin uses fiducial markers ([apriltag](https://april.eecs.umich.edu/software/apriltag.html)) to build a 3d model of the environment and track the headset's pose within it. Note, only markers of the default `tag36h11` family are currently supported by the head pose tracker plugin.

Expand Down
7 changes: 7 additions & 0 deletions src/core/terminology.md
Expand Up @@ -57,6 +57,13 @@ Quality assessment of the _pupil detection_ for a given _eye_ image.

_Gaze data_ inherits the _confidence_ value from the _pupil data_ it was based on.

## Blinks

Blinking is defined as a rapid closing and re-opening of the eyelids. Characteristics associated with blinks, like blink
rate, can provide insights into factors such as cognitive workload and stress. As such, blinking is of increasing interest
to researchers. Pupil Core implements a blink detector that provides blink classification both in
[real-time](/core/software/pupil-capture/#blink-detector) and [post-hoc](/core/software/pupil-player/#blink-detector) contexts.

## Calibration

Process of understanding the relationship between _pupil positions_ and their
Expand Down
11 changes: 11 additions & 0 deletions src/developer/core/network-api.md
Expand Up @@ -256,6 +256,17 @@ The Offline Fixation Detector in Pupil Player additionally includes the followin
# ...
```

## Blink Messages
The online [Blink Detector](/core/software/pupil-capture/#blink-detector) in Pupil Capture publishes the following notification:
```python
{ # blink datum
'topic': 'blink',
'confidence': <float>, # blink confidence
'timestamp': <timestamp float>,
'base_data': [<pupil positions>, ...]
'type': 'onset' or 'offset'
}
```

### Remote Annotations
You can also create [annotation](/core/software/pupil-capture/#annotations) events
Expand Down
Binary file added src/media/core/imgs/pp-blinks.jpg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.