+ +
+

FAQ

+
+

Loading and visualizing polysomnography data

+
+
+
+
+ How can I load an EDF file in Python? +
+
+
+

If you have polysomnography data in European Data Format (.edf), you can use the MNE package to load and preprocess your data in Python. MNE also supports several other standard formats (e.g. BrainVision, BDF, EEGLab). A simple preprocessing pipeline using MNE is shown below.

+
import mne
+# Load the EDF file
+raw = mne.io.read_raw_edf('MYEDFFILE.edf', preload=True)
+# Downsample the data to 100 Hz
+raw.resample(100)
+# Apply a bandpass filter from 0.1 to 40 Hz
+raw.filter(0.1, 40)
+# Select a subset of EEG channels
+raw.pick_channels(['C4-A1', 'C3-A2'])
+
+
+
+
+
+ +
+
+
+ Can I visualize my polysomnography data in YASA? +
+
+
+

YASA is a command-line software and does not support data visualization. To scroll through your data, we recommend the free software EDFBrowser (https://www.teuniz.net/edfbrowser/):

+
+_images/edfbrowser_with_hypnogram.png +
+
+
+
+

Event detection

+
+
+
+ How do the spindle detection and slow-waves detection algorithms work? +
+
+
+

The spindles detection is a custom adaptation of the Lacourse et al 2018 method. A step-by-step description of the algorithm can be found in this notebook.

+

The slow-waves detection combines the methods proposed in Massimini et al 2004 and Carrier et al 2011. A step-by-step description of the algorithm can be found here.

+
+

Important

+

Both algorithms have parameters that can (and should) be fine-tuned to your data, as explained in the next question.

+
+
+
+
+ +
+
+
+ How do I find the optimal parameters for my data? +
+
+
+

There are several parameters that can be adjusted in the spindles / slow-waves / artefact detection. While the default parameters should work reasonably well on most data, they might not be adequate for your data, especially if you’re working with specific populations (e.g. older adults, kids, patients with certain disorders, etc).

+

For the sake of example, let’s say that you have 100 recordings and you want to apply YASA to automatically detect the spindles. However, you’d like to fine-tune the parameters to your data. We recommend the following approach:

+
    +

  1. Grab a few representative recordings (e.g. 5 or 10 out of 100) and manually annotate the sleep spindles. You can use EDFBrowser to manually score the sleep spindles. Ideally, the manual scoring should be high-quality, so you may also ask a few other trained individuals to score the same data until you reach a consensus.

    2. +

  2. Apply YASA on the same recordings, first with the default parameters and then by slightly varying each parameter. For example, you may want to use a different detection threshold each time you run the algorithm, or a different frequency band for the filtering. In other words, you loop across several possible combinations of parameters. Save the resulting detection dataframe.

    3. +

  3. Finally, find the combination of parameters that give you the results that are the most similar to your own scoring. For example, you can use the combination of parameters that maximize the F1-score of the detected spindles against your own visual detection.

    4. +

  4. Use the “winning” combination to score the remaining recordings in your database.

    5. +
+
+
+
+ +
+
+
+ Can I manually add or remove detected events? +
+
+
+

YASA does not currently support visual editing of the detected events. However, you can import the events as annotations in EDFBrowser and edit the events from there. If you simply want to visualize the detected events (no editing), you can also use the plot_detection method.

+
+
+
+

Sleep staging

+
+
+
+ How accurate is YASA for automatic sleep staging? +
+
+
+

YASA was trained and evaluated on a large and heterogeneous database of thousands of polysomnography recordings, including healthy individuals and patients with sleep disorders. Overall, the results show that YASA matches human inter-rater agreement, with an accuracy of ~85% against expert consensus scoring. The full validation of YASA can be found in the preprint article:

+ +

However, our recommendation is that YASA should not replace human scoring, but rather serve as a starting point to speed up sleep staging. If possible, you should always have a trained sleep scorer visually check the predictions of YASA, with a particular emphasis on low-confidence epochs and/or N1 sleep epochs, as these are the epochs most often misclassified by the algorithm. +Finally, users can also leverage the yasa.plot_spectrogram() function to plot the predicted hypnogram on top of the full-night spectrogram. Such plots are very useful to quickly identify blatant errors in the hypnogram.

+
+_images/spectrogram.png +
+
+
+
+
+
+ How do I edit the predicted hypnogram? +
+
+
+

YASA does not come with a graphical user interface (GUI) and therefore editing the predicted hypnogram is not currently possible. The simplest way is therefore to export the hypnogram in CSV format and then open the file — together with the corresponding polysomnography data — in an external GUI, as shown below.

+
+

EDFBrowser

+

EDFBrowser is a free software for visualizing polysomnography data in European Data Format (.edf), which also provides a module for visualizing and editing hypnograms.

+

The code below show hows to export the hypnogram in an EDFBrowser-compatible format. It assumes that you have already run the algorithm and stored the predicted hypnogram in an array named hypno.

+
# Export to a CSV file compatible with EDFBrowser
+import numpy as np
+import pandas as pd
+hypno_export = pd.DataFrame({
+  "onset": np.arange(len(hypno)) * 30,
+  "label": hypno,
+  "duration": 30})
+hypno_export.to_csv("my_hypno_EDFBrowser.csv", index=False)
+
+
+

You can then import the hypnogram in EDFBrowser by clicking on the “Import annotations/events” in the “Tools” menu. Then, select the “ASCII/CSV” tab and change the parameters as follow:

+
+_images/edfbrowser_import_annotations.png +
+

Click “Import”. Once it’s done, the hypnogram can be enabled via the “Window” menu. A dialog will appear where you can setup the labels for the different sleep stages and the mapping to the annotations in the file. The default parameters should work. +When using the Annotation editor, the hypnogram will be updated realtime when adding, moving or deleting annotations. Once you’re done editing, you can export the edited hypnogram with “Export anotations/events” in the “Tools” menu.

+
+_images/edfbrowser_with_hypnogram.png +
+
+

SpiSOP

+

SpiSOP is an open-source Matlab toolbox for the analysis and visualization of polysomnography sleep data. It comes with a sleep scoring GUI. +As explained in the documentation, the hypnogram should be a tab-separated text file with two columns (no headers). The first column has the sleep stages (0: Wake, 1: N1, 2: N2, 3: N3, 5: REM) and the second column indicates whether the current epoch should be marked as artefact (1) or valid (0).

+
hypno_int = pd.Series(hypno).map({"W": 0, "N1": 1, "N2": 2, "N3": 3, "R": 5}).to_numpy()
+hypno_export = pd.DataFrame({"label": hypno_int, "artefact": 0})
+hypno_export.to_csv("my_hypno_SpiSOP.txt", sep="\t", header=False, index=False)
+
+
+
+

Visbrain

+

Visbrain is an open-source Python toolbox that includes a module for visualizing polysomnography sleep data and scoring sleep (see screenshot below).

+
+_images/visbrain.PNG +
+

Visbrain accepts several formats for the hypnogram. The code below show how to export the hypnogram in the Elan software format (i.e. a text file with the .hyp extension):

+
hypno_int = pd.Series(hypno).map({"W": 0, "N1": 1, "N2": 2, "N3": 3, "R": 5}).to_numpy()
+header = "time_base 30\nsampling_period 1/30\nepoch_nb %i\nepoch_list" % len(hypno_int)
+np.savetxt("my_hypno_Visbrain.txt", hypno_int, fmt='%s', delimiter=',', newline='\n',
+           header=header, comments="", encoding="utf-8")
+
+
+
+
+
+
+
+ Can I use YASA to score animal data and/or human intracranial data? +
+
+
+

YASA was only designed for human scalp data and as such will not work with animal data or intracranial data. Adding support for such data would require the two following steps:

+
    +

  1. Modifying (some of) the features. For example, rodent sleep does not have the same temporal dynamics as human sleep, and therefore one could modify the length of the smoothing window to better capture these dynamics.

    2. +

  2. Re-training the classifier using a large database of previously-scored data.

    3. +
+

Despite these required changes, one advantage of YASA is that it provides a useful framework for implementing such sleep staging algorithms. For example, one can save a huge amount of time by simply re-using and adapting the built-in yasa.SleepStaging class. +In addition, all the code used to train YASA is freely available at https://github.com/raphaelvallat/yasa_classifier and can be re-used to re-train the classifier on non-human data.

+
+
+
+

Others

+
+
+
+ How can I be notified of new releases? +
+
+
+

Pingouin uses outdated, a Python package that automatically checks if a newer version of YASA is available upon loading. Alternatively, you can click “Watch” on the GitHub of YASA. +Whenever a new release is out there, you can upgrade your version by typing the following line in a terminal window:

+
pip install --upgrade yasa
+
+
+
+
+
+ +
+
+
+ I am not a programmer, how can I contribute to YASA? +
+
+
+

There are many ways to contribute to YASA, even if you are not a programmer, for example, reporting bugs or results that are inconsistent with other softwares, improving the documentation and examples, or, even buying the developpers a coffee!

+
+
+
+ +
+
+
+ How can I cite YASA? +
+
+
+

To cite YASA, please use the preprint publication:

+ +

BibTeX:

+
@article {Vallat2021.05.28.446165,
+  author = {Vallat, Raphael and Walker, Matthew P.},
+  title = {A universal, open-source, high-performance tool for automated sleep staging},
+  elocation-id = {2021.05.28.446165},
+  year = {2021},
+  doi = {10.1101/2021.05.28.446165},
+  publisher = {Cold Spring Harbor Laboratory},
+  abstract = {The creation of a completely automated sleep-scoring system that is highly accurate, flexible, well validated, free and simple to use by anyone has yet to be accomplished. In part, this is due to the difficulty of use of existing algorithms, algorithms having been trained on too small samples, and paywall demotivation. Here we describe a novel algorithm trained and validated on +27,000 hours of polysomnographic sleep recordings across heterogeneous populations around the world. This tool offers high sleep-staging accuracy matching or exceeding human accuracy and interscorer agreement no matter the population kind. The software is easy to use, computationally low-demanding, open source, and free. Such software has the potential to facilitate broad adoption of automated sleep staging with the hope of becoming an industry standard.Competing Interest StatementThe authors have declared no competing interest.AbbreviationsAHIapnea-hypopnea indexBMIbody mass indexEEGelectroencephalogramEOGelectrooculogramEMGelectromyogramOSAobstructive sleep apneaPSGpolysomnographyMCCMatthews correlation coefficientNREMnon rapid eye movement (sleep)REMrapid eye movement (sleep)},
+  URL = {https://www.biorxiv.org/content/early/2021/05/28/2021.05.28.446165},
+  eprint = {https://www.biorxiv.org/content/early/2021/05/28/2021.05.28.446165.full.pdf},
+  journal = {bioRxiv}
+}
+
+
+
+
+
+
+ + +
+ +
+
+
+
+

+ Back to top + +
+ + +

+

+ © Copyright 2018-2021, Dr. Raphael Vallat, Center for Human Sleep Science, UC Berkeley.
+ Created using Sphinx 3.0.3.
+

+
+
+ + \ No newline at end of file diff --git a/docs/build/html/generated/yasa-plot_spectrogram-1.png b/docs/build/html/generated/yasa-plot_spectrogram-1.png index 3851613..f14dc46 100644 Binary files a/docs/build/html/generated/yasa-plot_spectrogram-1.png and b/docs/build/html/generated/yasa-plot_spectrogram-1.png differ diff --git a/docs/build/html/generated/yasa-plot_spectrogram-2.png b/docs/build/html/generated/yasa-plot_spectrogram-2.png index fcc7761..29b856d 100644 Binary files a/docs/build/html/generated/yasa-plot_spectrogram-2.png and b/docs/build/html/generated/yasa-plot_spectrogram-2.png differ diff --git a/docs/build/html/generated/yasa-topoplot-1.png b/docs/build/html/generated/yasa-topoplot-1.png index 9b49839..d2cae33 100644 Binary files a/docs/build/html/generated/yasa-topoplot-1.png and b/docs/build/html/generated/yasa-topoplot-1.png differ diff --git a/docs/build/html/generated/yasa-topoplot-2.png b/docs/build/html/generated/yasa-topoplot-2.png index bc0e522..2011471 100644 Binary files a/docs/build/html/generated/yasa-topoplot-2.png and b/docs/build/html/generated/yasa-topoplot-2.png differ diff --git a/docs/build/html/generated/yasa-transition_matrix-1.png b/docs/build/html/generated/yasa-transition_matrix-1.png index 11ddc13..2fa259f 100644 Binary files a/docs/build/html/generated/yasa-transition_matrix-1.png and b/docs/build/html/generated/yasa-transition_matrix-1.png differ diff --git a/docs/build/html/generated/yasa.REMResults.html b/docs/build/html/generated/yasa.REMResults.html index aa9bd7a..daa0906 100644 --- a/docs/build/html/generated/yasa.REMResults.html +++ b/docs/build/html/generated/yasa.REMResults.html @@ -3,9 +3,10 @@ - yasa.REMResults — yasa 0.4.1 documentation + yasa.REMResults — yasa 0.5.0 documentation + @@ -37,13 +38,14 @@ yasa - 0.4.1 + 0.5.0
  • Functions
    • +
  • FAQ
  • What's new
  • Contribute
    • @@ -93,7 +95,7 @@

    yasa.REMResults

    Attributes
    -
    _eventspandas.DataFrame

    Output detection dataframe

    +
    _eventspandas.DataFrame

    Output detection dataframe

    _dataarray_like

    EOG data of shape (n_chan, n_samples), where the two channels are LOC and ROC.

    @@ -126,19 +128,22 @@

    yasa.REMResults

    __init__(self, events, data, sf, ch_names, …)

    Initialize self.

    -

    get_mask(self)

    +

    get_coincidence_matrix(self[, scaled])

    +

    + +

    get_mask(self)

    Return a boolean array indicating for each sample in data if this sample is part of a detected event (True) or not (False).

    -

    get_sync_events(self[, center, time_before, …])

    +

    get_sync_events(self[, center, time_before, …])

    Return the raw or filtered data of each detected event after centering to a specific timepoint.

    -

    plot_average(self[, center, time_before, …])

    +

    plot_average(self[, center, time_before, …])

    Plot the average REM.

    -

    plot_detection(self)

    +

    plot_detection(self)

    Plot an overlay of the detected events on the signal.

    -

    summary(self[, grp_stage, aggfunc, sort])

    +

    summary(self[, grp_stage, aggfunc, sort])

    Return a summary of the REM detection, optionally grouped across stage.

    @@ -168,13 +173,13 @@

    yasa.REMResults

    filttuple

    Optional filtering to apply to data. For instance, filt=(1, 30) will apply a 1 to 30 Hz bandpass filter, and filt=(None, 40) will apply a 40 Hz lowpass filter. Filtering is done using default -parameters in the mne.filter.filter_data() function.

    +parameters in the mne.filter.filter_data() function.

    Returns
    -
    df_syncpandas.DataFrame

    Ouput long-format dataframe:

    +
    df_syncpandas.DataFrame

    Ouput long-format dataframe:

    'Event' : Event number
 'Time' : Timing of the events (in seconds)
 'Amplitude' : Raw or filtered data for event
@@ -205,7 +210,7 @@ 
    yasa.REMResults
    
 
    filttuple
    Optional filtering to apply to data. For instance, filt=(1, 30)
 will apply a 1 to 30 Hz bandpass filter, and filt=(None, 40)
 will apply a 40 Hz lowpass filter. Filtering is done using default
-parameters in the mne.filter.filter_data() function.
    
+parameters in the mne.filter.filter_data() function.
    
 
    
 
    figsizetuple
    Figure size in inches.
    
 
    
@@ -255,7 +260,7 @@ 
    yasa.REMResults
    
       
     
    
     
    
-        © Copyright 2018-2021, Raphael Vallat.
    
+        © Copyright 2018-2021, Dr. Raphael Vallat, Center for Human Sleep Science, UC Berkeley.
    
       Created using Sphinx 3.0.3.
    diff --git a/docs/build/html/generated/yasa.SWResults.html b/docs/build/html/generated/yasa.SWResults.html index 873b938..f02895b 100644 --- a/docs/build/html/generated/yasa.SWResults.html +++ b/docs/build/html/generated/yasa.SWResults.html @@ -3,9 +3,10 @@ - yasa.SWResults — yasa 0.4.1 documentation + yasa.SWResults — yasa 0.5.0 documentation + @@ -37,13 +38,14 @@ yasa - 0.4.1 + 0.5.0
    • Functions
      • +
    • FAQ
    • What's new
    • Contribute
      • @@ -93,7 +95,7 @@

      yasa.SWResults

      Attributes
      -
      _eventspandas.DataFrame

      Output detection dataframe

      +
      _eventspandas.DataFrame

      Output detection dataframe

      _dataarray_like

      EEG data of shape (n_chan, n_samples).

      @@ -124,23 +126,78 @@

      yasa.SWResults

      __init__(self, events, data, sf, ch_names, …)

      Initialize self.

      -

      get_mask(self)

      +

      get_coincidence_matrix(self[, scaled])

      +

      Return the (scaled) coincidence matrix.

      + +

      get_mask(self)

      Return a boolean array indicating for each sample in data if this sample is part of a detected event (True) or not (False).

      -

      get_sync_events(self[, center, time_before, …])

      +

      get_sync_events(self[, center, time_before, …])

      Return the raw data of each detected event after centering to a specific timepoint.

      -

      plot_average(self[, center, hue, …])

      +

      plot_average(self[, center, hue, …])

      Plot the average slow-wave.

      -

      plot_detection(self)

      +

      plot_detection(self)

      Plot an overlay of the detected slow-waves on the EEG signal.

      -

      summary(self[, grp_chan, grp_stage, …])

      +

      summary(self[, grp_chan, grp_stage, …])

      Return a summary of the SW detection, optionally grouped across channels and/or stage.

      +
      +
      +get_coincidence_matrix(self, scaled=True)[source]
      +

      Return the (scaled) coincidence matrix.

      +
      +
      Parameters
      +
      +
      scaledbool

      If True (default), the coincidence matrix is scaled (see Notes).

      +
      +
      +
      +
      Returns
      +
      +
      coincidencepd.DataFrame

      A symmetric matrix with the (scaled) coincidence values.

      +
      +
      +
      +
      +

      Notes

      +

      Do slow-waves occur at the same time? One way to measure this is to +calculate the coincidence matrix, which gives, for each pair of +channel, the number of samples that were marked as a slow-waves in both +channels. The output is a symmetric matrix, in which the diagonal is +simply the number of data points that were marked as a slow-waves in +the channel.

      +

      The coincidence matrix can be scaled (default) by dividing the output +by the product of the sum of each individual binary mask, as shown in +the example below. It can then be used to define functional +networks or quickly find outlier channels.

      +

      References

      + +

      Examples

      +

      Calculate the coincidence of two binary mask:

      +
      >>> import numpy as np
+>>> x = np.array([0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1])
+>>> y = np.array([0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1])
+>>> x * y
+array([0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1])
+
      +
      +
      >>> (x * y).sum()  # Coincidence
+3
+
      +
      +
      >>> (x * y).sum() / (x.sum() * y.sum())  # Scaled coincidence
+0.12
+
      +
      +
      +
      get_mask(self)[source]
      @@ -166,13 +223,13 @@

      yasa.SWResults

      filttuple

      Optional filtering to apply to data. For instance, filt=(1, 30) will apply a 1 to 30 Hz bandpass filter, and filt=(None, 40) will apply a 40 Hz lowpass filter. Filtering is done using default -parameters in the mne.filter.filter_data() function.

      +parameters in the mne.filter.filter_data() function.

      Returns
      -
      df_syncpandas.DataFrame

      Ouput long-format dataframe:

      +
      df_syncpandas.DataFrame

      Ouput long-format dataframe:

      'Event' : Event number
 'Time' : Timing of the events (in seconds)
 'Amplitude' : Raw or filtered data for event
@@ -207,7 +264,7 @@ 
      yasa.SWResults
      
 
      filttuple
      Optional filtering to apply to data. For instance, filt=(1, 30)
 will apply a 1 to 30 Hz bandpass filter, and filt=(None, 40)
 will apply a 40 Hz lowpass filter. Filtering is done using default
-parameters in the mne.filter.filter_data() function.
      
+parameters in the mne.filter.filter_data() function.
      
 
      
 
      figsizetuple
      Figure size in inches.
      
 
      
@@ -274,7 +331,7 @@ 
      yasa.SWResults
      
       
     
      
     
      
-        © Copyright 2018-2021, Raphael Vallat.
      
+        © Copyright 2018-2021, Dr. Raphael Vallat, Center for Human Sleep Science, UC Berkeley.
      
       Created using Sphinx 3.0.3.
      diff --git a/docs/build/html/generated/yasa.SleepStaging.html b/docs/build/html/generated/yasa.SleepStaging.html index 2499a21..a474d51 100644 --- a/docs/build/html/generated/yasa.SleepStaging.html +++ b/docs/build/html/generated/yasa.SleepStaging.html @@ -3,9 +3,10 @@ - yasa.SleepStaging — yasa 0.4.1 documentation + yasa.SleepStaging — yasa 0.5.0 documentation + @@ -37,13 +38,14 @@ yasa - 0.4.1 + 0.5.0
      • Functions
        • +
      • FAQ
      • What's new
      • Contribute
        • @@ -99,7 +101,7 @@

        yasa.SleepStaging

        Parameters
        -
        rawmne.io.BaseRaw

        An MNE Raw instance.

        +
        rawmne.io.BaseRaw

        An MNE Raw instance.

        eeg_namestr

        The name of the EEG channel in raw. Preferentially a central electrode referenced either to the mastoids (C4-M1, C3-M2) or to the @@ -124,6 +126,16 @@

        yasa.SleepStaging

        Notes

        +

        If you use the SleepStaging module in a publication, please cite the following preprint:

        + +

        We provide below some key points on the algorithm and its validation. For more details, +we refer the reader to the preprint article. If you have any questions, +make sure to first check the +FAQ section of the documentation. +If you did not find the answer to your question, please feel free to open an issue on GitHub.

        1. Features extraction

        For each 30-seconds epoch and each channel, the following features are calculated:

        @@ -140,9 +152,10 @@

        yasa.SleepStaging

      • Higuchi and Petrosian fractal dimension

      In addition, the algorithm also calculates a smoothed and normalized -version of these features. Specifically, a 5-min centered weighted rolling -average and a 10 min past rolling average are applied. The resulting -smoothed features are then normalized using a robust z-score.

      +version of these features. Specifically, a 7.5 min centered +triangular-weighted rolling average and a 2 min past rolling average +are applied. The resulting smoothed features are then normalized using a +robust z-score.

      The data are automatically downsampled to 100 Hz for faster computation.

      2. Sleep stages prediction

      @@ -209,6 +222,9 @@

      yasa.SleepStaging

      >>> sls.plot_predict_proba()