[Sleep] Generalize Sleep module to facilitate sleep scoring of non-human EEG/LFP data

[paulbrodersen]

First of all: fantastic project. It installs fine (although I would change the installation instructions a bit), mostly everything just works, the docs are great, and the code is very readable (at least in the parts that I have looked at).

Secondly, I am raising this issue because I hope this will ultimately end in a PR (from me) but I wanted to get the conversation started before I start coding, so that you can stop me dead in the track before I veer off into areas that you (or any of the other maintainers) don't want to go.

Goal

Generalize the Sleep submodule to allow / facilitate sleep scoring of non-human EEG/EMG data, specifically from mouse.

Motivation

I have implemented an automated sleep scoring algorithm that achieves a very high accuracy / concordance with experienced experimenters (98.5% on a held out test set). Nevertheless, most experimenters would still like to be able to amend the annotation, e.g. to annotate artifact states.
However, the sleep scoring software (sleepsign) that people in my lab and other labs in the institution use is a) proprietary (including the file format) and b) doesn't allow the import of a hypnogram from a different source (it has a lot of other issues, too, but those are the most pressing for me at the moment). Trying to integrate my pipeline with their existing sleep scoring pipeline is hence a bit of a non-starter. The integration with visbrain, in contrast, seems pretty straightforward (I love the fact that you provide templates to replace existing functionality).

Proposed changes

This list will undoubtedly grow as I and others in my lab actively use visbrain / Sleep.
So far I have got:

1. Allow definition of arbitrary vigilance states, e.g. in a config file, instead of hard coding N1, N2, N3, REM, wake, and artifact. "Mouse people" don't distinguish between different non-REM states; however, they do denote an additional "sleep movement" state, which occurs when the animal moves (and is potentially briefly awake) during sleep phases.

2. Associated with arbitrary vigilance states, allow arbitrary(-ish) shortcut keys, which could again be implemented by separating out some keybindings to a config file. My intended target audience has scored dozens and dozens of complete days of data at 4 second intervals, and I don't want to mess with their muscle memory at this stage.

I would love to hear your thoughts and feedback on this proposal.

I would completely understand if you would rather keep everything human-centric, as the proposed generalisation will increase the complexity and scope of the project. That being said, the code base seems pretty modular so I don't think this would entail massive changes all in all (but you would have a better idea of this). Also, such a move could drastically increase your user base (which would be much deserved!), and bring in some fresh ideas.

Looking forward to your reply,
Paul

[EtienneCmb]

Hello @paulbrodersen ,

First of all, thanks for compliments. Indeed, the entire idea of Visbrain is to offer modularity. About the doc and the description of the installation, feel free to fix/enhance the current description.

For the generalization to non-humans, I've nothing against but may be @raphaelvallat have something to add.

But it will requires some minor and major adjustments. May be the best would be to have a json configuration file. Something like this :

{
    "REM": {
        "color": "green",
        "shortcut": "r",
        "value": 0,
    },
    "N1": {
        "color": "red",
        "shortcut": "1"
        "value": 1
    },
    "N2": {
        "color": "blue",
        "shortcut": "2"
        "value": 2
    }
}

An then the Sleep class could take it as an input, something like Sleep(stage_config='myconfig.json').

[paulbrodersen]

That is exactly what I had in mind. The human config file would obviously be the default (thus keeping backwards compatibility) but then we could provide other config files and maybe a recipe in the docs on how to make your own.

Thanks for the quick reply!

[grahamfindlay]

+1, I would also love to see this project add support scoring for non-human EEG, LFP, and EMG. My experience very much echos Paul's. I work in a lab that uses SleepSign, because it is the industry standard software for scoring animal sleep. I would love to see an open source alternative.

As Paul mentioned, most people in the field score in 4-second epochs. Perhaps the largest impediment to using Sleep for manual scoring of animal sleep is that there does not currently appear to be a way to separate the epoch length from the window length. Unlike with human sleep, it is essential to be able to see epochs immediately before and after the 4-second epoch being scored. SleepSign's UI for this is excellent: example.

Flexible vigilance state definitions are also essential. As Paul mentioned, rodent vigilance states differ from those of humans, and also differ from lab to lab. Some labs, for instance, allow for an "intermediate stage" during transitions to REM, and often distinguish between "quiet wake" and "active wake." I know that my lab often needs to score non-sleep states (e.g. drug induced EEG states) as well. I think it is worth pointing out that this feature would likely also be useful for researchers doing human studies as well. For example, my lab uses custom software that include "seizure" and "post-ictal" vigilance states, and uses this to score sleep in epileptic patients.

If these two things were changed, I think that you would likely see other researchers transition to Sleep.

[EtienneCmb]

@raphaelvallat what do you think about it?

[raphaelvallat]

Hi all,
Thank you @paulbrodersen and @grahamfindlay for your great suggestions.

@EtienneCmb it seems to me that the hardest part is to separate the epoch length from the window length, isn't it? That said, I agree that this could indeed represent a big step forward for sleep.

@paulbrodersen quick question, is there any chance that you could send us a SleepSign hypnogram file? I think we could definitely add an option to load such file formats if you think that this might be useful too.

Have a great day,
Raphael

[paulbrodersen]

@raphaelvallat I can send you an example file, sure. I don't have a copy of sleepsign myself but I am meeting my collaborator early next week, and then I can get one for you. However, a problem with the sleepsign hypnogram format is that it is not a format. It is my understanding that pretty much everything in that file can change depending on your export settings, including number of header rows, number of columns in the body, etc. You can probably mitigate most of that with some regex but it might still end up a nightmare to maintain.

Personally, I would stick to the sane formats that you are already supporting. Anybody I know who uses sleepsign then analyzes their data in python or matlab. So I would maybe add a recipe in the docs on how to convert sleepsign hypnograms to your format (and then let the user do the work) but I wouldn't necessarily make it part of the pipeline.

All of that being said, I can ask my collaborator to export the hypnogram of one example data set with a bunch of different export settings, and then either you or I can have a go at trying to figure out if we can reliably recover the states and time intervals from those regardless of the export settings. It might end up not being all that bad -- I just haven't tried yet.

[EtienneCmb]

@raphaelvallat actually I don't think the hardest part would be to separate epoch length from the window length.

To me the hardest part is to make stages and properties modular (i.e colors, shortcuts...). But this is totally doable, just a question of time.

In the first place, I can to try to make our current version modular using a json file without hard coded stages and properties. The split of epoch length and window length could comes in a second place.

@paulbrodersen

• in your first message you said that installation description could be improved. I think this is a very important aspect of the doc. How do you think we can make this description better? (thx in advance :) )
• as you already know, Sleep offers the possibility to replace the default detections. I'm just thinking that we also should provide the possibility to use an automatic scoring algo.

[grahamfindlay]

@raphaelvallat The SleepSign "Results Analysis File" format is proprietary, but you can export the hynogram info as a text file by going to "Trend -> Text output." As Paul mentioned, there are various options, but here is an minimal example of what is produced by default: SleepSign_TrendExample_TSV.txt. I agree with Paul, that providing a recipe for people to convert these files to a format you already support makes sense.

@EtienneCmb When I first installed Sleep, it took me a moment to realize that it is a module of the visbrain package, and not available as a standalone package. It only took me about 30 seconds to figure this out, but I think that adding a note in the "Import and Use Sleep" section explicitly saying that you need to install visbrain first might be useful.

[paulbrodersen]

@EtienneCmb Regarding the installation instructions, I would additionally note that if you want to use any of the Sleep functionality depending on MNE-python, that it makes sense to setup the MNE conda environment first before doing anything else. MNE python has a ton of dependencies, some with very specific version requirements. Luckily, they provide an environment file, otherwise that would be a nightmare. The flipside of this is that if you first install all of the visbrain dependencies and optional dependencies that are listed in the installation instructions before MNE, you find yourself having to abandon the virtual environment you just set up in favour of the MNE environment.

Regarding the automatic scoring, I am happy to write some bindings to the stuff that I have been developing. The advantages of my algorithm are that

1. it is pretty easy to use as there are no free parameters apart from the data that you feed it and the target time resolution of the annotation, and

2. it does pretty well on mouse data (98% concordance with manual scoring at 1 second time resolution).

I will have a look at human data next week. If you are interested, I will keep you updated.
The disadvantage of my method with regard to an integration with Sleep is that the algorithm is strictly supervised, i.e. the model needs some training data. That would mean that we either require that the user provides training data or a trained model, or we provide a selection of pre-trained models. For mouse data, the latter would not be too problematic as the data source is fairly standardised: it is my understanding that independent of the full setup, pretty much everybody also records 2 EEGs (frontal and occipital referenced to cerebellum) and 1 EMG, all from the same standardised locations. I don't know if the situation is similar in humans, i.e. if there are just one or a few typical setups that we could provide pre-trained models for. Also this would add scikit-learn and pomegranate to the list of dependencies (I could maybe get rid of the scikit-learn dependency as I am only using 1 function and 1 class from it).

@raphaelvallat For comparison with @grahamfindlay's sleepsign hypnogram, here are the first 100 lines from one of my files.
There is no separate footer, such that the format of the rest of the file matches lines 20-100 in the file given.

@grahamfindlay Thanks for posting the default file format for me, now I don't have to chase my collaborator.

[EtienneCmb]

Thanks @paulbrodersen for your feedback about the installation. I don't think you need to install every optional dependencies of both visbrain and MNE in order to only load files. But I can list the needed packages for Sleep only so that users install only what's needed. Also, I can specify that Sleep is a module of visbrain in the doc (thanks @grahamfindlay ).

[jasmainak]

I don't think you really need the MNE environment unless you plan on using mayavi in MNE. I've used it with plain pip installation and it works fine

[dougollerenshaw]

My lab is exploring software for mouse EEG analysis and @grahamfindlay just pointed me to this thread when I asked him by email for software suggestions. I just wanted to chime in and say how great it'd be to see this feature get implemented.

[EtienneCmb]

Hi @dougollerenshaw

I'll take a look at it next week. I'll post updates on this thread as soon as possible.

Thanks for your comments !

[paulbrodersen]

Hi @EtienneCmb and @raphaelvallat, my work schedule is a bit less crazy the next couple of weeks so I am planning to fork out a day a week to work on this. I think it would be good if we could get together (virtually) and make a list of architecture changes that is as concrete as possible (i.e. function get_foo needs new argument baz and the return type needs to be changes to bar). If we have a concrete framework with defined input/output types for all functions that need changing then I (and anybody who wants to join in) can get cracking without botching up your code architecture too much. Also this would help me to get an overview of where things are in the code base, and how the parts play together so I don't make any glaring mistakes. I would be available for a skype call (or similar) any time in this and the next week apart from tomorrow evening and both Wednesdays.

[TomBugnon]

Hi all,
I work with @grahamfindlay and will spend some time to try and implement the minimal updates (and possibly more) to make non-human scoring possible in Sleep. I will implement something no matter what because our lab needs it at that point but, naturally, it would be better if this was done in such a way that the updates can be merged to the main branch rather than remain funky code that lives in an obscure fork that only our lab uses, so I suppose it's time to reactivate this thread and discuss how precisely these things could be done properly. @paulbrodersen, @EtienneCmb @raphaelvallat, have you by chance implemented anything yet or thought of what the framework could look like?

To frame the discussion a bit, here is a first batch of ideas about what would/could be needed, and proposals on what it could look like, hopefully without messing to much with the current architecture.

1. Decouple the "scoring window" from the display window and display stages around the current scoring window. : Currently, when scoring, the hypnogram value is changed for all the samples within the displayed window. As Graham mentioned above, we'd need to be able to score only part of that window, and display the stages around the segment of interest as in SleepSign.

   1. Represent the "scoring window": We could simply add a new "scoring window" (_ScorWin ?) UI parameter analogous to the "Window" parameter (_SigWin). The displayed window is currently represented implicitly from the current step index, the step size and the window size. Similarly we could represent the "scoring window" implicitly from the (scoring) window size and the step and (display) window parameters by always keeping the scoring window centered relative to the display window. _ScorWinwould be by default equal to _SigWin (for compatibility with human scoring) and bound to be smaller than _SigWin.
   2. Display the scoring window: As in Sleepsign, we could add two vertical bars on each of the ChannelPlot canvas, that would look similar to the "grid" lines. I'm not exactly sure how to best do that, though. Maybe we can use the Indicator class to draw the lines onto the ChannelPlot canvas, and add a ChannelPlot.redraw_scoring_indicators method that would be called similarly to ChannelPLot.set_data when moving the slider, and would slide the two indicator bars so that they are always centered within the displayed window. Sounds like it should work.
   3. Score within the "scoring window" and slide to the next scoring window: When hitting one of the shortcuts, we could change the hypnogram values for the samples within this "scoring window" rather than within the full "display window", and move the slider to the next step. If the slider step is by default equal to or smaller than the 'scoring window' parameter, we won't miss data. Note that if we fixate the _ScorWin and step size parameters to identical values, that enforces a scoring by fixed epochs, and otherwise the user can score arbitrarily precisely by using small step sizes and/or scoring sizes.
   4. Display the hypnogram around the "scoring window" of interest: Lastly, we need to somehow show the stages for the data that is within the displayed window, but outside of the scoring window. The solution in SleepSign is to split the signal in epochs, draw the corresponding cases and display the stage overlayed onto the signal for each of the epochs. The underlying hypnogram in Sleep is encoded on a sample-per-sample basis so it's a bit unnatural to enforce and display epochs. An alternative -consistent with the flexible sample-per-sample scoring- could be to draw upon the already existing code and create a second "hypnogram" visual object that would display the hypnogram within the currently displayed window (rather than for the whole recording), and could be displayed (or not) in the main window, say, in between the channel plots and the spectrogram ? It might make the GUI a bit crowded but, well, it's possible to hide the overall histogram, and a double hypnogram shouldn't be too bad on big screens.

This is the most urgent aspect as far as we are concerned (since we can hack around each of the following points) so I'd work on this first. There's probably a whole lot of subtleties here that I'm not envisioning (yet?) so feedback is most welcome :)

2. Allow for modular stages and properties: I haven't thought too much about this but as Etienne hinted it seems like the stages and properties are hardcoded a bit everywhere. I'm not fully clear on the whole architecture but it seems like it's going to be a matter of passing a stage_config parameter all the way down from the Sleep object to the Hypnogram and CanvasShortcuts objects at initialization, and refactoring things a bit everywhere?

3. Support for data formats used in animal recording: If Sleep extends to animal data scoring it would be nice be able to load formats such as flat binary or NWB, and possibly in the longer term to support loading of (very) large datasets such as Neuropixel data. Loading extra data formats is just a matter of adding a couple functions to read_sleep.py. Even without this, the fact that the Sleep objects take data arrays and channel files as arguments makes it possible to preload any type of data. For big (neuropixel) datasets, however, it's impossible to load the whole recording so the user would need to identify, load and score only a few channels of interest. I imagine this can be a bit cumbersome and lack flexibility of showing and hiding channels online directly in the GUI. Ideally, Sleep would at some point be able to take advantage of the flat binary format by creating a memory map and loading data online, only for the displayed channels and time window. Is this something that you have ever been considering ? How much of an impossible can of worms do you thing that would be?

I'd be very happy to discuss and get feedback on this before I get my hands too dirty! @EtienneCmb , I am currently in Paris, maybe we could have a chat on the phone and why not possibly meet at some point ?
Cheers,
Tom

[EtienneCmb]

@TomBugnon thanks for your feedback, clearly explains ! This represents quite a lot of work, and to be honest with you, I'll have no time to contribute to the development of animal support (also, I've never worked with animal data !). I think the easiest thing for now would be that you fork visbrain, make your changes and once it's working ping me and @raphaelvallat so that we can see your modifications.

About file support, we tried to keep the number of dependencies as low as possible, also because visbrain contains other modules than Sleep.

Cheers from marseille,

[TomBugnon]

Hi @EtienneCmb @raphaelvallat @grahamfindlay and all,

I implemented the decoupling between scoring window and display window, and the "window hypnogram" that I outlined above in this fork: https://github.com/TomBugnon/visbrain/commits/scoring_window

On startup, the GUI is in "Locked" mode, meaning the scoring window and the slider stepsize are equal to, and follow the display window, so user experience remains the same.

When changing the "scoring window" size, we "unlock" the scoring window from the display window. The indicator bars for the scoring window are now visible, and the slider stepsize now follows the scoring window.

The user can now score in shorter slices while seeing the peripheral data, and also see the stages of the window data by displaying the "window hypnogram"

Re-locking the scoring window onto the display window hides the "window hypnogram" and the scoring window indicator bars, and sets the scoring window and stepsize back to the display window value as in the original state.

Note that the staging of the very start and end of the recording is not possible (or a bit funky) but I suppose that's not too big a deal for now.

Let me know if you have specific suggestions about the coding or interface and I can incorporate them, otherwise I'll send a pull request?
Cheers, Tom

[raphaelvallat]

Wow, impressive! This looks great @TomBugnon! I'll have to try it with some of my own data, but it seems like a great addition to Visbrain. Have you updated the documentation and the tests as well? Please feel free to submit a PR so that @EtienneCmb and I can have a deeper look at the changes.
Thanks!

[TomBugnon]

Hi @raphaelvallat and @EtienneCmb, I updated the documentation and tests and submitted the pull request. It would be great if you could have a look at it, we are starting to actually use this updated version to sleep score real data in our lab and it would make things easier if these changes were merged. I'll be happy to make some changes to the PR as you see fit.

[TomBugnon]

Scoring window is done 🍾
next step is arbitrary vigilance states