using scp.read_opus(name) in Jupyter Notebook gets stuck

[3mrrrx]

What happened?

When using scp.read_opus(name) when reading a number of OPUS files in Jupyter Notebook the cell finishes but the kernel is then stuck.
The issue happens randomly at different files for same file sets.
Running the same code out side a Jupyter Notebook Env. (e.g. Console) works well!

What did you expect to happen?

No response

Minimal Complete Verifiable Example

data_list = []
for i, file in enumerate(glob.glob(path)): 

    data_list.append(scp.read_opus(name))

Relevant log output

No response

Anything else we need to know?

The notebook cell does not give any errors. It is just stuck

Environment

INSTALLED VERSIONS

SpectroChemPy's API - v.0.6.7

commit: None
python: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:40:31) [MSC v.1929 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 154 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: ('en_US', 'ISO8859-1')

[fernandezc]

Hi @3mrrrx

I have tried to reproduce your issue:

import spectrochempy as scp
from spectrochempy import preferences as prefs
import glob

DATADIR = prefs.datadir
OPUSDATA = DATADIR / "irdata" / "OPUS"

path = OPUSDATA
data_list = []
for i, file in enumerate(glob.glob(str(path)+"/*")): 
    if "background" in file:
        continue # remove the background.0 file that can not been interpreted for now.
    data_list.append(scp.read_opus(file))
data_list

and got this result (without no stuck in Jupyter lab).

[NDDataset: [float64] a.u. (shape: (y:1, x:2567)),
 NDDataset: [float64] a.u. (shape: (y:1, x:2567)),
 NDDataset: [float64] a.u. (shape: (y:1, x:2567)),
 NDDataset: [float64] a.u. (shape: (y:1, x:2567))]

Can you tell us more about the file your are reading. May be a problem with memory if they are very large?

[3mrrrx]

Thank ypu fpr the quick reply and the code example.

In tje example you are removing the background measurement. I didn't do this step in my code. I will try and see if this helps.

The files have a normal size resulting from the default settings of the ir sensor. Nevertheless i will try and monitor the memory while load the files.

I both cases i will be happy to report back on this. I could also try gave an example (sharing also the opus files) of how to produce the error.

First thing for Monday

Best regards and i am very happy to use scp

[3mrrrx]

Hallo,

so far i have tried to recreate and narrow down the issue, and could exclude that it is not memory related.

weirdly the issue also does not happen at all, in which case this message is (repeatedly but different comm number) printed out in the console:

No such comm: bd2e3fe89c4746b0962297872cb4fd19

what is this message and why is it printed?

currently I am assuming that this issue might be related to a corrupted file or access issue (multiple programs accessing files).

i will be spending some time working with the code and dataset this week, will try to further narrow down the issue, and create a dummy dataset that i could share

[3mrrrx]

Apprently the juypter notebook gets stuck when reading the file one-by-one, reading the same files in bulk e.g. as folder does not result in this isssue!

[fernandezc]

No such comm: bd2e3fe89c4746b0962297872cb4fd19

what is this message and why is it printed?

Making a rapid google search, I found that may be it is related to Jupyter, not Spectrochempy:
for example jupyter-widgets/ipywidgets#3454, although it seems a different context.

Anyway it is difficult to help you as I can't reproduce this problem with my files.
May be share the files, and a the minimum print out the complete Jupyter logs when the error happens. If it is Jupyter related may be write an issue for Jupiter notebook.lab developper.

[3mrrrx]

thank you for your support so far.

i am also coming to the conclusion that this is related more to Jupyter Notebooks.

still based on some discussions i was having with some colleges there are two notes worth mentioning:

• the OPUS-Files that i am read include background measurements within them. Based on the code snippet example, that you gave, the background measurements seam to in be in an separate file.

• The read-opus function (mostly) works when only one or two files are read (sometimes more). Yet in these cases the functions return a very large repetition of the log/print -> "No such comm: ****"; this might be causing an overflow that is breaking the Notebooks.

[fernandezc]

• the OPUS-Files that i am read include background measurements within them. Based on the code snippet example, that you gave, the background measurements seam to in be in an separate file.

I don't really know about OPUS, as I've never used it. @atravert since you wrote this code, can you comment on it?

[atravert]

Hello @3mrrrx ,

Can you post some of these OPUS files ? I will have a look at them and if possible adapt the reader to avoid these problems in jupyter.

Thanks,
Arnaud

[3mrrrx]

Hello, and thank you for the replay!

I would be happy to send you these for testing. currently, I am not sure I can publish these publicly yet!

[atravert]

Hello @3mrrrx ,

I have looked at your issue and could reproduce it with jupyter notebook version 6.5.3, and it does not appear for notebook>= 6.5.5. The bug is actually not related to spectrochempy but to jupyter, and described here: jupyter/notebook#6721

Can you check, on your side, which notebook version you used ?

$ conda list notebook

You can fix it by upgrading jupyter notebook to v.6.5.5

$ mamba install notebook=6.5.5

or

$ conda install notebook=6.5.5

Or, instead of jupyter notebook, you can use Jupyter Lab, which we recommend for spectrochempy (https://www.spectrochempy.fr/latest/userguide/introduction/interface.html#Create-a-new-Jupyter-notebook).

I hope this helps. Of course let me know if this does not solve your problem ! My apologies for the delay....

Best regards,

Arnaud