FileNotFoundErrors: debug_resampled.tif & annotation_25_1-246Ld.nrrd

[saskra]

Since my server can't calculate a whole brain in between, I wanted to use only a small part of your data to test the TubeMap script. I took the autofluorescence from the first brain 39L completely and from the raw data only four consecutive files from the middle (Y02, X05-06, Filter0000-0001).

  #directories and files
  directory = '/home/saskra/PycharmProjects/ClearMap2/ClearMap/Tests/Data/TubeMap_Example'
  expression_raw      = 'Raw/14-16-41_tricocktail_UltraII[<Y,2> x <X,2>]_C00_UltraII Filter0001.ome.tif'
  expression_arteries = 'Raw/14-16-41_tricocktail_UltraII[<Y,2> x <X,2>]_C00_UltraII Filter0000.ome.tif'
  expression_auto     = 'Autofluorescence/14-02-13_auto_UltraII_C00_xyz-Table Z<Z,4>.ome.tif'

But I get this error message:

Graph reduction: initialized.
Graph reduction: Found 20393 branching and 592581 non-branching nodes: elapsed time: 0:00:00.591
Graph reduction: Scanned 20393/20393 branching nodes, found 1018 branches: elapsed time: 0:00:01.420
Graph reduction: Scanned 250000/592581 non-branching nodes found 11667 branches: elapsed time: 0:00:09.205
Graph reduction: Scanned 500000/592581 non-branching nodes found 22705 branches: elapsed time: 0:00:16.715
Graph reduction: Scanned 592581/592581 non-branching nodes found 26871 branches: elapsed time: 0:00:19.219
Graph reduction: Graph reduced from 612974 to 20393 nodes and 619452 to 26871 edges: elapsed time: 0:00:20.528
Transforming vertex property: coordinates -> coordinates_atlas
Traceback (most recent call last):
  File "/snap/pycharm-professional/209/plugins/python/helpers/pydev/pydevd.py", line 1438, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-professional/209/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 482, in <module>
    verbose=True);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Analysis/Graphs/GraphGt.py", line 1366, in transform_properties
    values = transformation(values);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 465, in transformation
    sink_shape=io.shape(ws.filename('resampled')));
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/IO.py", line 258, in shape
    source = as_source(source);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/IO.py", line 206, in as_source
    source =  mod.Source(source, *args, **kwargs);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/TIF.py", line 36, in __init__
    self._tif = tif.TiffFile(location, multifile = multi_file);
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 2183, in __init__
    fh = FileHandle(arg, mode='rb', name=name, offset=offset, size=size)
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 6408, in __init__
    self.open()
  File "/home/saskra/anaconda3/envs/ClearMap/lib/python3.7/site-packages/tifffile/tifffile.py", line 6421, in open
    self._fh = open(self._file, self._mode)
FileNotFoundError: [Errno 2] Datei oder Verzeichnis nicht gefunden: '/home/saskra/PycharmProjects/ClearMap2/ClearMap/Tests/Data/TubeMap_Example/debug_resampled.tif'

It is true there is now file called "debug_resampled.tif", but I do not know where it should have come from and how to get it.

[SaibotMagd]

I just saw this strange problem in the cellmap pipeline. When you use the command ws.debug = True, all future ws.filename are augmented by an "debug_"+X. But you didn't calc the resample.tif or resample_autofluroescence.tif or all the alignment parameters using the "debug_" filenames. On the other hand you need the whole parameter set to do a proper alignment.

I solved this issue this way:

• use the whole dataset for all the alignment, stitching parts
• create debug subset (...slicing = (slice(...) ... ws.create_debug('stitched', slicing=slicing))
• rename subset "debug_stitched" to only "stitched.npy" (don't overwrite the full stitched.npy; i saved it as old_stitched.npy)
• deactivate debug modus by calling ws.debug = False
  ...and go on with the pipeline

pretty annoying but possible.

[saskra]

Tanks a lot! Still, now I get a new error message:

transformix has finished at Wed Aug  5 15:24:17 2020.
Total time elapsed: 12.0s.
Traceback (most recent call last):
  File "/snap/pycharm-professional/211/plugins/python/helpers/pydev/pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-professional/211/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 507, in <module>
    edge_geometry_properties = {'coordinates_atlas' : 'annotation'});
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Analysis/Graphs/GraphGt.py", line 1413, in annotate_properties
    edge_geometry_properties=edge_geometry_properties);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Analysis/Graphs/GraphGt.py", line 1366, in transform_properties
    values = transformation(values);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Scripts/TubeMap.py", line 502, in annotation
    label = ano.label_points(coordinates, key='order');
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/Alignment/Annotation.py", line 445, in label_points
    atlas = io.read(annotation_file);   
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/IO.py", line 409, in read
    return mod.read(source, *args, **kwargs);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/NRRD.py", line 297, in read
    return source.array
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/NRRD.py", line 70, in array
    return _array(self.location);
  File "/home/saskra/PycharmProjects/ClearMap2/ClearMap/IO/NRRD.py", line 600, in _array
    with open(filename,'rb') as filehandle:
FileNotFoundError: [Errno 2] Datei oder Verzeichnis nicht gefunden: '/home/nicolas.renier/Documents/ClearMap_Ressources/annotation_25_1-246Ld.nrrd'
python-BaseException 

While it is obvious that this hard-coded path does not exist on my machine, I do not know with which one to replace. Is this a file I should have downloaded or created?

[SaibotMagd]

It's always better to open a new issue than to modify an old one, because your solution can help others so they can find it.

@question:
I don't work on the tubemap pipeline, but I found a zip archive containing additional files: https://idisco.info/clearmap-2/ ... the File you need should be: "annotation_25_full_nrrd" and then you have to cut it like you did with the reference file.

It's just a guess I didn't try this at all.

[saskra]

I found a zip archive containing additional files: https://idisco.info/clearmap-2/ ... the File you need should be: "annotation_25_full_nrrd" and then you have to cut it like you did with the reference file.

Thanks again, I was not aware of this additional source of files! Unfortunately, this does not look like the brain I have downloaded from here: https://osf.io/sa3x8/files/?view_only=4427a838cbd0468c9fbad9cab465d866

[SaibotMagd]

Thanks again, I was not aware of this additional source of files! Unfortunately, this does not look like the brain I have downloaded from here: https://osf.io/sa3x8/files/?view_only=4427a838cbd0468c9fbad9cab465d866

I don't get it. The annotation file is a part of the atlas (it's based on MRI). The osf.io files are actually LSM Data, so it shouldn't be the same.

[saskra]

Then I do not really understand what this file is, why I need it and how those two should go together - how shall I crop the MRI like the LSM when they have completely different resolution, orientation etc.? And I could not use this pipeline on our own data, if I needed MRI images of the same (?) brain as well. Or am I getting something completely wrong here?

[ChristophKirst]

The reference file is needed if you want to align the data to an atlas (in this case the Allan btrain Atlas).
If you only like to process your own data without atlas annotation you can skip all the steps that involved this file and the alignment.

If you work on graphs you should also skip the steps that transform the coordinates to the refernece etc.

Does this make sense?

In general, I see that some of the questions here arise by assuming the pipeline will work out of the box.
In fact, we provide the most complete pipeline that needs to be modified for your specific purposes, including removing certain steps that are not needed.

If you working on small data sets that are part of a larger one and like to use the atlas annotation, the pipeline also might needs tweaking to make this work with alignment of the full data.

Hope this helps ?

Feel free to reopen if this does not solve the issue