Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try to reproduce Well reordering issue with Fileset swapping #52

Open
will-moore opened this issue Apr 24, 2023 · 6 comments
Open

Try to reproduce Well reordering issue with Fileset swapping #52

will-moore opened this issue Apr 24, 2023 · 6 comments

Comments

@will-moore
Copy link
Member

See e.g. IDR/idr-metadata#652 (comment)
The reordering of Wells seen there is hard to work with since it needs .pattern files, which can't be easily imported alongside NGFF plates, since we don't have a server that can read both.

Try to test and use other HCS formats to reproduce the issue...
Use small mias plate to test... - unfortunately, the converted NGFF plate failed to import...

$ rsync -rvP --progress idr0-slot3.openmicroscopy.org:/uod/idr/repos/curated/mias/frans/siRNA_PRIM1_03102008 .
$ bioformats2raw --version
Version = 0.4.0
Bio-Formats version = 6.8.0
$ bioformats2raw siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt plate.zarr
$ ls plate.zarr/
0	OME

Import original and NGFF plates... into merge-ci...

$ omero import siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt
...
2023-04-24 14:16:42,726 90075      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /Users/wmoore/Desktop/images/mias/siRNA_PRIM1_03102008/001-365700055641/Well0013/mode1_z000_t000_im1_1.tif
Plate:68655
Other imported objects:
Fileset:140757

$ omero import plate.zarr/OME/METADATA.ome.xml --depth=100
Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-04-24 14:19:06,331 467        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.12
2023-04-24 14:19:06,363 499        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0 revision: f8b46c2458c43cffdf5bc67cc4bf9dfc6e93167b date: 31 May 2022
2023-04-24 14:19:06,479 615        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-04-24 14:19:07,109 1245       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2023-04-24 14:20:01,084 55220      [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /Users/wmoore/Desktop/images/mias/plate.zarr/OME/METADATA.ome.xml
java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Integer (java.lang.String and java.lang.Integer are in module java.base of loader 'bootstrap')
	at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:539)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:287)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:662)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
	at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
	at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:997)
2023-04-24 14:20:01,092 55228      [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 53966ms. (53983ms total) [0 unknowns]
2023-04-24 14:20:01,182 55318      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2023-04-24 14:20:01,851 55987      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-04-24 14:20:02,225 56361      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.5.12
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.14.1
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2023-04-24 14:20:02,262 56398      [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 12.2.1
No imports due to errors!

Also tried removing /OME dir and...

omero import plate.zarr/.zattrs --depth=100
No imports found
@sbesson
Copy link
Member

sbesson commented Apr 24, 2023

Based on the stack trace, I suspect the issue is related to the acquisition.id metadata - which has been clarified in ome/ngff#120, implemented in glencoesoftware/bioformats2raw#152 and released in bioformats2raw 0.5.0. It would be useful to test whether the same plate converted with a recent version of bioformats2raw displays the same issue

@will-moore
Copy link
Member Author 

$ conda update -c ome bioformats2raw
$ bioformats2raw --version
Version = 0.6.1
Bio-Formats version = 6.12.0
NGFF specification version = 0.4

@will-moore
Copy link
Member Author

Want to try and setup a new env on idr0125-pilot for doing the conversion and import to that server, which has recent bf2raw fixes etc.

Created new env on idr0125-pilot, install bioformats2raw with $ conda install -c ome bioformats2raw :

$ bioformats2raw --version
Version = 0.6.1
Bio-Formats version = 6.12.0
NGFF specification version = 0.4

But can't get the data there...

(bioformats2raw) [wmoore@pilot-idr0125-omeroreadwrite mias]$ rsync -rvP --progress idr0-slot3.openmicroscopy.org:/uod/idr/repos/curated/mias/frans/siRNA_PRIM1_03102008 .
ssh: Could not resolve hostname idr0-slot3.openmicroscopy.org: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]

Since I have a local copy, upload to idr-pilot...

$ rsync -rvP --progress siRNA_PRIM1_03102008 idr-pilot.openmicroscopy.org:/home/wmoore/

Then copy from there to idr0125-pilot:

$ ssh -A idr-pilot.openmicroscopy.org
$ rsync -rvP --progress siRNA_PRIM1_03102008 idr0125-omeroreadwrite:/home/wmoore/

$ ssh -A -o 'ProxyCommand ssh idr-pilot.openmicroscopy.org -W %h:%p' idr0125-omeroreadwrite -L 1080:localhost:80
$ mv siRNA_PRIM1_03102008 test_data/mias/
$ conda activate bioformats2raw
$ bioformats2raw siRNA_PRIM1_03102008/001-365700055641/results/NEOlog2008-09-18-14h37m07s.txt plate.zarr

cd ../
mv mias/ /ngff/
cd /ngff
sudo chown -R omero-server mias

$ sudo chown -R omero-server
$ source /opt/omero/server/venv3/bin/activate

$ omero import --transfer=ln_s --depth=100 siRNA_PRIM1_03102008/
Plate:10322
Other imported objects:
Fileset:5286923
==> Summary
139 files uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:01:31.977

(venv3) (venv3) (base) [omero-server@pilot-idr0125-omeroreadwrite mias]$ omero import --transfer=ln_s --depth=100 plate.zarr
Plate:10323
Other imported objects:
Fileset:5286924
==> Summary
358 files uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:06:17.050

@will-moore
Copy link
Member Author 

$ python swap_filesets.py Plate:10322 Plate:10323
UPDATE pixels SET name = '.zattrs', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-04/24/16-49-56.080/plate.zarr' where image in (select id from Image where fileset = 5286924);

$ source /opt/omero/server/venv3/bin/activate
$ export OMERODIR=/opt/omero/server/OMERO.server
$ omero config get --show-password
$ PGPASSWORD=****** psql -U omero -d idr -h 192.168.10.102
UPDATE pixels SET name = '.zattrs', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-12/2023-04/24/16-49-56.080/plate.zarr' where image in (select id from Image where fileset = 5286924);
UPDATE 16

Original Data:
Screenshot 2023-04-24 at 18 23 37

NGFF Plate:
Screenshot 2023-04-24 at 18 23 46

Original Plate after NGFF Fileswap and regenerate thumbnails (Save to All in webclient):
Screenshot 2023-04-24 at 18 27 32

@will-moore
Copy link
Member Author 

sudo -u omero-server -s
cd
wget https://github.com/glencoesoftware/raw2ometiff/releases/download/v0.4.1/raw2ometiff-0.4.1.zip
unzip raw2ometiff-0.4.1.zip

cd /ngff/mias/
/opt/omero/server/raw2ometiff-0.4.1/bin/raw2ometiff plate.zarr plate.ome.tiff

source /opt/omero/server/venv3/bin/activate
export OMERODIR=/opt/omero/server/OMERO.server
omero import --transfer=ln_s  plate.ome.tiff 

Plate:10324
Other imported objects:
Fileset:5286925

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 16 images imported, 0 errors in 0:03:58.218

Set the original imported Plate to use the ome.tiff Fileset...

python swap_filesets.py Plate:10322 Plate:10324

// psql - NB: needed to use plate.ome.tiff
idr=> UPDATE pixels SET name = 'plate.ome.tiff', path = 'demo_2/Blitz-0-Ice.ThreadPool.Server-14/2023-04/24/19-12-28.091/' where image in (select id from Image where fileset = 5286925);
UPDATE 16

This fixes the Wells ordering in the Original imported Plate, now using ome.tiff Fileset:

Screenshot 2023-04-24 at 20 32 14

@sbesson
Copy link
Member

sbesson commented Apr 24, 2023 •
edited

Thanks @will-moore. This seems to indicate 2 things:

  • the fileset swapping strategy could be used immediately for datasets converted to OME-TIFF
  • the issue with the wells ordering for NGFF datasets is likely specific to ZarrReader

@dgault dgault mentioned this issue May 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@will-moore @sbesson