Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Difficulty running pipeline using command line interface #3554

Open
caticoa3 opened this issue Apr 26, 2018 · 8 comments

Comments

@caticoa3
Copy link

@caticoa3 caticoa3 commented Apr 26, 2018

Hi,

First off thank you for the time and effort that goes into CellProfiler.

I'm trying to run a pipeline which I can successfully run using the GUI. I exported the image list (exported_image_list.csv) and pipeline (IC_pipeline_DAPI_GFAP_1.cppipe).
Now I would like to process many more images and try to integrate with Python and command line. I tried running the pipeline from the command line but didn't have much luck:

$ cellprofiler -p IllumCorr/IC_pipeline_DAPI_GFAP_1.cppipe --data-file=exported_image_list.csv -r

error: CP-JAVA 00:07:15.006 [Thread-0] WARN o.c.imageset.ChannelFilter - Empty image set list: no images passed the filtering criteria.

The error that appears in the GUI after executing the command:
"Cannot run pipeline
The pipeline cannot be started because of
a configuration problem:
The pipeline did not identify any image sets.
Please correct any problems in your input module settings
and try again."

Any suggestions or help would be appreciated

Best

PS - Im working on Ubuntu16.04 and installed cellprofiler using conda as described in the wiki.

@caticoa3 caticoa3 changed the title Having difficulty running using command line interface Difficulty running pipeline using command line interface Apr 26, 2018
@AetherUnbound

This comment has been minimized.

Copy link
Collaborator

@AetherUnbound AetherUnbound commented Apr 28, 2018

Hello @caticoa3! Do you mind trying the --file-list flag rather than the --data-file flag? I believe that --data-file is for use in the LoadData module only.

@AetherUnbound

This comment has been minimized.

Copy link
Collaborator

@AetherUnbound AetherUnbound commented Apr 28, 2018

Additionally, can you post the pipeline and the file list/files you're using?

@caticoa3

This comment has been minimized.

Copy link
Author

@caticoa3 caticoa3 commented Jun 21, 2018

Hi AetherUnbound,
Thanks for the reply.
I have since modified my pipeline. However, I managed to reproduce the error with the new pipeline. When I run from the command line the GUI opens but images are not processed.
I tried using both flags: --file-list and --data-file.

When I use the --file-list flag:
$ cellprofiler -p segmentation.cppipe --file-list=exported_image_list_1.csv -r
the error is "Encountered unrecoverable error in Images during startup:
URI is not hierarchical."

When using the --data-file flag:
$ cellprofiler -p segmentation.cppipe --data-file=exported_image_list_1.csv -r
the error is
"Cannot Run Pipeline
The pipeline cannot be started because of a configuration problem: The pipeline did not identify any image sets. Please correct any problems in your input module settings
and try again."

When I used the --file-list flag I see images listed in the Images Module. However, when I used --data-file I do not see any images listed in the images module, which agrees with your recommendation to use --file-list. So that solves one error :).

I've attached my file list and pipeline.
files#3554.zip

@AetherUnbound

This comment has been minimized.

Copy link
Collaborator

@AetherUnbound AetherUnbound commented Jun 23, 2018

@caticoa3 Ha! I'm glad that --file-list (sort of) worked! We'll try to improve our documentation there, I got confused myself :)
But just to be clear, the --file-list command still does not run your pipeline?

@caticoa3

This comment has been minimized.

Copy link
Author

@caticoa3 caticoa3 commented Jun 24, 2018

That is correct. When I used the --file-list command the pipeline did not run.

@caticoa3

This comment has been minimized.

Copy link
Author

@caticoa3 caticoa3 commented Jun 26, 2018

Good news. I had more success creating a batch, .h5, file and running headless from command line:

As described in the manual I added the CreateBatchFile module to the end of my pipeline, created the .h5 batch file, and ran Cell Profiler locally on my computer as indicated in step 7 - even though I am working locally and not submitting to a remote cluster.

$ cellprofiler -p "location of Batch_data.h5" -c -r -f "image set number to start processing"

Now, the problem I am running into is that cell profiler stops processing when it attempts to save a blank image.
"Error detected during run of module SaveImages...
...ValueError: zero-size array to reduction operation minimum which has no identity"

I searched for flags in the cellprofiler --help but none are available for ignoring save errors. Is there a way to skip to the next image and continue processing?
When this error occurs in the GUI the error can be canceled to continue processing.

@caticoa3

This comment has been minimized.

Copy link
Author

@caticoa3 caticoa3 commented Jun 27, 2018

To continue processing I wrote a while loop in a bash script. I scraped together a regex expression to parse the Image Set when the error occurs and have CellProfiler restart at the next image. I'm sure this can be done more elegantly. Hope this will help others:

`#!/bin/bash
n=1 #Image Set number to start processing batch
final_image=826 #Last Image Set number in your batch
cellprofiler -p ML/Batch_data.h5 -c -r -f $n -l $final_image |& tee log.txt #Here the output from terminal is save to log.txt.

while [ $? -ne 1 ]; do
#Parse log.txt file, extracting the image set # at which the error occurs
no_save_error_image=$(sed -n '/^.*ValueError:/{n;p}' log.txt | sed -n -e 's/^.Image # //p' | sed -n -e 's/, module SaveImages.$//p')
n_=$(($no_save_error_image + 1))
#Restart CellProfiler at the next image set
echo 'Restarting at Image set #' $n_
cellprofiler -p ML/Batch_data.h5 -c -r -f $n_ -l $final_image |& tee log.txt
#Once the final image set is reached stop the script
if ((n_==$final_image)); then
echo 'The last image set #:' $n_', has been processed. All images have been processed. The script will now exit.'
break
fi
done
`

@AetherUnbound

This comment has been minimized.

Copy link
Collaborator

@AetherUnbound AetherUnbound commented Jul 4, 2018

Wow @caticoa3, way to be resourceful! I'm sorry that this has continued to cause you trouble. I'm going to try and take some time to look into it, although this does look like two separate issues. Thanks for bringing your solution back here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.