Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ExportToDatabase properties file creation fails for MySQL #3714

Open
bethac07 opened this issue Jan 29, 2019 · 6 comments

Comments

Projects
3 participants
@bethac07
Copy link
Member

commented Jan 29, 2019

From the forum:

I recently set up a test MySQL database (v5.6.41) on AWS so that in the future I can use multiple copies of CellProfiler on our cluster. I performed a run of 308 images, and when I check the database via MySQL workbench it looks fine. However, right after finishing processing the last image in this run, I receive this error:

Traceback (most recent call last):
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/pipeline.py”, line 2194, in post_run
module.post_run(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 2376, in post_run
self.write_properties_file(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 3635, in write_properties_file
all_properties = self.get_property_file_text(workspace)
File “/home/nel56/.conda/envs/cellprofiler/lib/python2.7/site-packages/cellprofiler/modules/exporttodatabase.py”, line 4081, in get_property_file_text
“”" % (locals())
UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xff in position 254: ordinal not in range(128)

User confirms it works fine if he tries to make an SQLite, so it must be something about encoding the MySQL host into the properties file I'm guessing.

@edlaw89

This comment has been minimized.

Copy link

commented Mar 21, 2019

Hi,

Has there been any development with this? I have since set up my own MySQL server, and tried to run my pipeline from my own computer. Now the error appears before the pipeline runs or exports any data at all.

I really need this to work soon, because I can't run any analysis.

Cheers,

Ed

@edlaw89

This comment has been minimized.

Copy link

commented Mar 22, 2019

I have got it to export something in CellProfiler 2.2 it seems, however not CP 3.1.8. If this could be fixed for CP 3.1.8 that would be great as I need the improved functionality.

Cheers,

Ed

@edlaw89

This comment has been minimized.

Copy link

commented Mar 25, 2019

I have now tested it in several versions: it works fine in 2.2 and 3.0.0, but not 3.1.5 or 3.1.8, so the error must be creeping in at that point.

Ed

@edlaw89

This comment has been minimized.

Copy link

commented Apr 24, 2019

I'm still running into this issue. Even in a cluster environment I get the same error. Has there been any update on this?

@kmckusick-Vir

This comment has been minimized.

Copy link

commented Jul 11, 2019

I have run into this exact issue with exact error. Mysql database works fine in 2.x but not 3.1.5 nor 3.1.8. I have tried from both Windows and Linux cluster and it is reproducible. Based on details posted here I will back down to 2.2 or 3.0.0 but I don't like that solution too much and look forward to a solution that lets me run the latest code. Thanks!

@kmckusick-Vir

This comment has been minimized.

Copy link

commented Jul 16, 2019

I have found the root cause and developed a workaround.
In v3.1.x the cppipe file contains a bunch of unicode characters where in v2.x it was straight ascii. Some unicodes are not handled properly and cause a problem in exporttodatabase.py.

Specifically, in exporttodatabase.py I removed %(db_info)s from the long string that sets the contents string for the CellProfiler Analyst 2.0 properties file. (I do not need that file and could fill in that info by hand if I did need it).

Since this update to exporttodatabase.py works for my purpose I did not dig further into where the db_info unicode data is being mishandled. But I did notice that the unicode is ending up in the db_info definition in the (locals()) variable, while unicode present in the cppipe is stripped in locals() for the other data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.