Unicode scripts 12571 #3833

will-moore · 2015-05-20T10:07:51Z

This handles unicode script param values when running scripts.

To test, need to check that all parameters passed to scripts can handle unicode strings and different scripts handle these values correctly.
E.g String parameters, Lists with unicode values and Maps with unicode values.

Set up:

Rename a Dataset, Image and Channels with strings containing unicode characters. Also Tag the image with a unicode tag.

Scripts:

Split view Figure, with unicode in channels and figure name.
Thumbnail Figure, sorting by Tag with unicode.
Combine Images, naming new channels (map) with unicode names.

Corresponding scripts PR: ome/omero-scripts#101 has unicode handling to Split_View_Figure and Thumbnail_Figure as well as other small fixes.

joshmoore · 2015-05-21T07:10:18Z

./components/tools/OmeroWeb/omeroweb/webclient/views.py:3929:80: E501 line too long (81 > 79 characters)

manics · 2015-05-22T10:53:13Z

Dataset To Plate.py:

Traceback (most recent call last):
  File "./script", line 358, in <module>
    runAsScript()
  File "./script", line 348, in runAsScript
    newObj, message = datasets_to_plates(conn, scriptParams)
  File "./script", line 228, in datasets_to_plates
    datasetId, screen)
  File "./script", line 89, in dataset_to_plate
    plate.name = rstring(str(dataset.name))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 13-14: ordinal not in range(128)

manics · 2015-05-22T10:57:54Z

Also noticed the Thumbnail dialog's Choose Tags field has an unescaped javascript problem (user-1, https://trout.openmicroscopy.org/merge/webclient/figure_script/Thumbnail/?Dataset=1956)

joshmoore · 2015-05-25T09:03:27Z

Commits coming with the hope of being in today's rebuild.

will-moore · 2015-05-25T11:10:38Z

Commit ome/omero-scripts@7a0cf40 fixes Dataset_To_Plate issue above at ome/omero-scripts#101.
I'll look at the javascript escaping bug, but might leave that to be fixed later, since it's a long-standing issue not really related to this PR.

manics · 2015-05-25T14:35:59Z

Split view, Thumbnail and combine scripts work.
Dataset to plate still has a problem:

Traceback (most recent call last):
  File "./script", line 358, in <module>
    runAsScript()
  File "./script", line 348, in runAsScript
    newObj, message = datasets_to_plates(conn, scriptParams)
  File "./script", line 228, in datasets_to_plates
    datasetId, screen)
  File "./script", line 130, in dataset_to_plate
    % (image.id, image.name, row, col)
UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f638' in position 51: ordinal not in range(128)

Export as ome-tiff fails with imagename combinedImage £€

{'Export_Individual_Channels': True, 'Choose_Z_Section': 'Default-Z (last-viewed)', 'Choose_T_Section': 'Default-T (last-viewed)', 'Data_Type': 'Image', 'Format': 'OME-TIFF', 'IDs': [23851L], 'Zoom': '100%', 'Export_Merged_Image': True, 'Folder_Name': 'Batch_Image_Export', 'Individual_Channels_Grey': False}
Processing 1 images
Exporting image as OME-TIFF: combinedImage Â£â‚¬
  Saving file as: Batch_Image_Export/combinedImage Â£â‚¬.ome.tif

WARNING:omero.gateway:InternalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <39ea1fa8-ef8d-4efa-88c8-0fc9bd96e2f0omero.api.Exporter> generateTiff((<ServiceOptsDict: {'omero.session.uuid': 'e358ac9d-1565-4cde-a737-9eac568c2ded', 'omero.client.uuid': '0812352a-ac11-4d32-b432-cd955548dd71'}>,), {})
Traceback (most recent call last):
  File "/opt/hudson/workspace/OMERO-5.1-merge-deploy/src/dist/lib/python/omero/gateway/__init__.py", line 4123, in __call__
    return self.f(*args, **kwargs)
  File "/opt/hudson/workspace/OMERO-5.1-merge-deploy/src/dist/lib/python/omero_api_Exporter_ice.py", line 170, in generateTiff
    return _M_omero.api.Exporter._op_generateTiff.invoke(self, ((), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = java.lang.NullPointerException
    at ome.services.blitz.impl.ExporterI.getMetadataBytes(ExporterI.java:576)

joshmoore · 2015-05-25T15:04:40Z

cc: @jburel -- see https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/blitz/src/ome/services/blitz/impl/ExporterI.java#L576 for the export issue.

joshmoore · 2015-05-25T15:06:07Z

NB: If this is intended for 5.1.2, we can't expect to have everything fixed at this point. We will have to either pick our battles, or push for a more extensive review for 5.1.3.

will-moore · 2015-05-25T15:08:15Z

I have a fix for the first issue there. The second is nothing to do with unicode I think. Probably due to exporting an image with channels shared with other images - related to https://trac.openmicroscopy.org.uk/ome/ticket/11532?

will-moore · 2015-05-25T15:10:47Z

will-moore/scripts@fbe427e fixes the Dataset_To_Plate issue above

File "./script", line 130, in dataset_to_plate
    % (image.id, image.name, row, col)
UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f638' in position 51: ordinal not in range(128)

will-moore · 2015-05-25T15:15:37Z

Also found a bug with "Export as TIFF/PNG/JPEG" fixed in the commit above.

joshmoore · 2015-05-26T03:50:51Z

Closing & re-opening to bump travis status since this was marked as "failed" as of yesterday evening, meaning I had to cancel https://ci.openmicroscopy.org/job/OMERO-5.1-merge-daily/483/

gusferguson · 2015-05-26T08:06:25Z

@will-moore

Tested using trout merge user-3 read-only-1 with test string !"#$%&'()* added to names etc.

Channel Offsets.py

runs without error
result opens

Combine Images.py

runs without error
result opens

Dataset To Plate.py

runs without error
Activity window message: "Dataset To PlateNew plate created: !"#$%&'()* Test 15-05-26 08:43:17 but could not be attached."
result opens appears as expected

Images From ROIs.py

runs without error
result as expected

Create Thumbnail Figure

with tags with unicode characters used for sorting
runs without error
result as expected

Create Split View Figure

runs without error
result as expected

No errors in these tests. I think I have covered all the points in thePR but difficult to tell.
Good to merge.

gusferguson · 2015-05-26T08:12:23Z

@will-moore

Sorry missed adding unicode channel name on first test - when tried to use test string as channel name in script window got following error:

Traceback (most recent call last):
  File "./script", line 649, in <module>
    runAsScript()
  File "./script", line 635, in runAsScript
    images, message = combineImages(conn, parameterMap)
  File "./script", line 483, in combineImages
    dataset, colourMap)
  File "./script", line 392, in makeSingleImage
    rawPixelStoreUpload, plane2D, theZ, theC, theT)
  File "/opt/hudson/workspace/OMERO-5.1-merge-deploy/src/dist/lib/python/omero/util/script_utils.py", line 1015, in uploadPlaneByRow
    rawPixelsStore.setRow(convertedRow, y, z, c, t)
  File "/opt/hudson/workspace/OMERO-5.1-merge-deploy/src/dist/lib/python/omero_api_RawPixelsStore_ice.py", line 1381, in setRow
    return _M_omero.api.RawPixelsStore._op_setRow.invoke(self, ((buf, y, z, c, t), _ctx))
omero.ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Y '736' greater than sizeY '736'.
    at ome.services.RawPixelsBean.handleException(RawPixelsBean.java:688)
    at ome.services.RawPixelsBean.setRow(RawPixelsBean.java:629)
    at sun.reflect.GeneratedMethodAccessor1532.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:218)
    at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:202)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
    at com.sun.proxy.$Proxy99.setRow(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor1532.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
    at com.sun.proxy.$Proxy99.setRow(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor1534.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
    at ome.services.throttling.Callback.run(Callback.java:56)
    at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
    at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:149)
    at ome.services.blitz.impl.RawPixelsStoreI.setRow_async(RawPixelsStoreI.java:230)
    at sun.reflect.GeneratedMethodAccessor1533.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at omero.cmd.CallContext.invoke(CallContext.java:78)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
    at com.sun.proxy.$Proxy100.setRow_async(Unknown Source)
    at omero.api._RawPixelsStoreTie.setRow_async(_RawPixelsStoreTie.java:276)
    at omero.api._RawPixelsStoreDisp.___setRow(_RawPixelsStoreDisp.java:1151)
    at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1546)
    at IceInternal.Incoming.invoke(Incoming.java:222)
    at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
    at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
    at Ice.ConnectionI.message(ConnectionI.java:1213)
    at IceInternal.ThreadPool.run(ThreadPool.java:321)
    at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
    at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
    at java.lang.Thread.run(Thread.java:745)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Y '736' greater than sizeY '736'.
}

manics · 2015-05-26T08:26:06Z

@gusferguson You need to test with non-ascii characters (e.g. €, accented characters, arabic/chinese, etc).

gusferguson · 2015-05-26T08:42:45Z

@manics - sorry this is not stated in the PR - it says unicode.

manics · 2015-05-26T09:04:26Z

Looks fine to me, one possible issue (mentioned offline).

joshmoore · 2015-05-26T09:34:59Z

Thanks for all the testing, @gusferguson & @manics. My reading of this is that many of the conditions that were failing are now passing, even if some are still remaining. Since I think it's less likely that a user will have unicode in their channel name, I'd like to go ahead and get this in to help users with other locales, and follow-up in 5.1.3 (see https://trello.com/c/eLd8m6rE/524-script-unicode et al). Likely a thorough testing effort will be needed both in robot and the CLI tests to pick up all of these cases.

Unicode scripts 12571

joshmoore · 2015-05-26T09:35:21Z

NB: the other issue seen was:

 'utf8' codec can't decode bytes in position 15-16: invalid continuation byte.

AND it goes away with omero.web.debug=true

will-moore · 2015-05-26T09:52:57Z

Summary of remaining issues:
As @manics reported and @joshmoore commented: Adding an attachment named 20150514-01 x£€_n§± 😸 さようなら to a dataset gives:

DjangoUnicodeDecodeError: 'utf8' codec can't decode bytes in position 15-16: invalid continuation byte. You passed in '20150514-01 x\xc2\xa3\xe2\x82...\xb8 \xe3\x81\x95\xe3\x82\x88\xe3\x81\x86\xe3\x81\xaa\xe3\x82\x89' (<type 'str'>)

but this is also true on Octopus, so not caused by this PR.

Also, display of unicode Tag in "Batch Annotate" panel gives error.

These issues added to https://trello.com/c/eLd8m6rE/524-script-unicode

Bug from @gusferguson is not unicode related but due to particular images and inputs of Combine_Images script:

serverStackTrace = ome.conditions.ApiUsageException: Y '736' greater than sizeY '736'.
    at ome.services.RawPixelsBean.handleException(RawPixelsBean.java:688)

will-moore added 2 commits May 20, 2015 10:10

Handle unicode for new Tags

01a960c

Running scripts handles unicode values

d8c5a05

will-moore mentioned this pull request May 20, 2015

Unicode support ome/omero-scripts#101

Merged

jburel added the develop label May 21, 2015

Flake8 fix

f00f6c3

Fix name of file for downloading image As jpeg/png/tiff

9030126

joshmoore closed this May 26, 2015

joshmoore reopened this May 26, 2015

joshmoore added a commit that referenced this pull request May 26, 2015

Merge pull request #3833 from will-moore/unicode_scripts_12571

d12d29a

Unicode scripts 12571

joshmoore merged commit d12d29a into ome:develop May 26, 2015

will-moore deleted the unicode_scripts_12571 branch May 26, 2015 09:49

joshmoore added this to the 5.1.2 milestone May 27, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unicode scripts 12571 #3833

Unicode scripts 12571 #3833

will-moore commented May 20, 2015

joshmoore commented May 21, 2015

manics commented May 22, 2015

manics commented May 22, 2015

joshmoore commented May 25, 2015

will-moore commented May 25, 2015

manics commented May 25, 2015

joshmoore commented May 25, 2015

joshmoore commented May 25, 2015

will-moore commented May 25, 2015

will-moore commented May 25, 2015

will-moore commented May 25, 2015

joshmoore commented May 26, 2015

gusferguson commented May 26, 2015

gusferguson commented May 26, 2015

manics commented May 26, 2015

gusferguson commented May 26, 2015

manics commented May 26, 2015

joshmoore commented May 26, 2015

joshmoore commented May 26, 2015

will-moore commented May 26, 2015

Unicode scripts 12571 #3833

Unicode scripts 12571 #3833

Conversation

will-moore commented May 20, 2015

joshmoore commented May 21, 2015

manics commented May 22, 2015

manics commented May 22, 2015

joshmoore commented May 25, 2015

will-moore commented May 25, 2015

manics commented May 25, 2015

joshmoore commented May 25, 2015

joshmoore commented May 25, 2015

will-moore commented May 25, 2015

will-moore commented May 25, 2015

will-moore commented May 25, 2015

joshmoore commented May 26, 2015

gusferguson commented May 26, 2015

gusferguson commented May 26, 2015

manics commented May 26, 2015

gusferguson commented May 26, 2015

manics commented May 26, 2015

joshmoore commented May 26, 2015

joshmoore commented May 26, 2015

will-moore commented May 26, 2015