FilterObjects by CalculateMath measurement works on macOS but fails on Linux

[bethac07]

From the forum:

"
I found interesting behavior of FilterObjects failing on both CentOS 7.3 and Ubuntu 16.04 Linux in CellProfiler 2.2.0. I was trying to reproduce this on Distributed CellProfiler, but looks like my AWS setup will take a while.

Steps to reproduce:

Grab the pipeline.
Drag and drop each of these 3 images into the File list.
In the FilterObjects module toward the end of the pipeline, set: Category = "Math".
Wait for a few seconds till the error text changes to "Math has an unmeasured feature name"
Set Measurement = "ect_signal"
Result: On Linux, the measurement Category drop-down menu immediately resets to [None]. There is no error message in the shell. Running with -L DEBUG just shows the usual HDF5Dict.flush(). However this works fine on macOS 10.11, and I get sensible output running the pipeline.

Other notes: using a different measurement works fine on Linux, for example:

• Category = "Location"
• Measurement = "CenterMassIntensity_X"
• Image = "ect_raw"

I tried to dive into debugging the module, but for some reason "Test" > "Break into debugger" is not available on macOS; only Linux. The relevant wiki section is empty but on Linux I can enter Test Mode and set break points running "Test" > "Break into debugger" and set a breakpoint at the beginning of the add_measurement() function:

(Pdb) break /home/omsai/code/CellProfiler/cellprofiler/modules/filterobjects.py:279
Breakpoint 1 at /home/omsai/code/CellProfiler/cellprofiler/modules/filterobjects.py:279
(Pdb) continue
... and the rerun the module in Test Mode. The breakpoint returns control back to the debugger when clicking on the drop-down and can inspect the module input validation code.

So my questions are:

Can anyone else reproduce the issue on Linux?
How does one access the debugger on macOS for me to compare why it is working there? On macOS, per David Logan's command I'm launching CellProfiler from the shell using:
arch -x86_64 /Applications/CellProfiler.app/Contents/MacOS/CellProfiler -L DEBUG
Although this doesn't even give the usual debug output. Also using -h, CellProfiler exists with no help message. It's like stdout / stderr is being directed elsewhere instead of the terminal.

"

[omsai]

Some progress:

On macOS, if I save the pipeline file after choosing the measurement, then it reopens fine in Linux. Additionally, the bug no longer happens on Linux.

This makes me think something is wrong with the original pipeline file itself that gets "fixed" when being saved on macOS. As the pipelines are in hdf5, I wrote a diff utility h5pipe2txt.py inspired by the test_measurements.py unittests:

#!/usr/bin/env python
'''Extract pipeline text from cellprofiler hdf5 file or stdin.'''
from __future__ import print_function

import cellprofiler.measurements as cpmeas


def get_pipeline_text(file_): # pylint: disable=redefined-outer-name
    '''Return pipeline text from cellprofiler pipeline hdf5 file.'''
    meas = cpmeas.Measurements(filename=file_, mode='r')
    return meas[cpmeas.EXPERIMENT, 'Pipeline_Pipeline']


if __name__ == '__main__':
    import os
    import sys
    import tempfile     # cellprofiler.Measure doesn't handle StringIO

    # pylint: disable=superfluous-parens,invalid-name
    if len(sys.argv) == 1:
        sys.stderr.write((("Usage {} [FILENAME]\n\n"
                           "When FILENAME is -, read standard input.\n")
                          .format(sys.argv[0])))
        sys.stderr.flush()
        sys.exit(1)
    elif sys.argv[1] == '-':
        # Standard input is useful to processes files fetched from
        # `git show ...`.
        stdin = []
        while True:
            line = sys.stdin.readline()
            if not line:
                break
            stdin.append(line)
        file_descriptor, filename = tempfile.mkstemp()
        with os.fdopen(file_descriptor, 'w') as file_object:
            file_object.writelines(stdin)
        sys.stdout.write(get_pipeline_text(filename))
        sys.stdout.flush()
        sys.stderr.flush()
        os.remove(filename)
    else:
        for file_ in sys.argv[1:]:
            print(get_pipeline_text(sys.argv[1]))

The only difference I see in the module values is the measurement itself:

omsai@Core-Leopard-Extreme:/share/Pariksheet/consultations/all-plates/omsai$ diff -u <( h5pipe2txt.py coc-orig.cpproj ) <( h5pipe2txt.py coc-rewrite.cpproj )
--- /dev/fd/63	2017-02-16 17:13:59.804529095 -0500
+++ /dev/fd/62	2017-02-16 17:13:59.804529095 -0500
@@ -403,9 +403,178 @@
     Measurement count:1
     Additional object count:0
     Assign overlapping child to:Both parents
-    Select the measurement to filter by:Math
+    Select the measurement to filter by:Math_ect_signal
     Filter using a minimum measurement value?:Yes
     Minimum value:0
     Filter using a maximum measurement value?:No
     Maximum value:1.0

Maybe something is wrong in the cpproj metadata?