Added tests for Datasetcolumn

[muhanadz]

What this PR does

Added tests for the new Datasetcolumn in response to ome/omero-py#309 along with an h5 reference table for tests. Created a new test and reference table to not overwrite old ones.

Testing this PR

Ran all tests in /openmicroscopy/components/tools/OmeroPy/test/integration/tablestest with pytest and omero-py from ome/omero-py#309 installed on the server and in the local venv:

(venv) ➜  openmicroscopy git:(datasetColumn_tests) ✗ pytest /Users/muhanad/Glencoe/openmicroscopy/components/tools/OmeroPy/test/integration/tablestest/.
=========================================== test session starts ============================================
platform darwin -- Python 3.7.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/muhanad/Glencoe/openmicroscopy/components/tools, configfile: pytest.ini
collected 33 items                                                                                         

components/tools/OmeroPy/test/integration/tablestest/test_backwards_compatibility.py .......         [ 21%]
components/tools/OmeroPy/test/integration/tablestest/test_populate_metadata.py .                     [ 24%]
components/tools/OmeroPy/test/integration/tablestest/test_service.py ....F...................F       [100%]

================================================= FAILURES =================================================
___________________________________________ TestTables.test2098 ____________________________________________

self = <test_service.TestTables object at 0x117bfd4d0>

    @pytest.mark.broken(ticket="11534")
    def test2098(self):
        """
        Creates and downloads an HDF file and checks
        that its size and hash match whats in the db
        """
        grid = self.client.sf.sharedResources()
        table = grid.newTable(1, "/test")
        assert table
    
        lc = columns.LongColumnI('lc', 'desc', [1])
    
        file = None
        try:
            file = table.getOriginalFile()
            assert file
            table.initialize([lc])
            table.addData([lc])
        finally:
            # Not deleting since queried
            table.close()
    
        # Reload the file
        file = self.client.sf.getQueryService().get(
            "OriginalFile", file.id.val)
    
        # Check values
>       p = path.path(self.tmpfile())
E       AttributeError: type object 'path' has no attribute 'path'

components/tools/OmeroPy/test/integration/tablestest/test_service.py:162: AttributeError
_______________________________________ TestTables.testReadOnlyFile ________________________________________

self = <test_service.TestTables object at 0x11bd82150>

    @pytest.mark.broken(ticket="unimplemented")
    def testReadOnlyFile(self):
        """
        Create an HDF5 file on the server, and then mark it read-only.
        The server should still allow you to load & read that file.
        """
        self.testBlankTable()  # ofile
    
>       filename = self.unique_dir + "/file.txt"
E       AttributeError: 'TestTables' object has no attribute 'unique_dir'

components/tools/OmeroPy/test/integration/tablestest/test_service.py:669: AttributeError
============================================= warnings summary =============================================
venv/lib/python3.7/site-packages/Ice.py:14
  /Users/muhanad/Glencoe/openmicroscopy/venv/lib/python3.7/site-packages/Ice.py:14: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import sys, string, imp, os, threading, warnings, datetime

venv/lib/python3.7/site-packages/_pytest/config/__init__.py:1233
  /Users/muhanad/Glencoe/openmicroscopy/venv/lib/python3.7/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: DJANGO_SETTINGS_MODULE
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

venv/lib/python3.7/site-packages/tables/__init__.py:99
venv/lib/python3.7/site-packages/tables/__init__.py:99
  /Users/muhanad/Glencoe/openmicroscopy/venv/lib/python3.7/site-packages/tables/__init__.py:99: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
    from .utilsextension import (

venv/lib/python3.7/site-packages/omero/util/populate_metadata.py:76
  /Users/muhanad/Glencoe/openmicroscopy/venv/lib/python3.7/site-packages/omero/util/populate_metadata.py:76: DeprecationWarning: This module is deprecated as of OMERO 5.4.8. Use the module in the omero-metadata project available from https://pypi.org/project/omero-metadata/ instead.
    DeprecationWarning)

OmeroPy/test/integration/tablestest/test_backwards_compatibility.py: 18 warnings
  /Users/muhanad/Glencoe/openmicroscopy/components/tools/OmeroPy/test/integration/tablestest/test_backwards_compatibility.py:84: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
    return numpy.fromstring(x, count=len(x), dtype=tables.UInt8Atom())

OmeroPy/test/integration/tablestest/test_service.py::TestTables::testMask
OmeroPy/test/integration/tablestest/test_service.py::TestTables::testMask
OmeroPy/test/integration/tablestest/test_service.py::TestTables::testAllColumnsSameTable
OmeroPy/test/integration/tablestest/test_service.py::TestTables::testAllColumnsSameTable
OmeroPy/test/integration/tablestest/test_service.py::TestTables::testAllColumnsSameTable
OmeroPy/test/integration/tablestest/test_service.py::TestTables::testAllColumnsSameTable
  /Users/muhanad/Glencoe/openmicroscopy/components/tools/OmeroPy/test/integration/tablestest/test_service.py:62: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
    return numpy.fromstring(x, count=len(x), dtype=tables.UInt8Atom())

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================= short test summary info ==========================================
FAILED components/tools/OmeroPy/test/integration/tablestest/test_service.py::TestTables::test2098 - Attri...
FAILED components/tools/OmeroPy/test/integration/tablestest/test_service.py::TestTables::testReadOnlyFile
================================ 2 failed, 31 passed, 29 warnings in 12.17s ================================

2 failures are expected as they are marked with @pytest.mark.broken.

Related reading

Link to cards, tickets, other PRs:

PR in reference: ome/omero-py#309
DatasetColumn header definition: https://github.com/ome/omero-blitz/blob/bf2802814f4a1d5056c64ad825b583339f474ede/src/main/slice/omero/Tables.ice#L62
Omero-metadata ready to handle: https://github.com/ome/omero-metadata/blob/148b6a4d4e5cf1deb78e3f087b5907024c749828/src/omero_metadata/populate.py#L457

[will-moore]

I don't see these new tests at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/984/testReport/OmeroPy.test.integration.tablestest.test_backwards_compatibility/
I guess this PR may not be included....

[sbesson]

@muhanadz sent you an invite to the organization so that your PRs get included by default in the future

[muhanadz]

Thanks @sbesson, I have accepted. Do I need to reopen this PR?

[sbesson]

No need to re-open, it should be automatically included in the nightly CI builds.

[will-moore]

New tests now passing at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/985/testReport/OmeroPy.test.integration.tablestest.test_backwards_compatibility/TestBackwardsCompatibility_5_10_3/

[will-moore]

Hi, this seems like a fair bit of code duplication to test one more column type. Is it the lack of other tables tests that prompted the reuse of backwards-compatibility tests or do we actually need to test the backwards compatibility with 5.10.3?
I guess it doesn't hurt - just trying to understand what's needed (e.g. more tables tests elsewhere) to make it easier to add other column types in the future e.g. ShapeColumn.

[muhanadz]

@will-moore You are absolutely correct in that it's a fair bit of code duplication. I was under the impression that the tests written for 5_3_4 were mainly done in response to new column types being added. I also found out that any modification to the written functions to test dataset column would fail since the reference h5 tables are missing the column type (specifically here). So I wrote a new test with a new reference table and labeled the version to be whatever server I was on when I generated that table.

Perhaps we don't need to reuse backward compatibility, I just wasn't exactly sure where to add tests for the dataset column.

[will-moore]

I don't know how important it is to test the backwards compatibility here (is it useful to keep the new test)? cc @joshmoore

But I imagine the code you've added could be moved to a new test_tables.py or something like that, and simply create the table and check it within the test itself. This could then be extended etc. in future without needing to duplicate.

[muhanadz]

@will-moore cc @joshmoore I have moved my tests to test_tables.py to test the new DatasetColumn. Tests are passing:
https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/lastCompletedBuild/testReport/OmeroPy.test.integration.tablestest.test_tables/

This test should be easier to modify or add to in case of new column types without needing to duplicate anything.

[will-moore]

Minor suggestions. Only 1 tiny code fix.

[will-moore]

Great, thanks. LGTM 👍 .
Let's just let the tests run once more before we merge...

[will-moore]

Just 1 test now, and passing: https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/1011/testReport/OmeroPy.test.integration.tablestest.test_tables/TestTableIntegrity/
👍