Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge populate_metadata PRs (rebased onto develop) #5241

Merged
merged 100 commits into from
May 9, 2017
Merged

Merge populate_metadata PRs (rebased onto develop) #5241

merged 100 commits into from
May 9, 2017

Commits on Apr 12, 2017

  1. Add columns flag to the parser

    emilroz authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    a6e536e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    71f6468 View commit details
    Browse the repository at this point in the history
  3. Flake8

    emilroz authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    ffebddc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c1798dd View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    325f641 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    2114ea8 View commit details
    Browse the repository at this point in the history
  7. Return long not int

    emilroz authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    dd7f54c View commit details
    Browse the repository at this point in the history
  8. Set the StringColumn size

    emilroz authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    c631b75 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d2d3868 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    5182e8b View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    da298c2 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    cea2b3d View commit details
    Browse the repository at this point in the history
  13. populate_metadata: add support for datasets

    Most methods for dataset loading and parsing were left unimplement.
    Now a `Dataset:`-style object can be passed to populate_metadata.py
    and images will be looked up by name. Note: there's a small bug with
    name lookup that will be corrected separately.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    65966a6 View commit details
    Browse the repository at this point in the history
  14. populate_metadata: fix naming issue

    The assumptions for well/imaging naming in a plate or screen
    differ from those from image naming in a dataset since there's
    no unique way to reference an image in a dataset like there is
    well "A1" for example. This commit loosens some of those rules
    to allow image columns and image name columns to work together
    in the case of datasets. The assumption is that for population
    the ID of the image in a dataset won't be known. Instead names
    of images will be used as a unique identifier. Currently only
    a warning is issued if the name is not unique.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    c0a8743 View commit details
    Browse the repository at this point in the history
  15. populate_metadata: Fix map and delete contexts

    In general, populate_metadata.py looks to be in line for a
    refactoring. The number of if-clauses as well as the unhandled
    cases (like no catch-all for unknown targets in delete) is
    making this ever harder to work with.
    
    All tests passing.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    dfa1b15 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    8da0f5e View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    e123e79 View commit details
    Browse the repository at this point in the history
  18. Tables: Add DatasetColumn (new API)

    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    fb4fc17 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    4ed75f5 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    f70c0ae View commit details
    Browse the repository at this point in the history
  21. populate_metadata: hide ValueResolver implementations

    In order to allow Projects to smartly handle multiple images
    with the same name (though not in the same dataset), the internals
    of ValueResolver have been hidden within a ValueWrapper class.
    ValueResolver chooses once which ValueWrapper to use internally
    after which the various if/then blocks based on target object are
    no longer necessary (needs further refactoring). There *are* still
    if/then blocks basked on column-type. These could use some cleaning
    but will likely remain to be necessary for multiple-dispatch style
    handling.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    08e0a87 View commit details
    Browse the repository at this point in the history
  22. fix flake8

    atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    5001f38 View commit details
    Browse the repository at this point in the history
  23. populate_metadata.py: add batches to write_to_omero

    For extremely large screens (idr0016), both adding
    map annotations as well as deleting them lead to either
    PG errors or Ice.MessageSizeMax exceptions. Now both
    are done in batches of 1000.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    d0f1770 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    5d4d4bf View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    005d652 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    6431fbe View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    3f51da1 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    b42b797 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    e9e23b1 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    df0da46 View commit details
    Browse the repository at this point in the history
  31. populate_metadata.py: handle duplicates in delete

    The list of map annotations and file annotations
    have duplicates which cause secondary deletes to
    fail.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    9f0d355 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    4fa11b8 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    82734f4 View commit details
    Browse the repository at this point in the history
  34. Initial support for grouping map annotations

    Currently this assumes a single top-level default group
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    5704fe5 View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    3d52b8c View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    ec502c5 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    0eb1548 View commit details
    Browse the repository at this point in the history
  38. Integration test for multiple MapAnn groups. Fix Delete.

    DeleteMapAnnotationContext modified to delete all configured namespaces if a config file is provided.
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    9363daa View commit details
    Browse the repository at this point in the history
  39. Always include target when attaching bulk-map-annotations

    This partially reverts
    joshmoore@0e784ba
    which was a hack to modify the appearance of map annotations on wells vs images in plates in combination with
    https://github.com/openmicroscopy/openmicroscopy/pull/4038/files
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    41ed3b3 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    668673a View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    cad3145 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    f064707 View commit details
    Browse the repository at this point in the history
  43. Fix some travis style issues

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    91a6d6c View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    717a460 View commit details
    Browse the repository at this point in the history
  45. Rename map-ann group tests to ns

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    8a3b363 View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    823f622 View commit details
    Browse the repository at this point in the history
  47. Configuration menu
    Copy the full SHA
    4db80a0 View commit details
    Browse the repository at this point in the history
  48. Fix minor bug with namespaces

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    d781548 View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    c3d5288 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    837da83 View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    698a460 View commit details
    Browse the repository at this point in the history
  52. Configuration menu
    Copy the full SHA
    08a83d0 View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    ea2d4ca View commit details
    Browse the repository at this point in the history
  54. Add disabled _test_delete_map_annotation_context_dedup function

    This should be enabled once deletion is fixed
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    02ed997 View commit details
    Browse the repository at this point in the history
  55. Get rid of unique_keys in CanonicalMapAnnotation

    It just seems to add unnecessary complexity
    Also add ns and id to CanonicalMapAnnotation.__str__
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    a3a79cd View commit details
    Browse the repository at this point in the history
  56. populate_metadata: warn on missing image

    When a well is missing from a plate, a warning is printed. The
    same now happens when an image is missing from a dataset. Likely,
    a `--strict` argument should be added which will force the existence
    of all objects.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    955258c View commit details
    Browse the repository at this point in the history
  57. Use jinja2 when parsing metadata key-values

    This makes it easier to deal with URLs which require escaping.
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    6eae350 View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    0303d58 View commit details
    Browse the repository at this point in the history
  59. Configuration menu
    Copy the full SHA
    5559994 View commit details
    Browse the repository at this point in the history
  60. Configuration menu
    Copy the full SHA
    334f699 View commit details
    Browse the repository at this point in the history
  61. Change batch size integration tests: replace 1000 with 1

    It makes more sense to test small batch sizes since the integration tests use very few annotations.
    The alternative is to keep batch size 1000, but this unnecessarily increases the time to run the tests.
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    34caeae View commit details
    Browse the repository at this point in the history
  62. populate_metadata: reduce memory usage

    BulkToMapAnnotation for large screens (+100K wells)
    was being killed by the OOMKiller. Strategies include:
    
     * unload linked objects
     * drop `andReturn` where possible
     * return only sizes when possible
     * write actively in loops
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    81851a3 View commit details
    Browse the repository at this point in the history
  63. populate_metadata: don't store plate data

    Storing all of the plate data for idr0016 led to
    death-by-OOM-killer during *initialization*. Now,
    Ice objects are not being stored but wrapper very
    thin Data objects.
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    1f9b081 View commit details
    Browse the repository at this point in the history
  64. Configuration menu
    Copy the full SHA
    fe835af View commit details
    Browse the repository at this point in the history
  65. Configuration menu
    Copy the full SHA
    1dd9989 View commit details
    Browse the repository at this point in the history
  66. metadata: add wait option to CLI

    The metadata plugin now takes a wait option as with
    graph operations and converts this into loop and ms
    settings to be passed to all write_to_omero methods
    (which internally use BaseClient.submit).
    joshmoore authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    8ce428f View commit details
    Browse the repository at this point in the history
  67. Configuration menu
    Copy the full SHA
    74c726b View commit details
    Browse the repository at this point in the history
  68. Configuration menu
    Copy the full SHA
    8f5ce01 View commit details
    Browse the repository at this point in the history
  69. Configuration menu
    Copy the full SHA
    9fe808b View commit details
    Browse the repository at this point in the history
  70. Configuration menu
    Copy the full SHA
    41357e7 View commit details
    Browse the repository at this point in the history
  71. Configuration menu
    Copy the full SHA
    6166544 View commit details
    Browse the repository at this point in the history
  72. Fix missing list[] in get_namespaces()

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    577fc8d View commit details
    Browse the repository at this point in the history
  73. Configuration menu
    Copy the full SHA
    69fd026 View commit details
    Browse the repository at this point in the history
  74. test deletion with namespace filter

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    6bbcb31 View commit details
    Browse the repository at this point in the history
  75. Configuration menu
    Copy the full SHA
    026842b View commit details
    Browse the repository at this point in the history
  76. Split TestPopulateMetadata into two classes

    The new test method requires a clean environment for each test method, but this seems to break the existing test methods.
    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    7d52262 View commit details
    Browse the repository at this point in the history
  77. Create global test var MAPR_NS_GENE

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    d00fcc2 View commit details
    Browse the repository at this point in the history
  78. Configuration menu
    Copy the full SHA
    68962dc View commit details
    Browse the repository at this point in the history
  79. Configuration menu
    Copy the full SHA
    9909d00 View commit details
    Browse the repository at this point in the history
  80. Remove unused import

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    45ce378 View commit details
    Browse the repository at this point in the history
  81. Configuration menu
    Copy the full SHA
    432023e View commit details
    Browse the repository at this point in the history
  82. Configuration menu
    Copy the full SHA
    9d1647f View commit details
    Browse the repository at this point in the history
  83. Configuration menu
    Copy the full SHA
    5cce5a9 View commit details
    Browse the repository at this point in the history
  84. Handle a null mimetype in OriginalFile

    manics authored and atarkowska committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    28e92d9 View commit details
    Browse the repository at this point in the history
  85. Configuration menu
    Copy the full SHA
    2622bd1 View commit details
    Browse the repository at this point in the history
  86. Configuration menu
    Copy the full SHA
    fb05f6c View commit details
    Browse the repository at this point in the history
  87. Configuration menu
    Copy the full SHA
    1da4a4a View commit details
    Browse the repository at this point in the history
  88. Configuration menu
    Copy the full SHA
    a4dcf19 View commit details
    Browse the repository at this point in the history
  89. Configuration menu
    Copy the full SHA
    afffb18 View commit details
    Browse the repository at this point in the history
  90. Configuration menu
    Copy the full SHA
    8137902 View commit details
    Browse the repository at this point in the history
  91. Configuration menu
    Copy the full SHA
    004991c View commit details
    Browse the repository at this point in the history
  92. Configuration menu
    Copy the full SHA
    4f91bb8 View commit details
    Browse the repository at this point in the history
  93. Configuration menu
    Copy the full SHA
    c69e622 View commit details
    Browse the repository at this point in the history
  94. Configuration menu
    Copy the full SHA
    90f928f View commit details
    Browse the repository at this point in the history
  95. Configuration menu
    Copy the full SHA
    9a90bd1 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2017

  1. Configuration menu
    Copy the full SHA
    94e17b1 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2017

  1. Configuration menu
    Copy the full SHA
    fcce33e View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2017

  1. py26 fix

    atarkowska committed Apr 21, 2017
    Configuration menu
    Copy the full SHA
    77a02e0 View commit details
    Browse the repository at this point in the history
  2. improve skipping marker

    atarkowska committed Apr 21, 2017
    Configuration menu
    Copy the full SHA
    ac0d312 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2017

  1. fix py26 failing test

    atarkowska committed Apr 24, 2017
    Configuration menu
    Copy the full SHA
    07e9010 View commit details
    Browse the repository at this point in the history