-
Notifications
You must be signed in to change notification settings - Fork 100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge populate_metadata PRs (rebased onto develop) #5241
Merge populate_metadata PRs (rebased onto develop) #5241
Commits on Apr 12, 2017
-
Configuration menu - View commit details
-
Copy full SHA for a6e536e - Browse repository at this point
Copy the full SHA a6e536eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 71f6468 - Browse repository at this point
Copy the full SHA 71f6468View commit details -
Configuration menu - View commit details
-
Copy full SHA for ffebddc - Browse repository at this point
Copy the full SHA ffebddcView commit details -
Configuration menu - View commit details
-
Copy full SHA for c1798dd - Browse repository at this point
Copy the full SHA c1798ddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 325f641 - Browse repository at this point
Copy the full SHA 325f641View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2114ea8 - Browse repository at this point
Copy the full SHA 2114ea8View commit details -
Configuration menu - View commit details
-
Copy full SHA for dd7f54c - Browse repository at this point
Copy the full SHA dd7f54cView commit details -
Configuration menu - View commit details
-
Copy full SHA for c631b75 - Browse repository at this point
Copy the full SHA c631b75View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2d3868 - Browse repository at this point
Copy the full SHA d2d3868View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5182e8b - Browse repository at this point
Copy the full SHA 5182e8bView commit details -
Configuration menu - View commit details
-
Copy full SHA for da298c2 - Browse repository at this point
Copy the full SHA da298c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for cea2b3d - Browse repository at this point
Copy the full SHA cea2b3dView commit details -
populate_metadata: add support for datasets
Most methods for dataset loading and parsing were left unimplement. Now a `Dataset:`-style object can be passed to populate_metadata.py and images will be looked up by name. Note: there's a small bug with name lookup that will be corrected separately.
Configuration menu - View commit details
-
Copy full SHA for 65966a6 - Browse repository at this point
Copy the full SHA 65966a6View commit details -
populate_metadata: fix naming issue
The assumptions for well/imaging naming in a plate or screen differ from those from image naming in a dataset since there's no unique way to reference an image in a dataset like there is well "A1" for example. This commit loosens some of those rules to allow image columns and image name columns to work together in the case of datasets. The assumption is that for population the ID of the image in a dataset won't be known. Instead names of images will be used as a unique identifier. Currently only a warning is issued if the name is not unique.
Configuration menu - View commit details
-
Copy full SHA for c0a8743 - Browse repository at this point
Copy the full SHA c0a8743View commit details -
populate_metadata: Fix map and delete contexts
In general, populate_metadata.py looks to be in line for a refactoring. The number of if-clauses as well as the unhandled cases (like no catch-all for unknown targets in delete) is making this ever harder to work with. All tests passing.
Configuration menu - View commit details
-
Copy full SHA for dfa1b15 - Browse repository at this point
Copy the full SHA dfa1b15View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8da0f5e - Browse repository at this point
Copy the full SHA 8da0f5eView commit details -
Configuration menu - View commit details
-
Copy full SHA for e123e79 - Browse repository at this point
Copy the full SHA e123e79View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb4fc17 - Browse repository at this point
Copy the full SHA fb4fc17View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ed75f5 - Browse repository at this point
Copy the full SHA 4ed75f5View commit details -
Configuration menu - View commit details
-
Copy full SHA for f70c0ae - Browse repository at this point
Copy the full SHA f70c0aeView commit details -
populate_metadata: hide ValueResolver implementations
In order to allow Projects to smartly handle multiple images with the same name (though not in the same dataset), the internals of ValueResolver have been hidden within a ValueWrapper class. ValueResolver chooses once which ValueWrapper to use internally after which the various if/then blocks based on target object are no longer necessary (needs further refactoring). There *are* still if/then blocks basked on column-type. These could use some cleaning but will likely remain to be necessary for multiple-dispatch style handling.
Configuration menu - View commit details
-
Copy full SHA for 08e0a87 - Browse repository at this point
Copy the full SHA 08e0a87View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5001f38 - Browse repository at this point
Copy the full SHA 5001f38View commit details -
populate_metadata.py: add batches to write_to_omero
For extremely large screens (idr0016), both adding map annotations as well as deleting them lead to either PG errors or Ice.MessageSizeMax exceptions. Now both are done in batches of 1000.
Configuration menu - View commit details
-
Copy full SHA for d0f1770 - Browse repository at this point
Copy the full SHA d0f1770View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d4d4bf - Browse repository at this point
Copy the full SHA 5d4d4bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 005d652 - Browse repository at this point
Copy the full SHA 005d652View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6431fbe - Browse repository at this point
Copy the full SHA 6431fbeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f51da1 - Browse repository at this point
Copy the full SHA 3f51da1View commit details -
Configuration menu - View commit details
-
Copy full SHA for b42b797 - Browse repository at this point
Copy the full SHA b42b797View commit details -
Configuration menu - View commit details
-
Copy full SHA for e9e23b1 - Browse repository at this point
Copy the full SHA e9e23b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for df0da46 - Browse repository at this point
Copy the full SHA df0da46View commit details -
populate_metadata.py: handle duplicates in delete
The list of map annotations and file annotations have duplicates which cause secondary deletes to fail.
Configuration menu - View commit details
-
Copy full SHA for 9f0d355 - Browse repository at this point
Copy the full SHA 9f0d355View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4fa11b8 - Browse repository at this point
Copy the full SHA 4fa11b8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 82734f4 - Browse repository at this point
Copy the full SHA 82734f4View commit details -
Initial support for grouping map annotations
Currently this assumes a single top-level default group
Configuration menu - View commit details
-
Copy full SHA for 5704fe5 - Browse repository at this point
Copy the full SHA 5704fe5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d52b8c - Browse repository at this point
Copy the full SHA 3d52b8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for ec502c5 - Browse repository at this point
Copy the full SHA ec502c5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0eb1548 - Browse repository at this point
Copy the full SHA 0eb1548View commit details -
Integration test for multiple MapAnn groups. Fix Delete.
DeleteMapAnnotationContext modified to delete all configured namespaces if a config file is provided.
Configuration menu - View commit details
-
Copy full SHA for 9363daa - Browse repository at this point
Copy the full SHA 9363daaView commit details -
Always include target when attaching bulk-map-annotations
This partially reverts joshmoore@0e784ba which was a hack to modify the appearance of map annotations on wells vs images in plates in combination with https://github.com/openmicroscopy/openmicroscopy/pull/4038/files
Configuration menu - View commit details
-
Copy full SHA for 41ed3b3 - Browse repository at this point
Copy the full SHA 41ed3b3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 668673a - Browse repository at this point
Copy the full SHA 668673aView commit details -
Configuration menu - View commit details
-
Copy full SHA for cad3145 - Browse repository at this point
Copy the full SHA cad3145View commit details -
Configuration menu - View commit details
-
Copy full SHA for f064707 - Browse repository at this point
Copy the full SHA f064707View commit details -
Configuration menu - View commit details
-
Copy full SHA for 91a6d6c - Browse repository at this point
Copy the full SHA 91a6d6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 717a460 - Browse repository at this point
Copy the full SHA 717a460View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a3b363 - Browse repository at this point
Copy the full SHA 8a3b363View commit details -
Configuration menu - View commit details
-
Copy full SHA for 823f622 - Browse repository at this point
Copy the full SHA 823f622View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4db80a0 - Browse repository at this point
Copy the full SHA 4db80a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for d781548 - Browse repository at this point
Copy the full SHA d781548View commit details -
Configuration menu - View commit details
-
Copy full SHA for c3d5288 - Browse repository at this point
Copy the full SHA c3d5288View commit details -
Configuration menu - View commit details
-
Copy full SHA for 837da83 - Browse repository at this point
Copy the full SHA 837da83View commit details -
Configuration menu - View commit details
-
Copy full SHA for 698a460 - Browse repository at this point
Copy the full SHA 698a460View commit details -
Configuration menu - View commit details
-
Copy full SHA for 08a83d0 - Browse repository at this point
Copy the full SHA 08a83d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for ea2d4ca - Browse repository at this point
Copy the full SHA ea2d4caView commit details -
Add disabled _test_delete_map_annotation_context_dedup function
This should be enabled once deletion is fixed
Configuration menu - View commit details
-
Copy full SHA for 02ed997 - Browse repository at this point
Copy the full SHA 02ed997View commit details -
Get rid of unique_keys in CanonicalMapAnnotation
It just seems to add unnecessary complexity Also add ns and id to CanonicalMapAnnotation.__str__
Configuration menu - View commit details
-
Copy full SHA for a3a79cd - Browse repository at this point
Copy the full SHA a3a79cdView commit details -
populate_metadata: warn on missing image
When a well is missing from a plate, a warning is printed. The same now happens when an image is missing from a dataset. Likely, a `--strict` argument should be added which will force the existence of all objects.
Configuration menu - View commit details
-
Copy full SHA for 955258c - Browse repository at this point
Copy the full SHA 955258cView commit details -
Use jinja2 when parsing metadata key-values
This makes it easier to deal with URLs which require escaping.
Configuration menu - View commit details
-
Copy full SHA for 6eae350 - Browse repository at this point
Copy the full SHA 6eae350View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0303d58 - Browse repository at this point
Copy the full SHA 0303d58View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5559994 - Browse repository at this point
Copy the full SHA 5559994View commit details -
Configuration menu - View commit details
-
Copy full SHA for 334f699 - Browse repository at this point
Copy the full SHA 334f699View commit details -
Change batch size integration tests: replace 1000 with 1
It makes more sense to test small batch sizes since the integration tests use very few annotations. The alternative is to keep batch size 1000, but this unnecessarily increases the time to run the tests.
Configuration menu - View commit details
-
Copy full SHA for 34caeae - Browse repository at this point
Copy the full SHA 34caeaeView commit details -
populate_metadata: reduce memory usage
BulkToMapAnnotation for large screens (+100K wells) was being killed by the OOMKiller. Strategies include: * unload linked objects * drop `andReturn` where possible * return only sizes when possible * write actively in loops
Configuration menu - View commit details
-
Copy full SHA for 81851a3 - Browse repository at this point
Copy the full SHA 81851a3View commit details -
populate_metadata: don't store plate data
Storing all of the plate data for idr0016 led to death-by-OOM-killer during *initialization*. Now, Ice objects are not being stored but wrapper very thin Data objects.
Configuration menu - View commit details
-
Copy full SHA for 1f9b081 - Browse repository at this point
Copy the full SHA 1f9b081View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe835af - Browse repository at this point
Copy the full SHA fe835afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1dd9989 - Browse repository at this point
Copy the full SHA 1dd9989View commit details -
metadata: add wait option to CLI
The metadata plugin now takes a wait option as with graph operations and converts this into loop and ms settings to be passed to all write_to_omero methods (which internally use BaseClient.submit).
Configuration menu - View commit details
-
Copy full SHA for 8ce428f - Browse repository at this point
Copy the full SHA 8ce428fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 74c726b - Browse repository at this point
Copy the full SHA 74c726bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f5ce01 - Browse repository at this point
Copy the full SHA 8f5ce01View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9fe808b - Browse repository at this point
Copy the full SHA 9fe808bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 41357e7 - Browse repository at this point
Copy the full SHA 41357e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6166544 - Browse repository at this point
Copy the full SHA 6166544View commit details -
Configuration menu - View commit details
-
Copy full SHA for 577fc8d - Browse repository at this point
Copy the full SHA 577fc8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 69fd026 - Browse repository at this point
Copy the full SHA 69fd026View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6bbcb31 - Browse repository at this point
Copy the full SHA 6bbcb31View commit details -
Configuration menu - View commit details
-
Copy full SHA for 026842b - Browse repository at this point
Copy the full SHA 026842bView commit details -
Split TestPopulateMetadata into two classes
The new test method requires a clean environment for each test method, but this seems to break the existing test methods.
Configuration menu - View commit details
-
Copy full SHA for 7d52262 - Browse repository at this point
Copy the full SHA 7d52262View commit details -
Configuration menu - View commit details
-
Copy full SHA for d00fcc2 - Browse repository at this point
Copy the full SHA d00fcc2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 68962dc - Browse repository at this point
Copy the full SHA 68962dcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9909d00 - Browse repository at this point
Copy the full SHA 9909d00View commit details -
Configuration menu - View commit details
-
Copy full SHA for 45ce378 - Browse repository at this point
Copy the full SHA 45ce378View commit details -
Configuration menu - View commit details
-
Copy full SHA for 432023e - Browse repository at this point
Copy the full SHA 432023eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9d1647f - Browse repository at this point
Copy the full SHA 9d1647fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5cce5a9 - Browse repository at this point
Copy the full SHA 5cce5a9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 28e92d9 - Browse repository at this point
Copy the full SHA 28e92d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2622bd1 - Browse repository at this point
Copy the full SHA 2622bd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb05f6c - Browse repository at this point
Copy the full SHA fb05f6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1da4a4a - Browse repository at this point
Copy the full SHA 1da4a4aView commit details -
Configuration menu - View commit details
-
Copy full SHA for a4dcf19 - Browse repository at this point
Copy the full SHA a4dcf19View commit details -
Configuration menu - View commit details
-
Copy full SHA for afffb18 - Browse repository at this point
Copy the full SHA afffb18View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8137902 - Browse repository at this point
Copy the full SHA 8137902View commit details -
Configuration menu - View commit details
-
Copy full SHA for 004991c - Browse repository at this point
Copy the full SHA 004991cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f91bb8 - Browse repository at this point
Copy the full SHA 4f91bb8View commit details -
Configuration menu - View commit details
-
Copy full SHA for c69e622 - Browse repository at this point
Copy the full SHA c69e622View commit details -
Configuration menu - View commit details
-
Copy full SHA for 90f928f - Browse repository at this point
Copy the full SHA 90f928fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a90bd1 - Browse repository at this point
Copy the full SHA 9a90bd1View commit details
Commits on Apr 19, 2017
-
Configuration menu - View commit details
-
Copy full SHA for 94e17b1 - Browse repository at this point
Copy the full SHA 94e17b1View commit details
Commits on Apr 20, 2017
-
Configuration menu - View commit details
-
Copy full SHA for fcce33e - Browse repository at this point
Copy the full SHA fcce33eView commit details
Commits on Apr 21, 2017
-
Configuration menu - View commit details
-
Copy full SHA for 77a02e0 - Browse repository at this point
Copy the full SHA 77a02e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for ac0d312 - Browse repository at this point
Copy the full SHA ac0d312View commit details
Commits on Apr 24, 2017
-
Configuration menu - View commit details
-
Copy full SHA for 07e9010 - Browse repository at this point
Copy the full SHA 07e9010View commit details