-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support connection to LIMS #93
Comments
The team implementing a LIMS API for CellLocator to work with baked in a validation mechanism that makes it important to track the expected number of pinned locations for a particular (slice) specimen. The number of pins in a JSON posted to the API must match that number. If a user intends to add a pin, then the Cell Locator will need to use another API call to first increment the number of expected cells. The 'kind' of data that tracks the number of expected cells is labeled: "IVSCC expected cell count". The base URL of the calls should be configurable. Our production system URL is: http://lims2 . My dev system URL is http://ibs-waynew-vm1:3000 (for example). Will add additional comments with API spec |
Fetch Specimen Metadata Returns json metadata of a specimen based on the kind of metadata. /specimen_metadata/view GET
Optional:
Success Response
Error Responses
Sample Call /specimen_metadata/view?specimen_id=1234&kind=IVSCC%20cell%20locations /specimen_metadata/view?specimen_id=1234&kind=IVSCC%20cell%20locations&version_number=3
|
Store Specimen Metadata Stores json metadata of a specimen based on the kind of metadata. /specimen_metadata/store POST
Data Params - POST body
Example Body: {
"specimen_id": 1234,
"kind": "IVSCC cell locations",
"data": {
"foo": "bar"
}
} Example Body: {
"specimen_id": 1234,
"kind": "IVSCC expected cell count",
"data": 2
} Required Headers
Success Response
Error Responses
Sample Call /specimen_metadata/store |
Note: You can format content easily, you can for example see how I ensured the json content is properly highlighted trying to edit this comment: #93 (comment) |
Hi @jcfr I corrected the duplicated sentence in the comment above. The specimen_id will come in a JSON that is specified when opening Cell Locator. I am trying to capture two related cases here. In one of them, we will pass in a stub with just the specimen_id. Can you propose a location in the JSON file to put the id? (and an associated name)? Summaries of the two cases:
PatchSeq Mouse morphology case:
|
Thanks for providing these additional details. It will allow to move forward. I will get back to you regarding where in the json the specimen_id should be. In the meantime, could you provide more details regarding the POST request that will be submitted to LIMS, what are the exact data expected to be submitted ? |
This example has the cell locator information for specimen_id 1234 in the 'data' part of the JSON: This could be posted to http://lims2/specimen_metadata/store to update the cell locations for that specimen. |
For now, URL could be added to https://github.com/BICCN/cell-locator/blob/master/Applications/CellLocatorApp/Resources/Settings/DefaultSettings.ini |
Is this a file that a Windows user has access to after they install the program? I tried to find that file on my installation but could not. |
This file is used to initialize the application settings saved in your home folder. In the case of CellLocator the settings file would be in |
For development purposes, I've created a small flask application to mock the behavior of the LIMS api locally. I'd like to verify that my behavior is correct: If I make a valid POST request to {
"kind": "IVSCC cell locations",
"specimen_id": 1234,
"data": {"foo": "bar"}
} Then I should expect a GET request to {
"version_number": 2,
"data": {
"kind": "IVSCC cell locations",
"specimen_id": 1234,
"data": {"foo": "bar"}
}
} Specifically, the path to the markup data in the result is nested by two "data" keys. Is this correct? Regarding LVSCC expected cell count: is the value for this expected to differ from I've been making HTTP requests with Advantages to the qt implementation would be asynchronous network calls, and we won't have to change that code when Python 3 comes around. @jcfr, do you have any thoughts on this? You can see my changes in a branch on my fork here: https://github.com/allemangd-forks/cell-locator/tree/lims Currently, all the behavior is enabled when a |
I suggest you look into https://github.com/commontk/qRestAPI, it is already available in Slicer. There is a test illustrating how it could be used: https://github.com/commontk/qRestAPI/blob/master/Testing/qMidasAPITest.cpp It is not expose in python, but would be easy to do so. |
Just for now, I've just uploaded my mock app to my own repo; allemangD/lims-mock. I'll transfer this to a more appropriate user or organization, but I'm unsure where exactly it should belong. @wbwakeman, could you verify that this correctly mimics the behavior of LIMS like we discussed in our meeting yesterday? |
The mock lims app has been transfered to the KitwareMedical organization: |
@jcfr @allemangD I had a chance today to compare the behavior of your mock LIMS API with my test system. Here is what difference that I notice |
Here is a related item that just probably was not clear. Running the GET with the 'IVSCC expected cell count" kind returns just an integer in the 'data' payload:
|
One other note too just FYI, the POST that adds new cell locations will do validation to ensure that there is a Markups field under 'data' that contains an array of cell locations A valid input is available here: |
This was clarified in discussion for BICCN/cell-locator#93.
This commit updates CellLocator and Slicer adding support for flags `--lims-specimen-id` and `--lims-base-url`. If `--lims-specimen-id` is provided, it is used: * At application startup time to (1) retrieve the corresponding annotation from LIMS and load it by using the `/specimen_metadata/view` endpoint. (2) enable the "Upload Annotation" button * After user initiate annotation upload to LIMS while using the `/specimen_metadata/store` endpoint. The "kind" parameter for both endpoints `/specimen_metadata/view` and `/specimen_metadata/store` is set to "IVSCC cell locations" IVSCC expected cell count is currently unsupported. For testing this functionality, a mock server as been implemented. Relevant instructions are available here: https://github.com/KitwareMedical/AllenInstituteMockLIMS List of Slicer changes: $ git shortlog 0661948e5..9b00dc521f --no-merges David Allemang (1): [cell-locator] ENH: Add command line args --lims-specimen-id and --lims-base-url for LIMS integration.
This was clarified in discussion for #93.
Interestingly, we now have a different message. Instead of:
we have
Consider changing:
into
|
Now we're getting somewhere:
This is expected, because there is not yet any specimen_metadata records for this specimen (that is what we are going to save as a result of this process).
|
Moving forward, we will make sure to log the status code and message. In the meantime, is there anything we need to change regarding the endpoints we call ? if yes, could you suggest change to https://github.com/KitwareMedical/AllenInstituteMockLIMS#endpoints |
I will have a look at the endpoints page. I guess the better log message would be "Failed to load any annotations for specimen ID " instead of "Failed to load specimen ID". The next issue is trying to save something. Now that I have opened Cell Locator and it knows what specimen I am working with (even though there are not yet any annotations stored in the database), I want to see the specimen id listed in the Cell Locator and I want to click on 'Save to LIMS'. This is the log when I try to save now:
|
To get more details, locally apply similar changes to function cell-locator/Modules/Scripted/Home/Home.py Lines 330 to 352 in 6938f2b
Few options:
Sounds good, we will use this message instead. |
I attempted that, but I've got some syntax incorrect, because there is a slight different in the arguments for those two.
My attempt:
|
For "specimen id listed in the Cell Locator", I vote for option #2, the text box. |
The following should be implemented then: try:
res = urllib2.urlopen(url, data=body)
except urllib2.URLError as e:
- logging.error('Failed to connect to LIMS server')
+ logging.error('Failed to connect to LIMS server. [url: %s, code: %s, reason: %s]' % (url, e.code, e.reason))
+ logging.error("payload: %s" % body)
return
if res.getcode() != 200:
- logging.error('Failed to store specimen ID %s to LIMS server.', specimenID)
+ logging.error('Failed to store specimen ID %s to LIMS server. [code: %s, url: %s]' % (specimenID, res.getcode(), url))
Sounds good. Next question: When the application is started with LIMS integration, I suggest we hide button like "New/Save/SaveAs/Load" and only keep "Save to LIMS". I anticipate this would allow to keep the UI simple and avoid confusion. What do you think ? |
Hmmm...I don't get it. This is the new output:
When I drop the same JSON into a REST tool ("Insomnia") and make the same POST to the same system, it succeeds. Regarding hiding "New/Save/SaveAs/Load" when the tool was launched using |
Does the POST include the required header: |
Good point. Thanks for testing 🙏 We will improving the error handling and switch to using requests to properly deal with header setting. We will also update the mock application to expect a json pay load to ensure local testing work as expected. Once this is done, we will get back to you for an other round of QA. |
@jcfr @allemangD I tried the new version and it looks like we are progressing. Here is the output from the console:
It does seem like the format of the saved output has changed. Based on the previous format, our developer put in a validation for the "Markups" key, as described in the July 13th comment and with an example in this gist. Let me know if you can change your output format to match, or if we need to update our output validation code. |
I see that the current implementation places the annotations in the key What validation is done for each element of that To change the key used locally, you can change lines 239 and 256 in the same file, cell-locator/Modules/Scripted/Home/Home.py Lines 235 to 266 in 948d187
diff --git a/Modules/Scripted/Home/Home.py b/Modules/Scripted/Home/Home.py
index 368839a..77dea97 100644
--- a/Modules/Scripted/Home/Home.py
+++ b/Modules/Scripted/Home/Home.py
@@ -236,7 +236,7 @@ class AnnotationManager:
"""Convert this collection to dict representation, suitable for json serialization."""
return {
- 'markups': [annotation.toDict() for annotation in self.annotations],
+ 'Markups': [annotation.toDict() for annotation in self.annotations],
'currentId': self.currentId,
'referenceView': self.referenceView,
@@ -253,7 +253,7 @@ class AnnotationManager:
manager = cls()
- manager.annotations = [Annotation.fromDict(item) for item in data['markups']]
+ manager.annotations = [Annotation.fromDict(item) for item in data['Markups']]
manager.currentId = data['currentId']
manager.referenceView = data['referenceView'] |
Thanks for the feedback. There is now a PR under review documenting the format: #145
To keep the code base simple, I think updating the validation code is the most sensible things to do. Keeping all key lower case makes sense. |
For sake of testing locally, applying the suggestion of @allemangD make sense 👍 |
The only other validation that we are doing is a count of the number of elements underneath the "Markups" key. We will check that count to see that it matches what is in the LIMS database. |
After making that change, it saves to LIMS :-) |
I am also thinking we should remove some properties from the annotation file and have there initialized at loading by the application. @wbwakeman What do you think ? See below for the proposed simplification Other information we should probably include in the JSON under a
Also which terminologies to you use in LIMS ? I think we should change the diff --git a/annotation.json b/annotation-cleaned.json
index 786ee63..e6ae130 100644
--- a/annotation.json
+++ b/annotation-cleaned.json
@@ -2,16 +2,9 @@
"markups": [
{
"markup": {
- "type": "ClosedCurve",
"coordinateSystem": "LPS",
- "locked": false,
- "labelFormat": "%N-%d",
"controlPoints": [
{
- "id": "1",
- "label": "MarkupsClosedCurve-1",
- "description": "",
- "associatedNodeID": "vtkMRMLScalarVolumeNode1",
"position": [
-7725.476777936878,
5071.924649397559,
@@ -28,16 +21,8 @@
0,
1
],
- "selected": true,
- "locked": false,
- "visibility": true,
- "positionStatus": "defined"
},
{
- "id": "2",
- "label": "MarkupsClosedCurve-2",
- "description": "",
- "associatedNodeID": "vtkMRMLScalarVolumeNode1",
"position": [
-8785.61316343005,
5514.035987881592,
@@ -54,16 +39,8 @@
0,
1
],
- "selected": true,
- "locked": false,
- "visibility": true,
- "positionStatus": "defined"
},
{
- "id": "3",
- "label": "MarkupsClosedCurve-3",
- "description": "",
- "associatedNodeID": "vtkMRMLScalarVolumeNode1",
"position": [
-7732.355384361564,
5984.77667260136,
@@ -80,16 +57,8 @@
0,
1
],
- "selected": true,
- "locked": false,
- "visibility": true,
- "positionStatus": "defined"
},
{
- "id": "4",
- "label": "MarkupsClosedCurve-4",
- "description": "",
- "associatedNodeID": "vtkMRMLScalarVolumeNode1",
"position": [
-6980.525470132225,
5682.868336976309,
@@ -106,16 +75,8 @@
0,
1
],
- "selected": true,
- "locked": false,
- "visibility": true,
- "positionStatus": "defined"
},
{
- "id": "5",
- "label": "MarkupsClosedCurve-5",
- "description": "",
- "associatedNodeID": "vtkMRMLScalarVolumeNode1",
"position": [
-6735.759199784232,
5461.593738502808,
@@ -132,49 +93,8 @@
0,
1
],
- "selected": true,
],
- "selected": true,
- "locked": false,
- "visibility": true,
- "positionStatus": "defined"
}
],
- "display": {
- "visibility": true,
- "opacity": 1,
- "color": [
- 0.4,
- 1,
- 1
- ],
- "selectedColor": [
- 1,
- 0.5000076295109483,
- 0.5000076295109483
- ],
- "propertiesLabelVisibility": true,
- "pointLabelsVisibility": false,
- "textScale": 3,
- "glyphType": "Sphere3D",
- "glyphScale": 1,
- "glyphSize": 5,
- "useGlyphScale": true,
- "sliceProjection": false,
- "sliceProjectionUseFiducialColor": true,
- "sliceProjectionOutlinedBehindSlicePlane": false,
- "sliceProjectionColor": [
- 1,
- 1,
- 1
- ],
- "sliceProjectionOpacity": 0.6,
- "lineThickness": 0.2,
- "lineColorFadingStart": 1,
- "lineColorFadingEnd": 10,
- "lineColorFadingSaturation": 1,
- "lineColorFadingHueOffset": 0,
- "handlesInteractive": false,
- "snapMode": "toVisibleSurface"
- }
},
"orientation": [
0.9423723483536421,
|
Regarding simplifying the JSON that is saved, so long as it still captures all the important information then I support that. Not sure if I am well-qualified to say what all is important. I am okay with having some number of 'markup' entries under the 'markups' key. Say, how does this play into the request for multiple annotations per file #90 ? |
Great. All the keys I removed are related to colors and internal details specific to the application.
"markups": [
{
"markup": {
"coordinateSystem": "LPS",
"controlPoints": [
[...]
],
},
"orientation": [
0.9423723483536421,
-0.10260749019479301,
-0.3184431817677528,
[...]
1
],
"representationType": "spline",
"thickness": 50
}
], We will make sure to have each Questions:
|
1-based indexing would work for me. Is there any way for users to add their own text that goes along with an annotation?
I'm not too fussed either way. "annotations / annotation" probably has broader appeal to scientific users. Just let me know if you change it; I will have to make a matching change on my side. w.r.t to multiple annotations, I guess this is enabled now? #90 is just adding the GUI components so that users can manage more than one annotation? (e.g. "Now I am done modifying this annotation, and want to start adjusting the other one...") |
@allemangD and I were wondering if that would be useful. For now, annotation are called "Markup", "Markup-1", ... and user will have the possibility to edit the text directly within the list.
We will be using "annotation" and "annotations", looking at the presentation to the stakeholder as well as the poster presented to SFN, the word annotate and annotation is a recurrent theme. @danielsf Any suggestions ?
Yes, the infrastructure now nicely support that. @allemangD is currently finalizing the integration and it will be available in the next release. Likely middle of next week, but I will let David comment regarding the timeline. |
Yes, I think it should be ready for QA in the next couple days, and could be finalized by mid-next week. |
What's the problem this feature will solve?
It should be possible to start CellLocator from LIMS.
Describe the solution you'd like
CellLocator will be associated with a file extension (e.g
.cell-locator.allen-lims
). When double clicking on the file, cell locator would be started.Additional context
See also #21The text was updated successfully, but these errors were encountered: