Initial Query API

[computationdoc]

Create the initial HuBMAP query API endpoint. At a minimum we need an endpoint running in a Docker container that serves a single service such as GET /entity/....
-All entity types
-All entities of a type (returns uuids)
-Detailed info of an entity given uuid

[ngehlenborg]

@cborromeo @shirey @mruffalo: @mccalluc and I need some input on which entity types are currently implemented in the database and which ones will be available once this issue has been addressed. That is a dependency for #136 and #133.

[mccalluc]

Will the API be some separate application, or will it use the existing api end point in the existing Django app?

[computationdoc]

“THE API” is a collection of APIs which may be at different physical locations and written in different languages (a micro services architecture). In general there will be a number of stand-alone endpoints (applications one will call in essence), but most importantly the API Gateway will handle the security and “pass to other endpoints”. This question has arisen several times, its critical for folks to understand, and current sprint is focused on some initial dev of APIs, so I conclude that this will best be answered via demo and explanation on Friday’s Sprint Planning review of initial APIs… Finally, in terms of specific UI building, there may be “shortcuts” where the application has security privileges in which a framework like Django has internal APIs that the application is practically using, but inevitably those will be calling database APIs…so if the question is “what to write against for now” my answer is JSON files that are set to contain what you need so you can progress... JCS

…

On Oct 1, 2019, at 10:24 AM, Chuck McCallum ***@***.***> wrote: Will the API be some separate application, or will it use the existing api end point in the existing Django app? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#138?email_source=notifications&email_token=AFZNQX3Z3TXMLINWXB7KJLDQMNTUFA5CNFSM4IX7PK2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEABVMRQ#issuecomment-537089606>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFZNQX7C6TSQFG5RYDFA6Y3QMNTUFANCNFSM4IX7PK2A>.

[shirey]

@ngehlenborg @mccalluc At the top level our entity types are Donor, Sample, Dataset. Where the "sample" is tissue and has sub-types like: organ, block, biopsy, fresh frozen tissue, etc...

WRT: APIs to reiterate what Jonathan said, APIs don't need to exist within Django, we'll follow a micro-services architecture where WS endpoints can exist in multiple locations and can be implemented in multiple technologies.

[frdougal]

This represents an initial draft of the REST API calls. In the future, we can expand on these and return more data as we develop more use cases.

/entities/ - lists the entity types within the Neo4j system
example JSON response: {"entity_types" : ["Sample", "Donor", "Dataset"]}

/entities/<type> - returns a list of all the UUIDs in the system
example URL: /entities/donors - returns a list of all the donor UUIDs in the system
example JSON response: {"uuids" : ["11866f9586fdac34c95ac8a2737a83dd", "b2b59b227e0dfd7ddd4799f909bc3f01", "c3323cecca52e99113a56946a2bcdd5f", "44e00b44dda813cd39da6934c8954d71", "c0bfcf32a6da83c37a5ed7b2eeabdefe", "c83b24f93af72ffcee0dd94ad1d61d7a", "a41210275f08ec8b8b973a4f10a73e5f"]}

/entities/<uuid> - returns all the provenance and metadata associated with a particular entity
example URL: /entities/11866f9586fdac34c95ac8a2737a83dd
example JSON response: {
"provenance_create_timestamp": 1567520904428,
"provenance_user_email": "shirey@pitt.edu",
"provenance_group_name": "hubmap-testing",
"user_group_uuid": "5bd084c8-edc2-11e8-802f-0e368f3075e8",
"description": "Test Donor 1",
"label": "Bill Test 1",
"uuid": "11866f9586fdac34c95ac8a2737a83dd",
"provenance_user_displayname": "William Shirey",
"provenance_group_uuid": "5bd084c8-edc2-11e8-802f-0e368f3075e8",
"provenance_user_sub": "e19adbbb-73c3-43a7-b05e-0eead04f5ff8",
"provenance_modified_timestamp": 1567520904428
}

/entities/samples?sample-type=<sample-type> - returns a list of all sample UUIDs where tissue type is sample-type
example URL: /entities/samples?sample-type=organ
example JSON response: {"uuids" : ["ae6bf50908a4a2089d2eacd8e464fe86", "45ad98de3c9b5983a274d801969ae464", "37ef1d1d79c1e2a50ea6d18cdbb3762c", "f7c3c6a3ded8876226a673249b4a8423"]}
options for sample-type (case-insensitive):
Organ
Blood
Biopsy
Cell lysate
FFPE block
PFA Fixed frozen OCT block
Fixed tissue piece
Flash frozen, liquid nitrogen
Formalin fixed OCT block
Fresh frozen tissue
Fresh frozen oct block
Fresh tissue
Frozen cell pellet (Buffy coat)
Module
PBMC
Plasma
Nuclei RNAlater
Organ Piece
RNAlater treated and stored
Segment
Serum
Single cell cryopreserved
CLARITY hydrogel
Cryosections/curls from fresh frozen OCT
Cryosections/curls RNAlater
FFPE slide
Fixed Frozen section slide
Fresh Frozen section slide
gDNA
Nuclei
Protein
RNA, total
ATACseq
CODEX
RNAseq
scATACseq
scRNAseq
seqFISH
snATACseq
snRNAseq
WES

/entities/samples?organ-type=<organ-type> - returns a list of all sample UUIDs where tissue type is organ-type
example URL: /entities/samples?organ-type=liver
example JSON response: {"uuids" : ["ae6bf50908a4a2089d2eacd8e464fe86", "37ef1d1d79c1e2a50ea6d18cdbb3762c"]}
options for organ-type (case-insensitive):
Bladder
Brain
Kidney (Right)
Kidney (Left)
Heart
Large Intestine
Small Intestine
Left Lung
Right Lung
Lymph Node
Spleen
Thymus
Ureter
Liver
Other

/entities/donors?created-by=<user-email> - returns a list of all donor UUIDs that were registered by the user-email address
example URL: /entities/donors?created-by=shirey@pitt.edu
example JSON response: {"uuids" : ["11866f9586fdac34c95ac8a2737a83dd", "b2b59b227e0dfd7ddd4799f909bc3f01", "c3323cecca52e99113a56946a2bcdd5f", "44e00b44dda813cd39da6934c8954d71", "c0bfcf32a6da83c37a5ed7b2eeabdefe", "c83b24f93af72ffcee0dd94ad1d61d7a", "a41210275f08ec8b8b973a4f10a73e5f"]}

[ngehlenborg]

Thanks @cborromeo, that is helpful. Obviously not for now, but something that we will need in the future is to replace the organ options and the sample options with ontology terms.

[mccalluc]

Thank you! This is what I've wanted to see.

[mruffalo]

I agree with @shirey that APIs don't need to be implemented in this Django application, but the API-centric Django app at hubmap/dataportal is a reasonable place. There is a nonzero cost to spinning up an entirely new project for a service, tracking another set of dependencies for security issues, adding new containers to any runtime configuration in development and production, etc.

I'd propose implementing APIs in this project if they're going to be written in Python, unless there's a very good reason not to.

[computationdoc]

Yes, thank you for pointing this out. I understand that it will be an add rather than replace (use of display names or ontology terms)…I have been hesitant to push that so far because there is this “build an ontology” effort going on at Indiana whereas I firmly believe we should pick from UMLS and where needed add new terms/codes/CUIs to a UMLS vocabulary…so, that needs to be resolved, or we could have three (display, UMLS CUI and HuBMAP ontology)…my gut feeling is I should start assigning UMLS CUIs to each display name soon...what are your thoughts?

…

On Oct 1, 2019, at 1:04 PM, Nils Gehlenborg ***@***.***> wrote: Thanks @cborromeo <https://github.com/cborromeo>, that is helpful. Obviously not for now, but something that we will need in the future is to replace the organ options and the sample options with ontology terms. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#138?email_source=notifications&email_token=AFZNQX4RF5AELYVF2CK7VUDQMOGMPA5CNFSM4IX7PK2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEACGDPA#issuecomment-537158076>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFZNQX3HCGHC5F6Z7JYAHEDQMOGMPANCNFSM4IX7PK2A>.

[mccalluc]

"Done" -- Closing.