Feature/user-api

[Rutvikrj26]

This Pull request introduces an API for Published Projects.

There are currently two paths :

1. /projects/api/v1/published : Provides a list of all the published projects (id, title, abstract)
2. /projects/api/v1/published/<project-slug>/<project-version> : Provides all MetaData for that specific Published Project.

The API is available without Authentication.

Pending :

• Discussion over Auth Implementation
• Dua & License field updates

[tompollard]

Thanks @Rutvikrj26! I've added a few comments inline. There are also a few styling issues that need fixing for the tests to pass. Please could you take a look?

[tompollard]

Example output for http://localhost:8000/projects/api/v1/published

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": 1,
        "title": "Demo eICU Collaborative Research Database",
        "abstract": "<p>The eICU Collaborative Research Database is a large multi-center critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology.</p>"
    },
    {
        "id": 2,
        "title": "Demo ECG Signal Toolbox",
        "abstract": "<p>The major components of the toolbox are the library, about 75 applications for signal processing and automated analysis.</p>"
    },
    {
        "id": 3,
        "title": "Demo 2018 BHI and BSN Data Challenge",
        "abstract": "<p>In collaboration with the IEEE Conference on Biomedical and Health Informatics (BHI) 2018 and the IEEE Conference on Body Sensor Networks (BSN), we are hosting a challenge to explore real clinical questions in critically ill patients using the MIMIC-III database. Participants in the challenge will be invited the present at the BHI &amp; BSN Annual Conference in Las Vegas, USA (4-7 March 2018):&nbsp;<a href=\"[https://bhi-bsn.embs.org/2018/\](https://bhi-bsn.embs.org/2018/%5C)">https://bhi-bsn.embs.org/2018/</a></p>"
    },
    {
        "id": 4,
        "title": "Self Managed Access Database Demo",
        "abstract": "<p>A dummy project to demonstrate a database project with self managed access</p>"
    },
    {
        "id": 5,
        "title": "Demo software for parsing clinical notes",
        "abstract": "<p>Demo abstract</p>"
    },
    {
        "id": 20,
        "title": "Demo for visualizing waveforms",
        "abstract": "<p>Here we add multiple different waveform files to test visualizers.</p>"
    }
]

Example output for http://localhost:8000/projects/api/v1/published/1

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 1,
    "title": "Demo eICU Collaborative Research Database",
    "abstract": "<p>The eICU Collaborative Research Database is a large multi-center critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology.</p>",
    "background": "<p>The <a href=\"[http://www.usa.philips.com/healthcare/product/HCNOCTN503/eicu-program-telehealth-for-the-intensive-care-unit\](http://www.usa.philips.com/healthcare/product/HCNOCTN503/eicu-program-telehealth-for-the-intensive-care-unit%5C)">Philips eICU program</a> is a transformational critical care telehealth program that delivers need-to-know information to caregivers, empowering them to care for the patients. It is a supplement &mdash; not a replacement &mdash; to the bedside team, and the data utilized by the remote care givers is archived for research purposes.</p>\r\n\r\n<p>Through this work, we have generated a large database which has potential for facilitating additional research initiatives on patient outcomes, trends, and other best practice protocols in use today at most healthcare facilities. The Philips eICU Research Institute (eRI), which maintains the data, has generously contributed the eICU Collaborative Research Database described here.</p>",
    "methods": "<p>The eICU Collaborative Research Database is populated with data from a combination of many critical care units throughout the continental United States. The data in the collaborative database covers patients who were admitted to critical care units in 2014 and 2015.</p>",
    "content_description": "<p>Identifiers are used across the database to identify unique concepts such as patients, hospitals, ICU stays, and so on. These identifiers include:</p>\r\n\r\n<ul>\r\n\t<li>hospitalid - which uniquely identifies each hospital in the database.</li>\r\n\t<li>uniquepid - uniquely identifies patients (i.e. it is always the same value for the same person)</li>\r\n\t<li>patienthealthsystemsstayid - uniquely identifies hospitals stays</li>\r\n\t<li>patientunitstayid - uniquely identifies unit stays (usually the unit is an ICU within a hospital)</li>\r\n</ul>\r\n\r\n<p>Almost all tables use patientunitstayid as the primary identifier.</p>",
    "usage_notes": "<p>Data from each patient is collected into a common warehouse only if certain &ldquo;interfaces&rdquo; are available. Each interface is used to transform and load a certain type of data: vital sign interfaces incorporate vital signs, laboratory interfaces provide measurements on blood samples, and so on. It is important to be aware that different care units may have different interfaces in place, and that the lack of an interface will result in no data being available for a given patient, even if those measurements were made in reality.</p>\r\n\r\n<p><strong>Inputs and Outputs</strong></p>\r\n\r\n<ul>\r\n\t<li>The medication table is essentially an interface to pharmacy data - i.e. prescribed medications.</li>\r\n\t<li>The intakeoutput and infusiondrug tables should be used for fluids and drugs, respectively. It is a challenge to decide whether a hospital is actually collecting and archiving data in the infusiondrug table.</li>\r\n</ul>\r\n\r\n<p><strong>Laboratory tests</strong></p>\r\n\r\n<ul>\r\n\t<li>The lab table is populated by ~160 &ldquo;standard&rdquo; laboratory measurements. When a hospital first participates in the eICU program, they must map these values to their local system. As a result, most common labs are well harmonized in this table. However, it is possible for the lab interface to be down and for standard labs to be recorded in the customlab table (e.g. during software upgrades). These downtimes are in principle rare, but an empirical study on how frequently this occurs is yet to be undertaken.</li>\r\n</ul>",
    "installation": "",
    "acknowledgements": "<p>The eICU Collaborative Research Database is made available largely through the work of Philips Healthcare and collaborators at MIT Laboratory for Computational Physiology. If you use the data, code or algorithms, please provide a citation to this project.</p>",
    "conflicts_of_interest": "<p>None</p>",
    "version": "2.0.0",
    "release_notes": "",
    "short_description": "",
    "access_policy": 2,
    "project_home_page": "https://eicu-crd.mit.edu/",
    "allow_file_downloads": true,
    "ethics_statement": "The authors declare no ethics concerns.",
    "submission_datetime": "2018-11-04T16:09:38.666000-05:00",
    "author_comments": "",
    "editor_assignment_datetime": "2018-11-09T16:09:54.565000-05:00",
    "revision_request_datetime": "2018-11-11T16:10:19.855000-05:00",
    "resubmission_datetime": "2018-11-12T16:10:24.665000-05:00",
    "editor_accept_datetime": "2018-11-14T16:10:49.361000-05:00",
    "copyedit_completion_datetime": "2018-11-17T16:11:00.997000-05:00",
    "author_approval_datetime": "2018-12-01T16:11:11.781000-05:00",
    "creation_datetime": "2018-12-14T16:11:17.040000-05:00",
    "version_order": 0,
    "main_storage_size": 3195,
    "compressed_storage_size": 3090,
    "incremental_storage_size": 3195,
    "publish_datetime": "2018-12-17T16:11:17.040000-05:00",
    "has_other_versions": false,
    "deprecated_files": false,
    "doi": "10.13026/G2F309",
    "slug": "demoeicu",
    "is_legacy": false,
    "full_description": "",
    "is_latest_version": true,
    "featured": 1,
    "has_wfdb": false,
    "display_publications": true,
    "resource_type": 0,
    "license": 6,
    "dua": 3,
    "core_project": "66951d55-e473-4a98-a24f-4fad05ed967b",
    "editor": 1,
    "parent_projects": [],
    "programming_languages": [],
    "required_trainings": [
        1
    ]
}

[tompollard]

I wonder if we should be using a look up table/dictionary to map variables from the models to the API data? (e.g. we could map "dua" on the model to "data_use_agreement" in the API data). This would (1) allow us to choose more meaningful names than those on the models and (2) protect the API from changes to the models.

[alistairewj]

Example output for http://localhost:8000/projects/api/v1/published

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": 1,
        "title": "Demo eICU Collaborative Research Database",
        "abstract": "<p>The eICU Collaborative Research Database is a large multi-center critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology.</p>"
    }, ...

-> slug/version instead of id

Example output for http://localhost:8000/projects/api/v1/published/1

Only output these fields:

"title", "abstract", "background", "methods", "content_description", "usage_notes", "installation", "acknowledgements", "conflicts_of_interest", "release_notes", "ethics_statement", "version",  "short_description", "project_home_page", "publish_datetime", "doi", "slug"

I think "license" and "dua" would be nice to include in some way, but not as integers as they are.

[tompollard]

I think "license" and "dua" would be nice to include in some way, but not as integers as they are.

I was thinking the same thing. Fixing some of the Enum types that we use for the class models would probably be a first step for this. The Django enum type (see: https://docs.djangoproject.com/en/4.1/ref/models/fields/#enumeration-types) supports human readable names, which is a nice feature. For some reason, AccessPolicy currently uses IntEnum:

physionet-build/physionet-django/project/modelcomponents/access.py

Lines 16 to 30 in 141d2a9

	class AccessPolicy(IntEnum):
	OPEN = 0
	RESTRICTED = 1
	CREDENTIALED = 2
	CONTRIBUTOR_REVIEW = 3
	do_not_call_in_templates = True
	@classmethod
	def choices(cls, gte_value=0):
	return tuple(
	(option.value, option.name.replace("_", " ").title())
	for option in cls if option.value >= gte_value
	)

[Rutvikrj26]

Current API Calls & their Responses :
Call : http://127.0.0.1:8000/projects/api/v1/published
Response :

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": 1,
        "title": "Demo eICU Collaborative Research Database",
        "abstract": "<p>The eICU Collaborative Research Database is a large multi-center critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology.</p>",
        "license": {
            "name": "PhysioNet Credentialed Health Data License 1.5.0",
            "is_active": true
        },
        "dua": {
            "name": "PhysioNet Credentialed Health Data Use Agreement",
            "is_active": true
        }
    },
    {
        "id": 2,
        "title": "Demo ECG Signal Toolbox",
        "abstract": "<p>The major components of the toolbox are the library, about 75 applications for signal processing and automated analysis.</p>",
        "license": {
            "name": "Creative Commons Attribution 4.0 International Public License",
            "is_active": true
        },
        "dua": null
    },
    {
        "id": 3,
        "title": "Demo 2018 BHI and BSN Data Challenge",
        "abstract": "<p>In collaboration with the IEEE Conference on Biomedical and Health Informatics (BHI) 2018 and the IEEE Conference on Body Sensor Networks (BSN), we are hosting a challenge to explore real clinical questions in critically ill patients using the MIMIC-III database. Participants in the challenge will be invited the present at the BHI &amp; BSN Annual Conference in Las Vegas, USA (4-7 March 2018):&nbsp;<a href=\"[https://bhi-bsn.embs.org/2018/\](https://bhi-bsn.embs.org/2018/%5C)">https://bhi-bsn.embs.org/2018/</a></p>",
        "license": {
            "name": "MIT License",
            "is_active": true
        },
        "dua": null
    },
    {
        "id": 4,
        "title": "Self Managed Access Database Demo",
        "abstract": "<p>A dummy project to demonstrate a database project with self managed access</p>",
        "license": {
            "name": "Creative Commons Zero 1.0 Universal Public Domain Dedication",
            "is_active": true
        },
        "dua": {
            "name": "PhysioNet Contributor Review Health Data Use Agreement",
            "is_active": true
        }
    },
    {
        "id": 5,
        "title": "Demo software for parsing clinical notes",
        "abstract": "<p>Demo abstract</p>",
        "license": {
            "name": "GNU General Public License version 3",
            "is_active": true
        },
        "dua": null
    },
    {
        "id": 20,
        "title": "Demo for visualizing waveforms",
        "abstract": "<p>Here we add multiple different waveform files to test visualizers.</p>",
        "license": {
            "name": "Open Data Commons Attribution License v1.0",
            "is_active": true
        },
        "dua": null
    }
]

Call : http://127.0.0.1:8000/projects/api/v1/published/demowave/1.0.0

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "title": "Demo for visualizing waveforms",
    "abstract": "<p>Here we add multiple different waveform files to test visualizers.</p>",
    "background": "<p>These waveforms have been extracted from multiple sources and de-identified.&nbsp;</p>",
    "methods": "<p>Conversion to WFDB format was done for most of the waveforms though some are also in EDF and WAV format.</p>",
    "content_description": "<p>wave_1: WAV (15 channels)</p>\n\n<p>wave_2: WFDB (2 channels)</p>\n\n<p>wave_3: WFDB (10 channels)</p>\n\n<p>wave_4: EDF (12 channels)</p>",
    "usage_notes": "<p>Visualize the waveforms with any of the available waveforms viewers.</p>",
    "installation": "",
    "acknowledgements": "",
    "conflicts_of_interest": "<p>None.</p>",
    "release_notes": "",
    "ethics_statement": "The authors declare no ethics concerns.",
    "version": "1.0.0",
    "short_description": "Initial Release",
    "project_home_page": "",
    "publish_datetime": "2020-12-23T11:18:00.509000-05:00",
    "doi": null,
    "slug": "demowave"
}

[Rutvikrj26]

Final API Calls :

Call : http://127.0.0.1:8000/projects/api/v1/published
Respponse :

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": 1,
        "title": "Demo eICU Collaborative Research Database",
        "abstract": "<p>The eICU Collaborative Research Database is a large multi-center critical care database made available by Philips Healthcare in partnership with the MIT Laboratory for Computational Physiology.</p>",
        "license": {
            "name": "PhysioNet Credentialed Health Data License 1.5.0"
        },
        "dua": {
            "name": "PhysioNet Credentialed Health Data Use Agreement"
        }
    },
    {
        "id": 2,
        "title": "Demo ECG Signal Toolbox",
        "abstract": "<p>The major components of the toolbox are the library, about 75 applications for signal processing and automated analysis.</p>",
        "license": {
            "name": "Creative Commons Attribution 4.0 International Public License"
        },
        "dua": null
    },
    {
        "id": 3,
        "title": "Demo 2018 BHI and BSN Data Challenge",
        "abstract": "<p>In collaboration with the IEEE Conference on Biomedical and Health Informatics (BHI) 2018 and the IEEE Conference on Body Sensor Networks (BSN), we are hosting a challenge to explore real clinical questions in critically ill patients using the MIMIC-III database. Participants in the challenge will be invited the present at the BHI &amp; BSN Annual Conference in Las Vegas, USA (4-7 March 2018):&nbsp;<a href=\"[https://bhi-bsn.embs.org/2018/\](https://bhi-bsn.embs.org/2018/%5C)">https://bhi-bsn.embs.org/2018/</a></p>",
        "license": {
            "name": "MIT License"
        },
        "dua": null
    },
    {
        "id": 4,
        "title": "Self Managed Access Database Demo",
        "abstract": "<p>A dummy project to demonstrate a database project with self managed access</p>",
        "license": {
            "name": "Creative Commons Zero 1.0 Universal Public Domain Dedication"
        },
        "dua": {
            "name": "PhysioNet Contributor Review Health Data Use Agreement"
        }
    },
    {
        "id": 5,
        "title": "Demo software for parsing clinical notes",
        "abstract": "<p>Demo abstract</p>",
        "license": {
            "name": "GNU General Public License version 3"
        },
        "dua": null
    },
    {
        "id": 20,
        "title": "Demo for visualizing waveforms",
        "abstract": "<p>Here we add multiple different waveform files to test visualizers.</p>",
        "license": {
            "name": "Open Data Commons Attribution License v1.0"
        },
        "dua": null
    }
]

Call : http://127.0.0.1:8000/projects/api/v1/published/demowave/1.0.0
Response :

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "title": "Demo for visualizing waveforms",
    "abstract": "<p>Here we add multiple different waveform files to test visualizers.</p>",
    "version": "1.0.0",
    "short_description": "Initial Release",
    "project_home_page": "",
    "publish_datetime": "2020-12-23T11:18:00.509000-05:00",
    "doi": null,
    "slug": "demowave"
}

[tompollard]

Thanks @Rutvikrj26, I added some minor comments inline, but overall this looks good to me.

One issue is that there is no way to identify the versions using the API. http://127.0.0.1:8000/projects/api/v1/published does not list project version.

This means that there is no clear way for someone to discover the project path.

For example, how would someone find the following URL: http://127.0.0.1:8000/projects/api/v1/published/demowave/1.0.0

[bemoody]

I still don't know what the application is here. Can you give a specific, concrete example?

What is it you're trying to do? Why are the existing views in export/ not sufficient for that purpose?

[Rutvikrj26]

@bemoody
What are you referring to in the comment when you say - "I Stilll don't know what the application is here"
I think, you meant the get_object, which is removed.
Please let me know if it is something else you are referring to.

[tompollard]

@Rutvikrj26 we caught up on this pull request today. The outcome was that we would like to combine your changes with the export app: https://github.com/MIT-LCP/physionet-build/tree/dev/physionet-django/export

Please could you move your changes into the export app using whichever aspects of each implementation you prefer? In general I would suggest defaulting to your code.

All of the current views at https://github.com/MIT-LCP/physionet-build/blob/dev/physionet-django/export/views.py can be removed.

[tompollard]

I still don't know what the application is here. Can you give a specific, concrete example?

@bemoody we have had several email requests from people who would like access to an API that allows them to search for project content using an API. One example is a group who maintain an index of datasets that might be of interest to their research staff. They would like to be able to populate their search index using an API such as the one implemented here.

Our goal is to introduce an API with very simple functionality and then build it out as we have a clearer understanding of the different uses cases.

[tompollard]

This looks good to me, except that Benjamin has found a dependency on the database-list function in the wfdb-python package. See: https://github.com/MIT-LCP/wfdb-python/blob/14248d17e87f3392ab87a30dc03013233414a3db/wfdb/io/download.py#L181-L210

def get_dbs():
    """
    Get a list of all the PhysioNet databases available.

    Parameters
    ----------
    N/A

    Returns
    -------
    dbs : list
        All of the databases currently available for analysis.

    Examples
    --------
    >>> dbs = wfdb.get_dbs()
    >>> dbs
    [
     ['aami-ec13', 'ANSI/AAMI EC13 Test Waveforms'],
     ['adfecgdb', 'Abdominal and Direct Fetal ECG Database'],
     ...
     ['wrist', 'Wrist PPG During Exercise']
    ]

    """
    with _url.openurl("https://physionet.org/rest/database-list/", "rb") as f:
        content = f.read()
    dbs = json.loads(content)
    dbs = [[d["slug"], d["title"]] for d in dbs]
    dbs.sort()


    return dbs

Is it possible to reinstate this view and URL temporarily? We can then work towards replacing it in the wfdb-python package and subsequent deprecation.

[Rutvikrj26]

Sure!
I will add the view and the URL for the Database List back into the export.
Thanks for pointing out the dependency!

Note :

Current API call : http://127.0.0.1:8000/api/v1/published

• api/ is a part of the main project URLs file, and the rest v1/published is in the application URLs file.

I am adding the temporary Database list with the same original /rest/database-list in the main project URLs file.

[bemoody]

Don't remove TEST_DEFAULTS from physionet-django/export/urls.py. Instead, define some useful test parameters, e.g.

TEST_DEFAULTS = {
    'slug': 'demoeicu',
    'version': '2.0.0',
}

[bemoody]

(It might be better to rename the URL parameter 'slug' to 'project_slug' for clarity and consistency with the rest of the site.)

[tompollard]

Thanks @Rutvikrj26, overall this looks good to me. I think there are several improvements that we should make, but this gives us a good starting point.

e.g.

• http://localhost:8000/api/v1/published: we probably don't want to be giving out the primary key for projects ("id": 1,)
• http://localhost:8000/api/v1/published: it isn't clear to me why "api" is one of the "format" options (http://localhost:8000/api/v1/published?format=api)
• The user should be able to request a list of different versions (e.g. we could display them at http://localhost:8000/api/v1/published). Right now there is no way of someone finding the URL path for a project (e.g. how would I know to look at: http://localhost:8000/api/v1/published/demowave/1.0.0)?