Grouping: CRUD: Concept set version

[Sigfried]

Overview

Concept set version create/updates should handle (i) metadata fields, (ii) individual concept expression items, and (iii) should be able to handle multiple versions.

Sub-tasks

Creates

• Allow for other users (@amin): Amin needs to change permissions so that other users, using their TOKENs, I believe, can make calls.
• Frontend
  • For CSV
  • For JSON
• Backend: enclave wrangler
  • For CSV
  • For JSON
  • Allow for multiple versions
• Backend: fast api routes
  • For CSV
  • For JSON
• Unit tests
  • For CSV
  • For JSON

Updates

This would be updating existing drafts.

Delete

Needed?

Read

Have?

Additional

• CSV upload: Add option: Leave out parent_version_codeset_id? #246
• CSV Upload: Support existing use cases #247

Documentation

In addition to tests and code comments. Where? Enclave wrangler? Developer docs?

[joeflack4]

Ah, this is what you were talking about on Teams. I think I already have laid the groundwork for this. I have 2 routes already: (i) for new concept set version w/ concept changes (or other metadata changes too, I think), and (ii) one for a new container. I think I just need to improve the robustness of that. The biggest improvement involves committing these changes to git in a stable, consistent manner. However, since we are moving to RDBMS, I think I might want to hold off on that until after RDBMS because the nature of how I do these commits (e.g. in batch) might change. But it might not either.

[Sigfried]

I agree -- RDBMS before worrying more about git persistence.

With the enclave update endpoints -- do you think it's ready for me to start writing a front end interface? Will they allow us to post on behalf of other users? And do they allow us to save actual versions, or just drafts that then have to be saved to versions in the concept set browser?

[joeflack4]

@Sigfried I think a lot of this was in the recent pull request I made, but we didn't have time to do a review. From what I remember: (i) for a new container, it also creates a 1st draft and finalizes it (you may want to try it and double check), (ii) for a new version, it should indeed create the version draft and also finalize it immediately. I think that the routes I wrote, as well as the enclave functions that do this, talk about this in the docstring, but not 100% sure.

For these routes/features specifically, I thought that a unit test would be prudent. But that has been precluded so far because I anticipate some extra time to work with the correct DELETE endpoint(s) for the teardown step.

[Sigfried]

finalize-draft-omop-concept-set-version/validate failing during call to route_upload_new_cset_version_with_concepts:

curl  -H "Content-type: application/json" \
            -H "Authorization: Bearer $PALANTIR_ENCLAVE_AUTHENTICATION_BEARER_TOKEN" \
            https://unite.nih.gov/api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/actions/create-new-draft-omop-concept-set-version/validate \
            --data '{"parameters": {"conceptSet": "[DM]Type2 Diabetes Mellitus", "intention": "", "provenance": "Created through TermHub.", "annotation": "Curated value set: [DM]Type2 Diabetes Mellitus", "intended-research-project": "RP-4A9E27", "versionId": 1000083089, "on-behalf-of": "5c560c3e-8e55-485c-9a66-f96285f273a0"}}'
            

curl  -H "Content-type: application/json" \
            -H "Authorization: Bearer $PALANTIR_ENCLAVE_AUTHENTICATION_BEARER_TOKEN" \
            https://unite.nih.gov/api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/actions/create-new-draft-omop-concept-set-version/apply \
            --data '{"parameters": {"conceptSet": "[DM]Type2 Diabetes Mellitus", "intention": "", "provenance": "Created through TermHub.", "annotation": "Curated value set: [DM]Type2 Diabetes Mellitus", "intended-research-project": "RP-4A9E27", "versionId": 1000083089, "on-behalf-of": "5c560c3e-8e55-485c-9a66-f96285f273a0"}}'
            

curl  -H "Content-type: application/json" \
            -H "Authorization: Bearer $PALANTIR_ENCLAVE_AUTHENTICATION_BEARER_TOKEN" \
            https://unite.nih.gov/api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/actions/finalize-draft-omop-concept-set-version/validate \
            --data '{"parameters": {"new-parameter1": "hello new-parameter1", "concept-set-container": "[DM]Type2 Diabetes Mellitus", "version": 1000083089, "new-parameter": "hello new-parameter", "on-behalf-of": "5c560c3e-8e55-485c-9a66-f96285f273a0"}}'
            
Unexpected <class 'requests.exceptions.HTTPError'>: 400 Client Error: Bad Request for url: https://unite.nih.gov/api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/actions/finalize-draft-omop-concept-set-version/validate
Traceback (most recent call last):
  File "/Users/sigfried/git-repos/TermHub/enclave_wrangler/new_enclave_api.py", line 543, in make_request
    raise err
  File "/Users/sigfried/git-repos/TermHub/enclave_wrangler/new_enclave_api.py", line 540, in make_request
    response.raise_for_status()
  File "/Users/sigfried/git-repos/TermHub/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://unite.nih.gov/api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/actions/finalize-draft-omop-concept-set-version/validate

@joeflack4: Do you recall how to get the documentation for an api call? Something is wrong with this call to finalize.

[joeflack4]

Just sent you a message over teams. This may show the query to get endpoint definitions, but if not, I can find out how: https://unite.nih.gov/documentation/static/foundry/api/ontology-resources/actions/list-action-types/

Here is for the specific endpoint, from my pesonal notes:

    {
      "apiName": "finalize-draft-omop-concept-set-version",
      "description": "",
      "rid": "ri.actions.main.action-type.d53c0a2b-db9d-4a72-b10b-ad5f467f3f9c",
      "parameters": {
        "new-parameter1": {
          "description": "",
          "baseType": "String"
        },
        "concept-set-container": {
          "description": "",
          "baseType": "OntologyObject"
        },
        "version": {
          "description": "",
          "baseType": "OntologyObject"
        },
        "currentMaxVersion": {
          "description": "",
          "baseType": "Double"
        },
        "new-parameter": {
          "description": "",
          "baseType": "String"
        }
      }
    },

I remember we were using something called OTHER_TOKEN; not sure if that is your issue here.