added flatten_nested_phenotype function

[ewartj]

Overview and purpose

Adds a function to flatten a nested JSON. This is then used in get_phenotype_stataistics() to turn the flattened json into a pandas dataframe (this part is untested as there is no nested data on the dev CloudOs).

Ideally this function would be called __flatten_nested_phenotype() but not having the __ was better for testing.

• Adds flatten_nested_phenotype method to flatten a nested json into a flat dictionary.

How to test

Install from Github. Python >= 3.8 and pip are required.
Clone the repo and install it using pip:

git clone https://github.com/lifebit-ai/cloudos-cli
cd cloudos-cli
git checkout flatten_nested_json
pip install -r requirements.txt
pip install -e .

Testing

The JSON input needed for testing is below in the dropdown. It should be saved as nested_list.json in the root directory of cloudos-cli.

nested json

[
  {
    "count": 83,
    "total": 1189409,
    "coding": "Chapter B",
    "label": "Chapter B - Endocrine System and Breast",
    "selectable": false,
    "children": [ 
      {
        "count": 0,
        "total": 1189409,
        "coding": "B01",
        "label": "B01 Excision of pituitary gland",
        "selectable": false,
        "children": [
          {
            "count": 0,
            "total": 1189409,
            "coding": "B011",
            "label": "B01.1 Transethmoidal hypophysectomy",
            "selectable": true,
            "children": []
          },
          {
            "count": 0,
            "total": 1189409,
            "coding": "B012",
            "label": "B01.2 Trans-sphenoidal hypophysectomy",
            "selectable": true,
            "children": []
          }
        ]
      },
      {
        "count": 7,
        "total": 1189409,
        "coding": "B08",
        "label": "B08 Excision of thyroid gland",
        "selectable": false,
        "children": [
          {
            "count": 2,
            "total": 1189409,
            "coding": "B081",
            "label": "B08.1 Total thyroidectomy",
            "selectable": true,
            "children": []
          },
          {
            "count": 0,
            "total": 1189409,
            "coding": "B082",
            "label": "B08.2 Subtotal thyroidectomy",
            "selectable": true,
            "children": []
          },
          {
            "count": 5,
            "total": 1189400,
            "coding": "B083",
            "label": "B08.3 Hemithyroidectomy",
            "selectable": true,
            "children": []
          }
        ]
      }
    ]
  },
  {
    "count": 893,
    "total": 1189409,
    "coding": "Chapter D",
    "label": "Chapter D - Ear",
    "selectable": false,
    "children": [
      {
        "count": 6,
        "total": 1189409,
        "coding": "D01",
        "label": "D01 Excision of external ear",
        "selectable": false,
        "children": [
          {
            "count": 0,
            "total": 1189409,
            "coding": "D011",
            "label": "D01.1 Total excision of external ear",
            "selectable": true,
            "children": []
          },
          {
            "count": 4,
            "total": 1189409,
            "coding": "D012",
            "label": "D01.2 Partial excision of external ear",
            "selectable": true,
            "children": []
          },
          {
            "count": 2,
            "total": 1189409,
            "coding": "D013",
            "label": "D01.3 Excision of preauricular abnormality",
            "selectable": true,
            "children": []
          }
        ]
      },
      {
        "count": 345,
        "total": 1189409,
        "coding": "D02",
        "label": "D02 Extairpation otf lesion oft external ear",
        "selectable": false,
        "children": [
          {
            "count": 123,
            "total": 1189409,
            "coding": "D021",
            "label": "D02.1 Excision of lesion of external ear",
            "selectable": true,
            "children": []
          },
          {
            "count": 2,
            "total": 1189409,
            "coding": "D022",
            "label": "D02.2 Destruction of lesion of external ear",
            "selectable": true,
            "children": []
          },
          {
            "count": 12,
            "total": 1189409,
            "coding": "D023",
            "label": "D02.3 Other specified extirpation of lesion of external ear",
            "selectable": true,
            "children": []
          }
        ]
      }
    ]
  }
]

>>> from cloudos.cohorts import Cohort
>>> apikey = "***"
>>> workspace_id = "5f7c8696d6ea46288645a89f"
>>> cloudos_url = "http://cohort-browser-dev-110043291.eu-west-1.elb.amazonaws.com"
>>> cohort_id = "62e8cf572e15235d1709f36a"

>>> import json
>>> json_file = "nested_list.json"
>>> with open(json_file) as f:
...     jsn = json.load(f)
... 
>>> c = Cohort(apikey, cloudos_url, workspace_id, cohort_id)
>>> flat_json = c.flatten_nested_phenotype(jsn)
>>> print(flat_json)
[{'full_path': ['Chapter B'], 'id': 'Chapter B', 'value': 'Chapter B - Endocrine System and Breast', 'count': 83}, {'full_path': ['Chapter B', 'B01'], 'id': 'B01', 'value': 'B01 Excision of pituitary gland', 'count': 0}, {'full_path': ['Chapter B', 'B01', 'B011'], 'id': 'B011', 'value': 'B01.1 Transethmoidal hypophysectomy', 'count': 0}, {'full_path': ['Chapter B', 'B01', 'B012'], 'id': 'B012', 'value': 'B01.2 Trans-sphenoidal hypophysectomy', 'count': 0}, {'full_path': ['Chapter B', 'B08'], 'id': 'B08', 'value': 'B08 Excision of thyroid gland', 'count': 7}, {'full_path': ['Chapter B', 'B08', 'B081'], 'id': 'B081', 'value': 'B08.1 Total thyroidectomy', 'count': 2}, {'full_path': ['Chapter B', 'B08', 'B082'], 'id': 'B082', 'value': 'B08.2 Subtotal thyroidectomy', 'count': 0}, {'full_path': ['Chapter B', 'B08', 'B083'], 'id': 'B083', 'value': 'B08.3 Hemithyroidectomy', 'count': 5}, {'full_path': ['Chapter D'], 'id': 'Chapter D', 'value': 'Chapter D - Ear', 'count': 893}, {'full_path': ['Chapter D', 'D01'], 'id': 'D01', 'value': 'D01 Excision of external ear', 'count': 6}, {'full_path': ['Chapter D', 'D01', 'D011'], 'id': 'D011', 'value': 'D01.1 Total excision of external ear', 'count': 0}, {'full_path': ['Chapter D', 'D01', 'D012'], 'id': 'D012', 'value': 'D01.2 Partial excision of external ear', 'count': 4}, {'full_path': ['Chapter D', 'D01', 'D013'], 'id': 'D013', 'value': 'D01.3 Excision of preauricular abnormality', 'count': 2}, {'full_path': ['Chapter D', 'D02'], 'id': 'D02', 'value': 'D02 Extairpation otf lesion oft external ear', 'count': 345}, {'full_path': ['Chapter D', 'D02', 'D021'], 'id': 'D021', 'value': 'D02.1 Excision of lesion of external ear', 'count': 123}, {'full_path': ['Chapter D', 'D02', 'D022'], 'id': 'D022', 'value': 'D02.2 Destruction of lesion of external ear', 'count': 2}, {'full_path': ['Chapter D', 'D02', 'D023'], 'id': 'D023', 'value': 'D02.3 Other specified extirpation of lesion of external ear', 'count': 12}]