get_hapi_metadata, get_hapi_results now return dictionaries #23

mcarans · 2023-09-18T01:38:12Z

get_hapi_results now returns a dictionary that looks like what is below. The results dictionary for the hapi-pipelines would have keys adminone and admintwo (the names passed to _create_configurable_scrapers in pipelines.py).

{
            "hdx_id": "6a41be98-75b9-4365-9ea3-e33d0dd2668b",
            "hdx_stub": "global-school-closures-covid19",
            "provider_code": "18f2d467-dcf8-4b7e-bffa-b3c338ba3a7c",
            "provider_name": "unesco",
            "reference_period": {
                "enddate": datetime(
                    2022, 4, 30, 23, 59, 59, tzinfo=timezone.utc
                ),
                "enddate_str": "2022-04-30T23:59:59+00:00",
                "ongoing": False,
                "startdate": datetime(2020, 2, 16, 0, 0, tzinfo=timezone.utc),
                "startdate_str": "2020-02-16T00:00:00+00:00",
            },
            "resource": {
                "download_url": "https://data.humdata.org/dataset/6a41be98-75b9-4365-9ea3-e33d0dd2668b/resource/3b5baa74-c928-4cbc-adba-bf543c5d3050/download/covid_impact_education.csv",
                "filename": "School Closures",
                "format": "csv",
                "hdx_id": "3b5baa74-c928-4cbc-adba-bf543c5d3050",
                "update_date": datetime(
                    2022, 4, 4, 9, 56, 5, tzinfo=timezone.utc
                ),
            },
            "results": {
                "national": {
                    "headers": (("School Closure",), ("#impact+type",)),
                    "values": ({"AFG": "Closed due to COVID-19"},),
                },
                "regional": {
                    "headers": (
                        ("No. closed countries",),
                        ("#status+country+closed",),
                    ),
                    "values": ({"ROAP": 1},),
                },
            },
            "title": "Global School Closures COVID-19",
        }

github-actions · 2023-09-18T01:43:45Z

Test Results

37 tests ±0 37 ✔️ ±0 3m 0s ⏱️ -3s
  1 suites ±0   0 💤 ±0
  1 files ±0   0 ❌ ±0

Results for commit 894122a. ± Comparison against base commit 1a5698e.

♻️ This comment has been updated with latest results.

turnerm · 2023-09-19T20:31:41Z

@mcarans Thanks so much for implementing this. However I'm not quite getting exactly the output that you've quoted ablove. For example, in this branch when I print the output of get_hapi_results, I obtain (truncated):

 '8520e386-9263-48c9-b1bf-b2349e019fbb': {'hdx_id': '8520e386-9263-48c9-b1bf-b2349e019fbb',
                                          'hdx_stub': 'cod-ps-col',
                                          'provider_code': '95aa8d05-b110-4607-9330-f2a779885493',
                                          'provider_name': 'unfpa',
                                          'reference_period': {'enddate': datetime.datetime(2023, 9, 19, 23, 59, 59, 706701, tzinfo=datetime.timezone.utc),
                                                               'enddate_str': '2023-09-19T23:59:59+00:00',
                                                               'ongoing': True,
                                                               'startdate': datetime.datetime(2023, 8, 8, 0, 0, tzinfo=datetime.timezone.utc),
                                                               'startdate_str': '2023-08-08T00:00:00+00:00'},
                                          'resource': {'download_url': 'https://data.humdata.org/dataset/8520e386-9263-48c9-b1bf-b2349e019fbb/resource/e8f7fb08-af9c-4bdf-8a49-a54c56a4a1b0/download/col_admpop_adm1_2023.csv',
                                                       'filename': 'col_admpop_adm1_2023.csv',
                                                       'format': 'CSV',
                                                       'hdx_id': 'e8f7fb08-af9c-4bdf-8a49-a54c56a4a1b0',
                                                       'update_date': datetime.datetime(2023, 8, 8, 19, 57, 17, tzinfo=datetime.timezone.utc)},
                                          'results': {'adminone': {'headers': (['T_TL',
                                                                                'M_TL',
                                                                                'F_TL',
                                                                                'F_00_04',
                                                                                'F_100Plus',
                                                                                'T_00_04',
                                                                                'T_100Plus'],
                                                                               ['#population+total',
                                                                                '#population+f+total',
                                                                                '#population+m+total',
                                                                                '#population+f+age_0_4',
                                                                                '#population+f+age_100_plus',
                                                                                '#population+age_0_4+total',
                                                                                '#population+age_100_plus+total']),
                                                                   'values': ({'CO05': '6994792',
                                                                               'CO08': '2835509',
                                                                               'CO11': '7968095',
                                                                               'CO13': '2258929',
                                                                               'CO15': '1267378',
                                                                               'CO17': '1046418',
                                                                               'CO18': '423857',

(and there is also an "admintwo" key in the results)

Namely, the output of get_hapi_results is a dictionary with the dataset ID as keys, and while there are keys for both admin levels as expected, the resource information is only for admin 1.

…t work

…ource_metadata alongside headers and values under a particular admin level

mcarans · 2023-09-21T02:28:47Z

@turnerm get_hapi_metadata returns something like this:

{"3d9b037f-5112-4afd-92a7-190a9082bd80": {"hdx_id": "3d9b037f-5112-4afd-92a7-190a9082bd80",
                                          "hdx_stub": "cod-ps-eth",
                                          "provider_code": "522a7e16-3ba7-4649-b327-df81fd6dd689",
                                          "provider_name": "ocha-ethiopia",
                                          "reference_period": {"enddate": datetime.datetime(2023, 9, 21, 23, 59, 59, 385291, tzinfo=datetime.timezone.utc),
                                                               "enddate_str": "2023-09-21T23:59:59+00:00",
                                                               "ongoing": True,
                                                               "startdate": datetime.datetime(2022, 1, 5, 0, 0, tzinfo=datetime.timezone.utc),
                                                               "startdate_str": "2022-01-05T00:00:00+00:00"},
                                          "resources": {"bfb57304-3e22-498f-8a82-a345a8976852": {"download_url": "https://data.humdata.org/dataset/3d9b037f-5112-4afd-92a7-190a9082bd80/resource/bfb57304-3e22-498f-8a82-a345a8976852/download/eth_admpop_adm2_2022_v2.csv",
                                                                                                 "filename": "eth_admpop_adm2_2022_v2.csv",
                                                                                                 "format": "CSV",
                                                                                                 "hdx_id": "bfb57304-3e22-498f-8a82-a345a8976852",
                                                                                                 "update_date": datetime.datetime(2022, 8, 4, 18, 15, 44, tzinfo=datetime.timezone.utc)}},
                                          "title": "Ethiopia - Subnational "
                                                   "Population Statistics"},...}

get_hapi_results returns something like:

{"3d9b037f-5112-4afd-92a7-190a9082bd80": {"hdx_id": "3d9b037f-5112-4afd-92a7-190a9082bd80",
                                          "hdx_stub": "cod-ps-eth",
                                          "provider_code": "522a7e16-3ba7-4649-b327-df81fd6dd689",
                                          "provider_name": "ocha-ethiopia",
                                          "reference_period": {"enddate": datetime.datetime(2023, 9, 21, 23, 59, 59, 385291, tzinfo=datetime.timezone.utc),
                                                               "enddate_str": "2023-09-21T23:59:59+00:00",
                                                               "ongoing": True,
                                                               "startdate": datetime.datetime(2022, 1, 5, 0, 0, tzinfo=datetime.timezone.utc),
                                                               "startdate_str": "2022-01-05T00:00:00+00:00"},
                                          "results": {"admintwo": {"hapi_resource_metadata": {"download_url": "https://data.humdata.org/dataset/3d9b037f-5112-4afd-92a7-190a9082bd80/resource/bfb57304-3e22-498f-8a82-a345a8976852/download/eth_admpop_adm2_2022_v2.csv",
                                                                                              "filename": "eth_admpop_adm2_2022_v2.csv",
                                                                                              "format": "CSV",
                                                                                              "hdx_id": "bfb57304-3e22-498f-8a82-a345a8976852",
                                                                                              "update_date": datetime.datetime(2022, 8, 4, 18, 15, 44, tzinfo=datetime.timezone.utc)},
                                                                   "headers": (["T_TL",
                                                                                "M_TL",
                                                                                "F_TL"],
                                                                               ["#population+total",
                                                                                "#population+f+total",
                                                                                "#population+m+total"]),
                                                                   "values": ({"ET0101": "902719",
                                                                               "ET0102": "1475184",
                                                                               ...},
                                                                              {"ET0101": "446866",
                                                                               "ET0102": "724860",
                                                                               ...},
                                                                              {"ET0101": "455853",
                                                                               "ET0102": "750324",
                                                                               ...})}},
                                          "title": "Ethiopia - Subnational "
                                                   "Population Statistics"},
 "8520e386-9263-48c9-b1bf-b2349e019fbb": {"hdx_id": "8520e386-9263-48c9-b1bf-b2349e019fbb",
                                          "hdx_stub": "cod-ps-col",
                                          "provider_code": "95aa8d05-b110-4607-9330-f2a779885493",
                                          "provider_name": "unfpa",
                                          "reference_period": {"enddate": datetime.datetime(2023, 9, 21, 23, 59, 59, 385291, tzinfo=datetime.timezone.utc),
                                                               "enddate_str": "2023-09-21T23:59:59+00:00",
                                                               "ongoing": True,
                                                               "startdate": datetime.datetime(2023, 8, 8, 0, 0, tzinfo=datetime.timezone.utc),
                                                               "startdate_str": "2023-08-08T00:00:00+00:00"},
                                          "results": {"adminone": {"hapi_resource_metadata": {"download_url": "https://data.humdata.org/dataset/8520e386-9263-48c9-b1bf-b2349e019fbb/resource/e8f7fb08-af9c-4bdf-8a49-a54c56a4a1b0/download/col_admpop_adm1_2023.csv",
                                                                                              "filename": "col_admpop_adm1_2023.csv",
                                                                                              "format": "CSV",
                                                                                              "hdx_id": "e8f7fb08-af9c-4bdf-8a49-a54c56a4a1b0",
                                                                                              "update_date": datetime.datetime(2023, 8, 8, 19, 57, 17, tzinfo=datetime.timezone.utc)},
                                                                   "headers": (["T_TL",
                                                                                "M_TL",
                                                                                "F_TL",
                                                                                "F_00_04",
                                                                                "F_100Plus",
                                                                                "T_00_04",
                                                                                "T_100Plus"],
                                                                               ["#population+total",
                                                                                "#population+f+total",
                                                                                "#population+m+total",
                                                                                "#population+f+age_0_4",
                                                                                "#population+f+age_100_plus",
                                                                                "#population+age_0_4+total",
                                                                                "#population+age_100_plus+total"]),
                                                                   "values": ({"CO05": "6994792",
                                                                               "CO08": "2835509",
                                                                               ...},
                                                                              {"CO05": "3382645",
                                                                               "CO08": "1382767",
                                                                               ...},
                                                                              {"CO05": "3612147",
                                                                               "CO08": "1452742",
                                                                               ...},...)},
                                                      "admintwo": {"hapi_resource_metadata": {"download_url": "https://data.humdata.org/dataset/8520e386-9263-48c9-b1bf-b2349e019fbb/resource/76e12f52-af0d-45b2-8024-e6b0e63913c4/download/col_admpop_adm2_2023.csv",
                                                                                              "filename": "col_admpop_adm2_2023.csv",
                                                                                              "format": "CSV",
                                                                                              "hdx_id": "76e12f52-af0d-45b2-8024-e6b0e63913c4",
                                                                                              "update_date": datetime.datetime(2023, 8, 8, 19, 57, 19, tzinfo=datetime.timezone.utc)},
                                                                   "headers": (["T_TL",
                                                                                "M_TL",
                                                                                "F_TL",
                                                                                "F_00_04",
                                                                                "F_100Plus",
                                                                                "T_00_04",
                                                                                "T_100Plus"],
                                                                               ["#population+total",
                                                                                "#population+f+total",
                                                                                "#population+m+total",
                                                                                "#population+f+age_0_4",
                                                                                "#population+f+age_100_plus",
                                                                                "#population+age_0_4+total",
                                                                                "#population+age_100_plus+total"]),
                                                                   "values": ({"CO05001": "2653729",
                                                                               "CO05002": "21246",
                                                                               ...},
                                                                              {"CO05001": "1400979",
                                                                               "CO05002": "10112",
                                                                               ...})}},
                                          "title": "Colombia - Subnational "
                                                   "Population Statistics"}}

turnerm

Thanks for making this change!

get_hapi_results now returns a dictionary not a list

fee936c

mcarans requested a review from turnerm September 18, 2023 01:38

mcarans mentioned this pull request Sep 18, 2023

Add pop admin 1 OCHA-DAP/hapi-pipelines#19

Merged

Mike added 3 commits September 21, 2023 11:11

Commit this in case I go down this route later, but I think it may no…

66221a9

…t work

Have separate HAPI dataset and resource metadata and include hapi_res…

4e745a4

…ource_metadata alongside headers and values under a particular admin level

get_hapi_metadata returns a dictionary as well and has a resources key

fbfa801

mcarans changed the title ~~get_hapi_results now returns a dictionary not a list~~ get_hapi_metadata, get_hapi_results now return dictionaries Sep 21, 2023

mcarans mentioned this pull request Sep 21, 2023

get_hapi_metadata, get_hapi_results now return dictionaries OCHA-DAP/hapi-pipelines#26

Merged

turnerm approved these changes Sep 21, 2023

View reviewed changes

get_hapi_metadata returns a dictionary as well and has a resources key

894122a

mcarans merged commit 7d4faa5 into main Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

get_hapi_metadata, get_hapi_results now return dictionaries #23

get_hapi_metadata, get_hapi_results now return dictionaries #23

Uh oh!

mcarans commented Sep 18, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Sep 18, 2023 •

edited

Loading

Uh oh!

turnerm commented Sep 19, 2023 •

edited

Loading

Uh oh!

mcarans commented Sep 21, 2023

Uh oh!

turnerm left a comment

Uh oh!

Uh oh!

get_hapi_metadata, get_hapi_results now return dictionaries #23

get_hapi_metadata, get_hapi_results now return dictionaries #23

Uh oh!

Conversation

mcarans commented Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

turnerm commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcarans commented Sep 21, 2023

Uh oh!

turnerm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mcarans commented Sep 18, 2023 •

edited

Loading

github-actions bot commented Sep 18, 2023 •

edited

Loading

turnerm commented Sep 19, 2023 •

edited

Loading