HDX Export - Send to HDX via API

[jrtricafort]

After mapping HXL fields during export (#2518), user can send data from the Ushahidi platform to HDX via the HDX API.

When the CSV is generated with the HXL tags and the export_job is complete, we need to run an additional set of actions

• Create or update a dataset [Q: how do we decide if an update or create is used here?]
• Create or update the dataset's resources

TODO - connection to HDX:

• HDX CKAN wrapper to create and update datasets (AKA packages) [ Platform API ] Setup CKAN API Client for laravel #2881
• HDX CKAN wrapper to add resources to a dataset ([ Platform API ] Setup CKAN API Client for laravel #2881)
• Basic HDX CKAN connection wrapper that isolates the connection details from the rest [ Platform API ] Setup CKAN API Client for laravel #2881
• add .ENV key for which humdata server to use (test/prod)
• Add Usecase to create datasets
• Add Usecase to update datasets
• Add Usecase to create resources
• Add a way to call the create dataset/update dataset usecase after the HXL file is in rackspace (using lumen queues)

steps

After the CSV + HXL file is uploaded to rackspace:

• If a dataset needs to be created: call usecase to create datasets using the HDX API wrapper
  • when the dataset is created, call the usecase to create resources with the dataset id, 'csv' as resource type, package name, and the URL .
  • when the resource creation finishes,
    • if successful: return the status of the action to the dataset usecase and update the export_job details with the "UPLOADED TO HDX" status
    • if the resource failed to create, update the export_job details with "FAILED TO UPLOAD TO HDX" status and log the error.
• If a dataset with the name the user requested already exists for that user, it needs to be updated:
  • Call usecase to update datasets using the HDX API wrapper
    • use the job information to get the dataset's id
    • update the dataset with the new changes
    • when the dataset is updated, call the usecase to create the dataset's resource with the dataset id, 'csv' as resource type, package name, and the URL .
    • when the resource creation finishes,
      • if successful: return the status of the action to the dataset usecase and update the export_job details with the "UPLOADED TO HDX" status
      • [ ]if the resource failed to create, update the export_job details with "FAILED TO UPLOAD TO HDX" status and log the error.

Creating a dataset

• POST https://data.humdata.org/api/action/package_create
  {
  "name": "test-dataset-romina",
  "author": "Romina",
  "maintainer": "romina-1223",
  "organization": "ushahidi",
  "private": true,
  "owner_org": "ushahidi",
  "title": "test-dataset-romina",
  "dataset_source": "category lat lon data",
  "data_update_frequency": "never",
  "methodology": "other",
  "tags": [{"name":"coordinates"}],
  "license_id": "cc-by-sa",
  "allow_no_resources": true
  }
• the option allow_no_resources to upload the files in a separate step
• HTTP Headers:
  • X-CKAN-API-Key & Authorization both with your HDX Api key as value
  • Content-Type: application/json
  • Accept: application/json

Adding resources to a dataset:

• POST https://data.humdata.org/api/action/resource_create
  {
  "package_id": "xxxx-xxxx-xxxx-8e96bcceb658",
  "url": "http://our-rackspace-files/hdx/countrylatlons.csv",
  "resource_type": "csv",
  "name": "package-csv-lat-lon"
  }
• HTTP Headers:
  • X-CKAN-API-Key & Authorization both with your HDX Api key as value
  • Content-Type: application/json
  • Accept: application/json
• Important: missing name or resource type will result in a HTTP error 500 instead of a 422 and it won't provide any details about how/why it failed. The documentation does not list those two fields as required.
• Important we should really validate everything on our side as well to minimize potential issues with CKAN HDX
• How to get the values we will send:
  • package_id is returned in the package (dataset) creation. You can also get it with the "Get dataset' http request listed here.
  • url the url of the file in our CDN/File storage service.

See details of a package to obtain id or other data

https://data.humdata.org/api/action/package_show?id=test-dataset-romina

[Erioldoesdesign]

Here's the screens from 'upload to HDX' modal onwards: https://xd.adobe.com/view/2690f082-d88d-4788-5db7-b04c9474a404-50a1/screen/01c9f863-e129-42cc-938c-c44b13193306/Data-export-HDX-single-survey-30?fullscreen

[rowasc]

@willdoran can we discuss PULL vs PUSH tomorrow?

[willdoran]

@rowasc yep ping me

[Erioldoesdesign]

@rowasc Did you want to talk to me about something? I had a notification about users updating a datasets name?

[rowasc]

@Erioldoesdesign sorry, I had a question about the dataset name but it's resolved since you added that field already. Thanks for checking in