Skip to content
This repository has been archived by the owner. It is now read-only.
Service to proxy csv and excel files on HDX
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app
tests
.gitignore
.pipignore
.travis.yml
LICENSE
MANIFEST.in
Makefile
Procfile
README.md
config.py
dev-requirements.txt
manage.py
prod-requirements.txt
requirements.txt
runtime.txt
setup.cfg
setup.py

README.md

HDX File Proxy API

Introduction

The hdx-file-proxy is a Flask powered service to proxy csv and excel files on HDX.

Requirements

hdx-file-proxy has been tested on the following configuration:

  • MacOS X 10.9.5
  • Python 2.7.10
  • postgres (PostgreSQL) 9.4.1

hdx-file-proxy requires the following in order to run properly in development mode:

Additionally, hdx-file-proxy requires the following in order to run properly in production mode:

Setup

local

(You are using a virtualenv, right?)

sudo pip install -r requirements.txt
manage setup
manage populate
manage serve

Production

pip install -r prod-requirements.txt
sudo -u postgres pg_ctl start
memcached -vv
manage -m Production setup
manage -m Production populate
gunicorn app:create_app('Production') -w 3 -k gevent
screen -dS worker -m python worker.py

Usage

hdx-file-proxy is intended to be used via HTTP requests.

Examples

cURL

Check the data returned from HDX API

# request
curl http://localhost:3000/v1/data/

# response
{
  "num_results": 0,
  "objects": [{
    "adm0_name": "Afghanistan",
    "adm1_name": "Badakhshan",
    "cm_name": "Bread",
    "id": 1,
    "mkt_name": "Fayzabad",
    "mp_month": "3",
    "mp_price": 50.0000000000,
    "mp_year": 2015,
    "utc_created": "2015-10-28T04:02:14.556843",
    "utc_updated": "2015-10-28T04:02:14.557005"
  }],
  "page": 1,
  "total_pages": 0
}

Python

initialize

# init requirements
import requests

endpoint = 'http://localhost:3000/v1'

Check the data returned from HDX API

# request
r = requests.get('%s/data/' % endpoint)

# response
r.json()
# same as cURL above

Configuration

API

All configuration options are available in config.py:

Option Description Default
ENDPOINT The HDX url https://data.hdx.rwlabs.org
RID Resource id of the file you wish to fetch b5b850a5-76da-4c33-a410-fd447deac042
API_KEY Your HDX API Key MY_API_KEY
DEBUG Enable the Flask debugger False
TESTING Testing mode False
PROD Production mode False
CHUNK_SIZE Number of rows to process at a time 10000

ckan

Under the hood, hdx-file-proxy uses ckanutils and uses the following Environment Variables if set:

Environment Variable Description
CKAN_API_KEY Your CKAN API Key

Scripts

hdx-file-proxy comes with a built in task manager manage.py.

Examples

View all available commands

manage

Run python linter and nose tests

pip install -r dev-requirements.txt
manage lint
manage test

Run dev server on custom port and with multiple threads

manage serve -tp 3001

License

hdx-file-proxy is distributed under the MIT License, the same as Flask.

You can’t perform that action at this time.