Skip to content
Convert data into valid Human Service Data Specification (HSDS) datapackages. (Produced by the Open Referral Initiative.)
Ruby Dockerfile
Branch: master
Clone or download
switzersc Update README.md
Update with correct Travis build image URL
Latest commit b581e87 Aug 2, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Set theme jekyll-theme-cayman Feb 24, 2019
examples WIP cleanup Jun 25, 2019
lib Update dockerfile and add usage instructions to readme Jul 30, 2019
spec Rename open_referral_transformer to hsds_transformer Jul 30, 2019
tmp
.env.example Rename open_referral_transformer to hsds_transformer Jul 30, 2019
.gitignore Add organization transformation Feb 13, 2019
.ruby-version Initialize repo with ruby and rspec Feb 12, 2019
.travis.yml Update .travis.yml Jul 30, 2019
CONTRIBUTING.md
Dockerfile Update dockerfile and add usage instructions to readme Jul 30, 2019
Gemfile
Gemfile.lock
LICENSE Create LICENSE Feb 24, 2019
README.md Update README.md Aug 2, 2019
config.ru Zip files and return via API Feb 24, 2019

README.md

HSDS Transformer

Build Status

Overview

This app allows you to convert data into an HSDS-compliant datapackage.

The Human Services Data Specification (HSDS) is a data model that describes health and human services.

This transformer currently transforms data into HSDS version 1.1.

Problem statement

Lots of people have health and human services data, but it's all in different structures and formats. Many groups want to share their data, or use data that's shared with them, but the lack of standardized data or an easy way to standardize the data presents a major barrier.

Solution

Open Referral provides a Human Services Data Specification that groups can use to share data via a common format. The HSDS Transformer tool (this project) enables a group or person to transform data into an HSDS-compliant data package, so that it can then be combined with other data or used in any number of applications.

Case Study: Illinois Legal Aid Online (ILAO)

The real-world case that's motivating this project is Illinois Legal Aid Online (ILAO), an organization that helps individuals in Illinois find legal help or resources online. ILAO has a resource directory of community services data, and they'd like to update this data to be formatted in compliance with HSDS version 1.1. They also are looking to incorporate other data sets in disparate formats and structures into their own database in order to better serve Illinois residents with more extensive and more accurate information.

This project uses ILAO's resource database in the form of three CSVs as the test case. The data was pulled in January 2019, with some records not having been updated since 2017, and therefore some of the data may not be accurate or up-to-date. The data is located in spec/fixtures/input.

Ideas for expansion

How to use this tool

First, double check whether your data is already HSDS-compliant with the OpenReferral Validator.

If it is, voila!

If it's not, proceed.

Installing

  1. Clone this repo locally.
  2. In terminal, cd into the root of the hsds_transformer directory.
  3. Create a new file called .env. Copy the contents of .env.example into the new .env file and update Users/your_user/dev/hsds_transformer to be the correct path to this directory on your local environment (you can run pwd in terminal to find this out).
  4. Install all the gems by running bundle install

Transforming

Here's how to transform the data if you're familiar with Ruby:

  1. Make sure your data is saved locally as CSVs in this directory.
  2. Create a mapping.yaml file and store it locally in this directory. This is what tells the transformer how to map fields from one set of CSVs into the HSDS format. See spec/fixtures/mapping.yaml for an example.
  3. Open up an interactive Ruby session in terminal by running irb (or pry - up to you!)
  4. Require the class: require "./lib/hsds_transformer"
  5. Run the transformer:
HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")
  1. Now check the tmp directory for your newly created HSDS files!

You can also pass two additional arguments to the .run command: include_custom and zip_output. The output is by default not zipped, but if you want it to be, you can pass true as the value of this field. If your input data includes non-HSDS fields you want to see in the output files as well, you can pass true for include_custom.

Using the API

Start the API from the root of the project directory:

rackup -p 4567

Make a POST request with params: input_path, mapping, include_custom. Each of thse should contain a path to a CSV file and a mapping file (for mapping).

E.g.

curl -X POST -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform

The response will be a zip file of the transformed data. You can also pass add -F "include_custom=true" if your input data has custom non-HSDS columns you wish to include.

The API then streams a zip file back with the properly transformed data. The zip is also saved locally on the API server (maybe your local env) at data.zip in the root directory

Starting the API with Docker

Before anything else, make sure you have docker installed and running.

First, build the image locally:

docker build . -t transformer_api

Now, run it:

docker run -p 4567:4567 transformer_api

You should now be able to interact with the API at http://localhost:4567.

Custom Transformers

The BaseTransformer maps data from the input directory to compliant HSDS datapackage and CSVs using the mapping.yaml, and it requires a pretty one-to-one and straightforward mapping. You may need additional cleanup, parsing, or mapping, such as parsing out schedule text. If so, you can create a custom transformer and specify it when running the script or using the API. Check out the lib/hsds_transformer/custom directory for examples.

  1. Write your custom transformation code.
  2. Save it as a class in lib/hsds_transformer/custom following the naming conventions already there.
  3. Add the class name to the array of valid custom transformers in the HsdsTransformer::Runner class.
  4. Specify this custom transformer when invoking the transformer:
HsdsTransformer::Runner.run(custom_transformer: "Open211MiamiTransformer", input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")

or when making a request to the API:

curl -X POST -F "custom_transformer=Open211MiamiTransformer" -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform

Examples

You can find examples of data and mappings in the examples directory.

Related Projects

Open Referral Playground App

  • Playground with various tools and code for working with Open Referral data
  • Github

Open Referral Drupal

Open Referral Wordpress

Open Referral Laravel Services

  • Laravel/MySQL/Vue.js app
  • Github

Open Referral Validator

Open Referral datapackage code

  • Repo containing some code exploring creating Open Referral Datapackages
  • Github

Open Referral Sample Data

  • Repo of sample data for use in projects
  • Github

Open Referral Gem

  • This doesn't seem to have any actual code yet, but maybe the maintainer will update!
  • Github

Open Referral OpenAPI Specification

You can’t perform that action at this time.