Write code to detect shelters that are missing from irma-api

[simonw]

This repo now contains the (revision-tracked) output of a number of different scrapers.

We have a team of people working to keep https://irma-api.herokuapp.com/ up-to-date - that's the dataset that powers https://www.irmashelters.org/

We have avoided automatically adding shelters to irma-api because the data sources are often unreliable, and we feel it is better to have a human being manually check each entry, avoid duplicates, clean up the data and maybe use other sources to confirm that each shelter exists and is open.

BUT... there's no reason we can't have automated tools make suggestions to our human editors.

Here's the task: write code which pulls the current list of shelters from https://irma-api.herokuapp.com/api/v1/shelters and then pulls one of the scraped JSON files from this repo - then compares the two and tries to detect shelters that are missing from irma-api.

A useful starting point would be this file, which is scraped from http://www.floridadisaster.org/shelters/summary.aspx - the most recent JSON can be loaded from here: https://raw.githubusercontent.com/simonw/irma-scraped-data/master/florida-shelters.json

A useful script would do the following:

• Retrieve the current list of shelters from https://irma-api.herokuapp.com/api/v1/shelters (or from the mirror of that file in GitHub: https://raw.githubusercontent.com/simonw/irma-scraped-data/master/irma-shelters.json )
• Pull the latest scraped data from https://raw.githubusercontent.com/simonw/irma-scraped-data/master/florida-shelters.json
• Loop through the list in florida-shelters.json and, for each shelter, try to determine if it is already listed in irma-api. My hunch is that the best way to do this is by comparing latitude/longitude points, since the names and addresses of the shelters frequently have different spellings. A fuzzy text match against the name or address might work well too - try different things and see what works out. I've been using a 7 character Geohash of the latitude/longitude for dupe detection elsewhere and it seems to work very well.
• Output a list of possible dupes. Share that with the irma-response-slack.herokuapp.com community in the #shelters channel - manually at first, but once it's proved itself set it up to automatically share in some way.

The way my irma-dupe-detection script works may be a useful inspiration: https://github.com/simonw/irma-scrapers/blob/59cf9906d1005972ab2e172ec494336b7b8b8434/irma_shelters.py#L174-L217

[simonw]

The florida-shelters.json file generated from http://www.floridadisaster.org/shelters/summary.aspx is a useful starting point... but the really interesting data comes from FEMA. It's a lot larger and potentially harder to work with, but if you're looking for an ambitious, impactful task try doing the above but for these JSON files instead:

• https://github.com/simonw/irma-scraped-data/blob/master/fema-open-shelters.json
• https://github.com/simonw/irma-scraped-data/blob/master/fema-nss.json

[commadelimited]

Nit, should that be florida-shelters.json @simonw?

[simonw]

Typo fixed :)

That "commit that referenced this issue" above is junk - one of the shelters mentioned in there happened to have #1 in its name. Unfortunately I don't think I can delete rogue commits-that-referenced-this-issue.

[simonw]

I have a weak preference for getting this solved in Python because that would make it trivial for me to hook it into the rest of the bot scraping infrastructure I have set up already - but getting this solved quickly is much more important so feel free to use whatever technology you are most comfortable with.

[mjturtora]

pefect, python and i are buds

[simonw]

I've built the first one of these, to compare the florida shelters site with the shelters listed on irma-api. Here's the implementation: simonw/irma-scrapers@113b221