Skip to content
master
Go to file
Code

README.md

OpenSanctions.org

This repository contains the home page for OpenSanctions.org, an open-source repository of sanctions data, politically exposed persons, and other entities of interest.

The scrapers are executed using memorious, a scraping framework.

OpenSanctions.org is generated using Jekyll, and hosted on GitHub pages. Please feel free to submit pull requests for any suggested improvements.

Data sources

We collect data sources for this project here. Please feel free to contribute and to add possible sources or scrapers.

Running

  1. Bring up the services:

     docker-compose up
    
  2. Open a shell in the worker container:

     docker-compose exec worker sh
    
  3. Run a crawler:

     memorious run un_sc_sanctions
    
  4. Export to CSVs:

     ftm store iterate -d un_sc_sanctions | ftm export-csv
    

The Memorious should be available at localhost:8000. pgweb interface for the database should be available at localhost:9633.

Pushing data into Aleph

To push crawled entities to Aleph, add ALEPHCLIENT_HOST and ALEPHCLIENT_API_KEY as environment variables to the worker container and rerun the crawlers. Scraped entities are pushed to Aleph after a crawler is finished running.

You can also push crawled entities to Aleph manually.

  1. Open a shell in worker container:

    docker-compose exec worker sh
    
  2. Iterate over scraped entities and send them to Aleph

    ftm store iterate -d un_sc_sanctions | alephclient write-entities -f un_sc_sanctions
    

About

An open database of persons of interest and politically exposed persons

Topics

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.