Skip to content

critocrito/sugarcube

main
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Sugarcube

Sugarcube - Data pipelines for human rights

GitHub GitHub Workflow Status Coverage Status

Synopsis

Sugarcube is a framework to fetch, transform and export data. Data processes are described using plugins, which are chained in sequence to model complex data processes.

It is a tool designed to support journalists, non-profits, academic researchers, human rights organisations and others with investigations using online, publicly-available sources (e.g.tweets, videos, public databases, websites, online databases).

Learn how to use Sugarcube on your own project.

This code is licensed under the GPL 3.

Documentation

All documentation can be found on the website.

Examples

There are more examples and explanations on the website. Here is one to get you started.

sugarcube -p http_import,media_warc,media_screenshot,elastic_export \
          -c config.json \
          -Q http_url:'https://mwatana.org/en/airstrike-on-detention-center/'

This example will fetch and extract the contents and meta data of an online article, archive the website as a Web ARChive, take a screenshot of the website and store the data in an Elasticsearch database.

Data processes, like from the example above, can be codified in order to repeat them. Once a data process has been defined, Sugarcube allows to scale and automate it's operation.

Testimony

  • Syrian Archive uses Sugarcube to archive video evidence of human rights violations in Syria. Further, Sugarcube is used to monitor human rights documentation that is taken down by social media companies. The systems and workflows developed with Syrian Archive are now being expanded to do similar work in Yemen, Sudan and other areas.

  • Built using Sugarcube, the Data Scores investigation tool provided evidence and insights for research into how data analytics and data-driven "scoring" were being used in the public sector of the UK to make decisions. This research was conducted by the Data Justice Lab.

License

Sugarcube is licensed under the GPL 3.0.