Data Scope - a framework for data mashups
Python HTML JavaScript Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
core Merge pull request #111 from fako/datagrowth Aug 18, 2018
datagrowth Fixes imports Aug 18, 2018
datascope Merge pull request #111 from fako/datagrowth Aug 18, 2018
future_fashion Cleans Future Fashion application a bit for later improvements Jul 20, 2018
nautilus Locafora product update Jun 22, 2018
online_discourse Sets default ranking for online discourse ranking Aug 18, 2018
open_data Moves some sources into open_data Aug 18, 2018
setup_utrecht Moves trolls into setup_utrecht Aug 18, 2018
sources Moves some sources into open_data Aug 18, 2018
system Starts the datagrowth package for eventual separation of "core" funct… Jul 20, 2018
topic_research Moves sources from topic_research into wiki_scope Aug 18, 2018
trolls Moves trolls into setup_utrecht Aug 18, 2018
visual_translations Updates the delete command to use the country as input. Dec 3, 2017
wiki_feed New way to calculate uri's for manifestations Jul 24, 2018
wiki_scope Moves sources from topic_research into wiki_scope Aug 18, 2018
.gitignore Improves on NumericFeaturesFrame. Allows content to be loaded from disk Jul 28, 2018
.travis.yml Fixes travis config Jul 24, 2018
LICENSE Changes to newest GPL to include patents. Nov 15, 2017
Makefile Starts the datagrowth package for eventual separation of "core" funct… Jul 20, 2018
README.md Fixes typo in README and install scripts. Nov 25, 2017
manage.py Importing necessary files, trying to make things run again. Oct 28, 2014
package.json Patch to allow order creation May 29, 2018

README.md

Data Scope

Data Scope is a data mashup framework meant to facilitate the execution of complex search queries. Think about search queries like: “which websites name at least three experts from field X?”, “which websites support stance X?”, “what are iconical images for X in culture A, B & C?” and “which people are known for similar reasons that person X is known for?”. Data Scope allows a programmer to reach over walled gardens of information that he/she may need and combine the information into new information. Due to its nature Data Scope is very suitable for software that tries to break out of filter bubbels.

Prerequisites

This is the environment that you should have on your machine before installing Data Scope.

  • Python 3
  • MySQL

Optionally you may also want to install

  • Redis (for background processing)
  • PhantomJS (for web scraping)
  • Conda (for environment management)

Installation

After cloning this repo to the machine you want to install Data Scope on. Activate the environment where you want to install Data Scope into. Then go into the repo directory and run the following for a local install.

cp datascope/environments/local_bootstrap.py datascope/bootstrap.py
cp datascope/environments/local_settings.py datascope/settings.py
cp datascope/environments/secrets_example.py datascope/secrets.py
pip install -r datascope/environments/local_requirements.txt

Alternatively you can change local in these commands to digital-ocean or wikipedia for an installation in the Digital Ocean or Wikipedia cloud respectively.

Then edit the datascope/bootstrap.py, datascope/settings.py and datascope/secrets.py to correct the setup of the machine/cloud.

After this you need to setup the database correctly. Run the following commands. You may need to specify the user and password flags in the MySQL command.

mysql -e "CREATE DATABASE datascope CHARSET utf8mb4;"
./manage.py migrate

Next steps

Depending on which project you're working on you may want to run additional installation steps. These steps are outlined in the readme's of these projects. Below are the links to all current projects:

Roadmap

The roadmap can be followed by keeping an eye on the project boards of this repo on GitHub.