Skip to content

start a new (random) life | World Bank data scraper on django rest app

Notifications You must be signed in to change notification settings

mpszumowski/djangocrawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

45 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

djangocrawler

Do you feel lucky?

Take a look: http://mpszumowski.pythonanywhere.com/index.html

A django-rest app which gathers and updates data via scrapy framework

How does it work?

This app is designed to 1) extract data about world population and poverty rates using scrapy, 2) manage the data using django models, 3) express them by means of simple rest framework and ajax-driven webpage and 4) enable a simple drawing function which (a) selects a country each time "a person" "is born", (b) draws whether this "person" lives below or above the poverty rate of $3.10/day and (c) compares this with minimal daily food cost.

screenshot

The button at the top-left edge of the page draws a random country in the world based on its population. The marker highlights yellow if the data is incomplete, blue if it hit the segment of population above - and red if below the threshold of $3.10/day. If the daily food cost is higher than $3.10 the random function works normally, but if it is lower, there is a modificator introduced to lower the probability of finding oneself in the starving part of population.

In a word, the marker indicates whether you surely starve. If it turns blue it is still probable that the whole of your budget would be spent on food if you chose to eat the minimal 2,400kcal ration, but it is indeterminable by the nature of scraped data.

Disclaimer: This is not an exercise in statistics, but in scrapy and django frameworks. The accuracy of collected data relies on the methodology of World Bank and Numbeo.com. Their juxtaposition, on the other hand, is not fully correct, because the data have discrete rather than continuous character.

Sources:

List of countries and population: http://wdi.worldbank.org/table/2.1
Minimum $ amount for daily food ration: https://www.numbeo.com/food-prices/
Poverty thresholds: http://wdi.worldbank.org/table/1.2

The $3.10 poverty index provides a better comparison (in contrast with the usual $1.90 threshold) with food prices which are generally not lower than $3.00/ration/day

Implementation:

  • ./manage.py runserver --> open index.html
  • Scrapyd server running at port 6800 is necessary to run the scrapers from the level of browser.

TODO:

  • add data concerning ongoing armed conflicts
  • reserve the actualization function to logged user with appropriate permission

About

start a new (random) life | World Bank data scraper on django rest app

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published