Add a data set that includes total population figures

[cliftonmcintosh]

The wazimap framework will only display data when there is a data set that includes total population figures. Find a data set that has full population figures for each of the bodies at each of the new levels for federal Nepal. Data at higher levels can be derived from data at the local level. Create the appropriate table for these data and add them to the profiles and to the view templates.

See https://github.com/Code4Nepal/nepalmap_app/blob/dev/wazimap_np/demographics.py#L45-L48 as an example of how total population is used.

The data at http://cbs.gov.np/sectoral_statistics/population/Population%20of%20753%20Local%20Units should work.

[cliftonmcintosh]

@ravinepal

Do we have a data set that includes total population at the local level for the new local bodies?

[ravinepal]

thanks, @cliftonmcintosh.

yes, here are CBS files:

1. Province data
2. Districts
3. GauPalikas/Municipalities and Wards

Let me see if @Crackjack has suggestions or know how to efficiently scrape data out of the website.

In this website created by Ministry of Federal Affairs and General Administration: http://103.69.124.141/
there is data on: population of an admin level and area. Data is in Nepali. I'll reach out to a contact to see if there's data in English available.

also, @pratimakandel - would you be interested in helping us scrape/translate data from http://103.69.124.141/ in a Google doc? I can share more details on how to do it if you are interested. Pls let me know via email or by commenting here.

[openrijal]

the links you pointed to are PDFs and the website that is Nepali is build on Flash or something similar, the links do not change

if the data for only Province Level is enough, we can use the site, else need to figure out a way to grab data from the PDF.

I'm looking into it, will give some answer in a couple of hours.

[nikeshbalami]

Scrapping from PDF will be easy because its already in English, I have already scrapped Districts and Provinces data.

We need to work to scrap Local Units: http://cbs.gov.np/sectoral_statistics/population/Population%20of%20753%20Local%20Units

[cliftonmcintosh]

If we want to display local level information, then we will be required to have local level population figures. A basic rule in the wazimap framework is that we must have at least one data set that includes total population for the lowest geographic/political unit that we wish to display.

Also, the figures for higher level can be derived from the lowest unit since we can just add together the values from the local level to get the district and higher levels, so we only really need the data from the local level.

[ravinepal]

thanks, @cliftonmcintosh. I'm scraping data. Here's province level population and household data - I submitted the PR to data repo and requested you to review:

CodeforNepal/data#13 (FYI @Crackjack)

[ravinepal]

@nikeshbalami - can you submit pull request with district level data here: https://github.com/Code4Nepal/data/

I'll try to scrape local level data.

@cliftonmcintosh - then we will have data for all levels.

[cliftonmcintosh]

If we have the local level, we don't need data from any other level because we can just add up the local units to get the district level and add up that to get provinces, etc. That's probably preferable because then we know the data are consistent.

[nikeshbalami]

I started scrapping local levels, need a couple of nights to complete the whole.

[openrijal]

what tool are you using? @nikeshbalami

would it be faster if all of us run the tool and use our computing time and process different levels?

[nikeshbalami]

I use Tabula for the basic PDF scrapping @Crackjack

Here I have submitted a pull request of 3 districts: CodeforNepal/data#14

Have created the separate folder for Federal Datasets>Polutation and HouseHold>District Name>Local Units.., This pattern will help us for the easy aggregation to find out Districts and Province level data.

@ravinepal I think you need to remove the comma from those total numbers, cuz in CSV columns is represented by comma thus having a comma in amount and number may throw an error if someone downloads the open the dataset in excel or others.

[cliftonmcintosh]

The data that @nikeshbalami is collecting at https://github.com/nikeshbalami/data/tree/f398ee21ec2bd03d9047f68bb519dd0faa45a4e1/Federal%20Data/753%20Local%20Unit%20Population%20and%20HouseHold are exactly the sort of thing we need. From that we can derive population for all levels above the local level. I would suggest that people concentrate on completing that data set. If team members produce files that are exactly the same shape, it will make them easier to convert to something that can be used for the federal NepalMap project.

Of course, even though the map project doesn't need it, there may be other reasons to scrape data for other levels.

[nikeshbalami]

Had completed scrapping 9 districts: https://github.com/nikeshbalami/data/tree/master/Federal%20Data/753%20Local%20Unit%20Population%20and%20HouseHold

[openrijal]

local levels are actually divided into 4 categories (Metropolitan, Sub-Metropolitan, Municipality, Rural Municipality)

To get the statistics for districts, these categories under them should be summed up
To get the statistics for provinces, the districts should be summed up
To get the statistics for country, the provinces should be summed up

[ravinepal]

I think this is done too, yeah?

[cliftonmcintosh]

The data has not been imported into nepalmap_federal. It is now available in the data project here:
https://github.com/Code4Nepal/data/tree/master/Federal%20Data/753%20Local%20Unit%20Population%20and%20HouseHold.

It still needs to be transformed into a table in nepalmap_federal and added to a view. That can all be covered in #11.