Update Piecewise docs #135

critzo · 2017-05-15T18:51:40Z

This PR updates all public documentation to be consistent with the current state of Piecewise code.

A reviewer should be able to follow the documentation from the Getting Started section of the main project README, and end up with a working Piecewise server or VM.

This change is

georgiamoon · 2017-05-23T03:02:37Z

docs/system-requirements.md

+
+2. Clone your fork onto your local computer
+
+```$ git clone https://github.com/opentechinstitute/piecewise.git```


I think it'd be good for the code example here to reference a fork, rather than the home repo, e.g:
$ git clone https://github.com/<YOUR_FORK>/piecewise.git

georgiamoon · 2017-05-23T04:01:31Z

Reviewed 1 of 8 files at r1.
Review status: 1 of 8 files reviewed at latest revision, 5 unresolved discussions.

docs/config.md, line 22 at r1 (raw file):

* Enter your location in the box labelled "Find a place with Google ..." on [http://boundingbox.klokantech.com/](http://boundingbox.klokantech.com/) and press enter. Some example searches: "Baltimore, MD" or "Clearwater County, WA"
* In the lower left menu change the settings to: **CSV**
* Copy the coordinates at the bottom of the page into your documentation. They should like this: ```-116.4559, 46.2657, -114.5947, 46.9346```

I wasn't sure what documentation you were referring to here. I think we should help build a config file, or keep repeating that this should be in a note somewhere.

docs/config.md, line 60 at r1 (raw file):

#### Download and save shapefiles

Download and save your shapefiles to the folder ```local_customizations/geofiles/```. For example, listed below are the 2015 census block group shapefiles for Washington State, USA. The file ```wa_censusblocks.topojson``` is created in the next step, but included here because it will be saved in the same folder.

I expected this folder to be empty. What happens when I have multiple files in here? Is that OK? Or should there be a folder structure, e.g:

local_customizations
- seattle_wa
  - geofiles
- another_location
  - geofiles

Let's make/add a note about options here re:organizing your local_customizations folder.

docs/config.md, line 105 at r1 (raw file):

The global configuration file intended to be self-documented. Open it in your text editor and read through the comments from top to bottom. The file is extensively commented to advise you on what each variable is used for and what format to enter it in. Comment lines start with "#". 

You will see single variables and their values, for example:

I would note here is where you change the geo-references since it's not noted earlier. I would actually note in the file, everything that needs to be change, and add some smart errors to the ansible playbook so that it's easier to know what's going on. I missed about half of the needed configs the first time.

docs/deploy.md, line 7 at r1 (raw file):

To run this playbook and deploy your customized code to your server, open a terminal window and navigate to the directory where your Piecewise files are on your local computer. Then run the following command in your terminal window:

```$ ansible-playbook -i ansible/inventory/debian8-hosts ansible/deploy-piecewise.yml```

I'm getting bigquery errors here, and it doesn't look like any of my file copied over as a result.

Comments from Reviewable

meagdoh · 2017-06-01T16:25:35Z

docs/config.md

+* In the lower left menu change the settings to: **CSV**
+* Copy the coordinates at the bottom of the page into your documentation. They should like this: ```-116.4559, 46.2657, -114.5947, 46.9346```
+  * The four coordinates are the southwest and northeast corners of the area you selected. From left to right, these values are: ```<southwest longitude>,<southwest latitude,<northeast longitude>,<northeast latitude>```
+  * In your text file, put the values into the format below:


Which text file? piecewise > piecewise > requirements.txt? That is the only .txt file

@meagdoh Thanks for catching this. @georgiamoon reported this also. Will attempt to make it clearer.

meagdoh · 2017-06-02T13:43:27Z

docs/config.md


-```
+```TO DO: add reference to international shapefile sources```


http://gadm.org/country for Shapefiles by country.

acknowledged. thanks.

meagdoh · 2017-06-02T13:47:13Z

docs/config.md

-4. Also in the ```"bins"``` section, change: ```"key" : "geoid"``` to: ```"key" : "<unique key field name>"```. The key name is the unique field in your geodata file which is used to join aggregated M-Lab data to geodata regions.
-
-**Filters Section Changes**
+Download and save your shapefiles to the folder ```local_customizations/geofiles/```. For example, listed below are the 2015 census block group shapefiles for Washington State, USA. The file ```wa_censusblocks.topojson``` is created in the next step, but included here because it will be saved in the same folder.


Delete Washington State, USA example files. OR add instruction to remove files.

meagdoh · 2017-06-02T14:37:02Z

docs/config.md


-##### Edit the main Piecewise configuration file
+Once you locate the shapefile(s) you need, it's good practice to open the shapefile in [QGIS](http://www.qgis.org/en/site/) or another GIS program to confirm that it meets the requirements to use within your Piecewise application. When you open the shapefile(s) in your GIS program, **make a note of the name of its unique key field.** You'll use that later in these instructions.


A few additional instructions for QGIS:
(0) Download QGIS
(1) Project > New
(2) Layer > Add Layer > Add Vector Layer with local .shp files
(3) Choose target granularity level. In the case of gadm.org shapefiles, 0 equals country (least specific) to 3 is most specific.
(4) Remove unused local files. At this point, you should have 6 files: .cpg, .csv, .dbf, .prj, .shp, .shx. For more info on Shapefiles: http://wiki.gis.com/wiki/index.php/Shapefile

meagdoh · 2017-06-02T14:48:39Z

docs/config.md


-This JSON file is arranged in sections with sub-elements. The relevant sections we will change begin with **"aggregations"** and **"filters"**. Each section below begins with a code block from the relevant section, followed by instructions on what to change, and ending with the edited section of code. In our examples, we are using the folder and file names for our Baltimore example. Your folder and file names will likely differ.
+* Must contain at least one field that serves as a unique key, for example "geoid" or "geoid10" in the case of census block groups


Which files contains the unique key? The .csv has 'OBJECT_ID', but unsure if that's the correct id.

meagdoh · 2017-06-08T15:07:05Z

@critzo I realized I never mentioned you in the comments above. Hopefully, this helps with future deployments.

critzo · 2017-06-26T16:47:10Z

docs/config.md, line 22 at r1 (raw file):

Previously, georgiamoon (georgia bullen) wrote…

I wasn't sure what documentation you were referring to here. I think we should help build a config file, or keep repeating that this should be in a note somewhere.

Acknowledged. @meagdoh noted this as well. Upcoming commit will make this clearer.

Comments from Reviewable

critzo · 2017-06-26T16:49:40Z

docs/config.md, line 60 at r1 (raw file):

Previously, meagdoh (Meag Doherty) wrote…

Delete Washington State, USA example files. OR add instruction to remove files.

Adding instruction to remove files.

Comments from Reviewable

critzo · 2017-06-26T16:51:42Z

docs/config.md, line 53 at r1 (raw file):

Previously, meagdoh (Meag Doherty) wrote…

Which files contains the unique key? The .csv has 'OBJECT_ID', but unsure if that's the correct id.

That's going to be dependent on the files we obtain. In the future I'd like to automate the import of geodata and simplify this process more. For now, folks will need to determine the unique key.

Comments from Reviewable

critzo · 2017-06-26T16:54:29Z

docs/config.md, line 60 at r1 (raw file):

Previously, georgiamoon (georgia bullen) wrote…

I expected this folder to be empty. What happens when I have multiple files in here? Is that OK? Or should there be a folder structure, e.g:

local_customizations

seattle_wa

geofiles

another_location

geofiles

Let's make/add a note about options here re:organizing your local_customizations folder.

I'm adding some clearer notes at the beginning of this section. I like the folder structure you're proposing here, but currently the Ansible automation only supports one location.

Comments from Reviewable

critzo · 2017-06-26T16:55:13Z

docs/config.md, line 105 at r1 (raw file):

Previously, georgiamoon (georgia bullen) wrote…

I would note here is where you change the geo-references since it's not noted earlier. I would actually note in the file, everything that needs to be change, and add some smart errors to the ansible playbook so that it's easier to know what's going on. I missed about half of the needed configs the first time.

Adding a table of variables here, and notation in the config file for all variables to note whether they are required or default.

Comments from Reviewable

critzo · 2017-06-26T16:57:36Z

docs/deploy.md, line 7 at r1 (raw file):

Previously, georgiamoon (georgia bullen) wrote…

I'm getting bigquery errors here, and it doesn't look like any of my file copied over as a result.

I'll investigate. Maybe a package name change for the Python bigquery package in gcloud tools.

Comments from Reviewable

critzo · 2017-06-26T16:57:58Z

docs/config.md, line 49 at r1 (raw file):

Previously, meagdoh (Meag Doherty) wrote…

A few additional instructions for QGIS:
(0) Download QGIS
(1) Project > New
(2) Layer > Add Layer > Add Vector Layer with local .shp files
(3) Choose target granularity level. In the case of gadm.org shapefiles, 0 equals country (least specific) to 3 is most specific.
(4) Remove unused local files. At this point, you should have 6 files: .cpg, .csv, .dbf, .prj, .shp, .shx. For more info on Shapefiles: http://wiki.gis.com/wiki/index.php/Shapefile

Thanks.

Comments from Reviewable

critzo · 2017-06-26T17:06:55Z

@georgiamoon I've made updates based on the comments thus far. Please take a look when you have a chance?

critzo · 2017-08-29T17:35:14Z

@georgiamoon When you have a moment could you take a look through the updates I had previously made to the Piecewise docs PR? Thanks.

georgiamoon · 2017-09-09T04:08:16Z

Review status: 0 of 8 files reviewed at latest revision, 10 unresolved discussions.

docs/deploy.md, line 34 at r2 (raw file):

Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=233384409938-f4775mdhc8qocob1icesp76nqdtsero3.apps.googleusercontent.com&access_type=offline

Is this the link for everyone or an example?

Comments from Reviewable

georgiamoon · 2017-09-09T04:09:25Z

Reviewed 5 of 8 files at r1, 3 of 3 files at r2.
Review status: all files reviewed at latest revision, 10 unresolved discussions.

Comments from Reviewable

georgiamoon · 2017-09-09T04:11:00Z

Review status: all files reviewed at latest revision, 10 unresolved discussions.

Comments from Reviewable

critzo · 2017-09-09T11:30:53Z

docs/deploy.md, line 34 at r2 (raw file):

Previously, georgiamoon (georgia bullen) wrote…

Is this the link for everyone or an example?

It's an example.

Comments from Reviewable

Chris Ritzo added 3 commits May 15, 2017 14:43

Updated docs

ce9d20f

formatting bullets

98f6371

updated some formatting

a340395

critzo added in progress labels May 15, 2017

georgiamoon reviewed May 23, 2017

View reviewed changes

meagdoh reviewed Jun 1, 2017

View reviewed changes

georgiamoon added this to the mozsprint milestone Jun 1, 2017

meagdoh reviewed Jun 2, 2017

View reviewed changes

Chris Ritzo added 2 commits June 26, 2017 12:58

updated docs and config file per reviewer comments

8853a12

Formatting tweak to table

998331c

critzo merged commit 6b9032d into m-lab:master Sep 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Piecewise docs #135

Update Piecewise docs #135

critzo commented May 15, 2017 •

edited by georgiamoon

georgiamoon May 23, 2017

critzo Jun 26, 2017

georgiamoon commented May 23, 2017

meagdoh Jun 1, 2017

critzo Jun 26, 2017

meagdoh Jun 2, 2017

critzo Jun 26, 2017

meagdoh Jun 2, 2017

meagdoh Jun 2, 2017

meagdoh Jun 2, 2017

meagdoh commented Jun 8, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Aug 29, 2017

georgiamoon commented Sep 9, 2017

georgiamoon commented Sep 9, 2017

georgiamoon commented Sep 9, 2017

critzo commented Sep 9, 2017


		2. Clone your fork onto your local computer

		```$ git clone https://github.com/opentechinstitute/piecewise.git```


		```
		```TO DO: add reference to international shapefile sources```


		##### Edit the main Piecewise configuration file
		Once you locate the shapefile(s) you need, it's good practice to open the shapefile in [QGIS](http://www.qgis.org/en/site/) or another GIS program to confirm that it meets the requirements to use within your Piecewise application. When you open the shapefile(s) in your GIS program, make a note of the name of its unique key field. You'll use that later in these instructions.


		This JSON file is arranged in sections with sub-elements. The relevant sections we will change begin with "aggregations" and "filters". Each section below begins with a code block from the relevant section, followed by instructions on what to change, and ending with the edited section of code. In our examples, we are using the folder and file names for our Baltimore example. Your folder and file names will likely differ.
		* Must contain at least one field that serves as a unique key, for example "geoid" or "geoid10" in the case of census block groups

Update Piecewise docs #135

Update Piecewise docs #135

Conversation

critzo commented May 15, 2017 • edited by georgiamoon

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgiamoon commented May 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meagdoh commented Jun 8, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Jun 26, 2017

critzo commented Aug 29, 2017

georgiamoon commented Sep 9, 2017

georgiamoon commented Sep 9, 2017

georgiamoon commented Sep 9, 2017

critzo commented Sep 9, 2017

critzo commented May 15, 2017 •

edited by georgiamoon