topo grids folder only has Davis Mtns data

[dschwilk]

Is this correct? If so, we should move this to a subfolder and and the rest of the grids. But we need to deal with topographic data problems first (#19, #20)

[hpoulos]

Yes since that was the proof of concept site. We agreed that we were to build models separately for each range. I just have to clip the big grids down to size for BIBE and GUMO. They are made, but we should probably also talk about how much space that will take up in the repo. The daily grids will also take up Aton of space too.

[dschwilk]

Ok, good point. There are true "data", but we may run into logistical issues.
So if they are not expected to change, we can simply put them somewhere like
dropbox and refer to them from code. Read.csv can read in http addresses, and
with some work, https as well.

One example of using dropbox with the repmis library:
https://github.com/schwilklab/skyisland-climate/blob/master/scripts/wx-data.R

In current R that may not be strictly necessary as I think that read.csv can use
curl for https as needed, but I know that using repmis works.

On 10/18/2015 10:26 AM, hpoulos wrote:

Yes since that was the proof of concept site. We agreed that we were to build
models separately for each range. I just have to clip the big grids down to size
for BIBE and GUMO. They are made, but we should probably also talk about how
much space that will take up in the repo. The daily grids will also take up Aton
of space too.

—
Reply to this email directly or view it on GitHub
#22 (comment).

[dschwilk]

I think dropbox is probably the easiest for now. I can then just write a script that pulls that data down if it does not already exist locally and puts it into some data folder ignored by git. Just zip it all up and put it on dropbox for now. Maybe simple directory structure with directory for CM, DM, GM. The R code then just points to the local directory structure.

I looked into using git large file storage (git-lfs, https://git-lfs.github.com/). This is nice in that you can still version the files without making a monster repo, but I'm not sure how to use this and our own storage server yet. It is easy to use if we use github to store the large files, but our basic free schwilklab accounts allows only 1G of LFS space which is not enough, right? I could figure out how to use other storage sites with git-lfs (heck, I have server space galore as each one of my lab machines has a subdomain at ttu.edu and they are backed up nigthly) but we don't really need this. We can always move to git-lfs later and then we gain version control over the large files

[hpoulos]

|   |
|   |   |   |   |   |
| topo_grids.zipHelen Mills Poulos shared from Dropbox |
| |
| View on www.dropbox.com | Preview by Yahoo |
| |
|   |

Hi Dylan,I finally got all the grids to line up to standard extents in R. I ended up going back and reprojecting a grid from an earlier step and then the relelev_l and ldist_tovalley grids loaded fine in R.
I made a comment that I solved the grid issue for all but the wetness grid, which still had holes. I recalculated that grid since my last comment on the issue and reran the model, which ended up with a worse fit if wetness were included. I did that while I was waiting for the topo_grids.zip file to upload to Dropbox, so I think it's actually not neccessary to include wetness, since model performance drops by over 5% if included. I will assess that also with the other Mtn. Ranges, but for now, let's just leave it out. Here too, is the new sensors_topo.csv file with the correct topo data for each of the DM stations. 
I know you did some code cleaning, so please advise how I should proceed. I imagine I should fork from master and create a new branch since you said that you deleted the helenrfmodels branch. Please advise on that.
My next step is to get the loop I wrote to run to generate the daily tmins and tmaxs, and to then clip the BB and GM grids to standard extents so we can move on to the other ranges.
I also imagine that you plan to clean up the code now to have the files refer to grids on dropbox rather than on Git. Once you do that, let me know. 
Helen

 On Wednesday, October 21, 2015 10:43 PM, Dylan Schwilk <notifications@github.com> wrote:

I think dropbox is probably the easiest for now. I can then just write a script that pulls that data down if it does not already exist locally and puts it into some data folder ignored by git. Just zip it all up and put it on dropbox for now. Maybe simple directory structure with directory for CM, DM, GM. The R code then just points to the local directory structure.I looked into using git large file storage (git-lfs, https://git-lfs.github.com/). This is nice in that you can still version the files without making a monster repo, but I'm not sure how to use this and our own storage server yet. It is easy to use if we use github to store the large files, but our basic free schwilklab accounts allows only 1G of LFS space which is not enough, right? I could figure out how to use other storage sites with git-lfs (heck, I have server space galore as each one of my lab machines has a subdomain at ttu.edu and they are backed up nigthly) but we don't really need this. We can always move to git-lfs later and then we gain version control over the large files—
Reply to this email directly or view it on GitHub.

[hpoulos]

Loaded DM ascii topo_grids to dropbox now.

[dschwilk]

@hpoulos : I'm writing to code to extract topo var values for each sensor location -- easy enough to do if these are in same projection. But what is projection of the grid data? There is no .proj information any more with them. I see a corner designation and a cell size, but that appears to be in lat lon. Can that work? Don't we need a datum, etc to use this as a 2-d grid? I must be missing something.

[dschwilk]

@hpoulos : I guess since it is just lat lon I can just define a projection string and do it in R. I assume we want WGS84 as our datum. I can do that when we read in the data. Opening issue on this but I will fix.

[dschwilk]

DM grids are now in ./topo_grids/DM. We still need the others so that the code can run on all mtn ranges. meanwhile I'll revert the changes that allowed PCAs on all ranges.

[dschwilk]

@hpoulos sent dropbox links on Oct 30, 2015. But the different mtn ranges each had different numbers of ascii grids in these zip files:

> list.files(path="../topo_grids/DM", pattern = "*.asc", full.names=TRUE)
 [1] "../topo_grids/DM/elev.asc"         "../topo_grids/DM/flow_accum.asc"  
 [3] "../topo_grids/DM/flow_dir.asc"     "../topo_grids/DM/ldist_ridge.asc" 
 [5] "../topo_grids/DM/ldist_valley.asc" "../topo_grids/DM/msd.asc"         
 [7] "../topo_grids/DM/radiation.asc"    "../topo_grids/DM/relelev_l.asc"   
 [9] "../topo_grids/DM/relelev_shed.asc" "../topo_grids/DM/relelev_z.asc"   
[11] "../topo_grids/DM/slope.asc"        "../topo_grids/DM/wetness.asc"     
[13] "../topo_grids/DM/zdist_ridge.asc"  "../topo_grids/DM/zdist_valley.asc"
> list.files(path="../topo_grids/CM", pattern = "*.asc", full.names=TRUE)
 [1] "../topo_grids/CM/elev.asc"         "../topo_grids/CM/flow_accum.asc"  
 [3] "../topo_grids/CM/ldist_ridge.asc"  "../topo_grids/CM/ldist_valley.asc"
 [5] "../topo_grids/CM/msd.asc"          "../topo_grids/CM/radiation.asc"   
 [7] "../topo_grids/CM/relelev_l.asc"    "../topo_grids/CM/relelev_shed.asc"
 [9] "../topo_grids/CM/relelev_z.asc"    "../topo_grids/CM/slope.asc"       
[11] "../topo_grids/CM/wetness.asc"      "../topo_grids/CM/zdist_ridge.asc" 
[13] "../topo_grids/CM/zdist_valley.asc"
> list.files(path="../topo_grids/GM", pattern = "*.asc", full.names=TRUE)
 [1] "../topo_grids/GM/elev.asc"         "../topo_grids/GM/flow_accum.asc"  
 [3] "../topo_grids/GM/ldist_ridge.asc"  "../topo_grids/GM/msd.asc"         
 [5] "../topo_grids/GM/radiation.asc"    "../topo_grids/GM/relelev_l.asc"   
 [7] "../topo_grids/GM/relelev_shed.asc" "../topo_grids/GM/relelev_z.asc"   
 [9] "../topo_grids/GM/slope.asc"        "../topo_grids/GM/wetness.asc"     
[11] "../topo_grids/GM/zdist_ridge.asc"  "../topo_grids/GM/zdist_valley.asc"

Looks like CM is missing "flow_dir" and GM is missing "flow_dir" and "ldist_valley"