Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RevGeo RAM issue #9

Closed
Ekibe opened this issue Jun 15, 2020 · 2 comments
Closed

RevGeo RAM issue #9

Ekibe opened this issue Jun 15, 2020 · 2 comments

Comments

@Ekibe
Copy link

Ekibe commented Jun 15, 2020

Thank you for the package! It is perfect for what I am trying to do.

I am using your revgeo package on about 15,000 records.

When running the package by itself:
city_per_rec <- revgeo(NRM$lon[i], NRM$lat[i], provider = 'photon', API = NULL, output = 'frame', item = 'city’)

I get through about 1000 records and it hangs up. I first thought it was photon throttling but later realized it was not. I now believe it is a RAM issue so I wrote a separate rscript:
##Library
library(rstudioapi)
library(revgeo)

##Load iteration
load('iter.RData’)

##City Reverse Geocoding
for (j in 1:a)
{
for (i in (b+1):(b+100))
{
if (!is.na(NRM$TOWN[i]) && !is.na(NRM$LONGITUDE))
{
city_per_rec <- revgeo(NRM$lon[i], NRM$lat[i], provider = 'photon', API = NULL, output = 'frame', item = 'city’)
NRM$ChCity[i] <- toupper(as.character(city_per_rec$city))
rm(city_per_rec)
}
}
b <- (b+100)
gc()
print('Taking a Break!’)
Sys.sleep(30)
gc() }

##Update iteration
iter <- iter+1 save(iter, file = 'iter.RData’)

##Save Workspace Image
save.image()

##Run This Script Again
if(iter < a) { restartSession(command = 'source("City.R")') }

This should clear the workspace, release the RAM, and restart R to run the code on 100 records at a time however this still hangs up my computer after about 1000 records.

I got through the data by manually doing chunks of 500 and restarting my computer so I am thinking it is a RAM fragmentation issue (running on only 6GB).

Are you able to provide any advise?

Thank you,
Ethan

@mhudecheck
Copy link
Owner

What operating system & version of R are you using? 1,000 records shouldn't be enough to fill more than a couple hundred MBs of ram at a time (if that), so my best guess is that something else is going on.

@Ekibe
Copy link
Author

Ekibe commented Jul 24, 2020

R version 3.6.3
Mac OS Sierra 10.12.6
2.4 GHz quad core
6 GB DDR3

A coworker is going to try running it next week so I'll update then.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants