> 100% complete values from the GDP break the progress bar

[wdwatkins]

stencil <- geoknife::simplegeom(c(-88.6, 45.2))
fabric <- geoknife::webdata(url = 'https://cida.usgs.gov/thredds/dodsC/stageiv_combined', 
                            variables = "Total_precipitation_surface_1_Hour_Accumulation", 
                            times = c("2008-06-02", "2008-06-13"))
knife <- geoknife::webprocess("unweighted summary")
job <- geoknife::geoknife(stencil, fabric, knife, wait = TRUE, 
                          REQUIRE_FULL_COVERAGE=FALSE)

The progress bar function (from progress package) contains an assertion that the percent complete ratio is < 1, which fails. We could pre-empt this to give a nicer error message, but this only happens with the unweighted algorithm which generally shouldn't be used anyways.

There is something in the GDP itself that is actually causing the wild percent complete values.

[jordansread]

When this failure happens, is this the message?
Error: !self$finished is not TRUE?

[jordansread]

I think we need to catch some of the issues w/ progress
e.g.,

Error in pb_update(self, private, ratio, tokens):
!self$finished is not TRUE

[gjahansen2]

This error is a problem when pulling data for a large number of lakes within a loop to loop over years. It only causes a problem for a large number of lakes (I am not sure what the threshold number is). 20 lakes work. 12000 lakes does not work. So somewhere in between :)

sites=data.frame("id"=seq(1,12000), "lat"=seq(40, 48, length.out = 12000), "lon"=seq(-102, -88, length.out = 12000))

#set up geom
for (i in 1:length(sites$id)){
  df <- data.frame(c(sites$lon[i], sites$lat[i]))
  names(df) <- sites$id[i]
  if (i == 1)
    geom <- df
  else 
    geom <- cbind(geom, df)
}


#loop to pull data for july of each year in mid-century projections for each GCM
###############################################################
#all july, all years, all lakes
knife <- webprocess(REQUIRE_FULL_COVERAGE = FALSE, wait = TRUE)

#loop over GCMs
gcm.list=c("ACCESS", "CNRM", "GFDL", "IPSL", "MIROC", "MRI")

for(j in 1:length(gcm.list))
{
  current.gcm=gcm.list[j]
  webname=paste("dods://cida.usgs.gov/thredds/dodsC/notaro_",current.gcm,"_2040_2059", sep="")

#loop over years
for(i in 2040:2059){
timelist.temp=c(paste(i,'-07-01T00:00:00.000Z',sep=""),paste(i,'-07-31T23:00:00.000Z', sep=""))

wd <- webdata(url = webname, variables = "tas", times = timelist.temp)

newjob<- geoknife(geom, wd, knife)

mydata=result(newjob, with.units=TRUE)
table.name=paste(current.gcm,i, "_midcentury_july_real.csv", sep="")
write.csv(mydata, table.name)
}#end year loop
  
}#end gcm loop

produces this error at first iteration after progress bar goes to 100, although job is done and can be accessed via web server.

Error in pb_update(self, private, ratio, tokens) : 
  !self$finished is not TRUE

[gjahansen2]

with smaller numbers of lakes (4000-6000), the looping code works but not reliably. It will work for a few years and then fail with the same error.

[wdwatkins]

I think this is a different problem than what this issue was created for --- it looks like the error you and Gretchen are getting is because checkResult is returning 100% twice in a row (I assume just because something in the GDP hasn't happened fast enough), and at the second 100% the second time the progress bar has already finished, so updating it causes an error. Haven't tested this yet, but I think we can just make a check so that the progress bar doesn't update if a 100% value has already been returned.

This may solve the original issue too.

[jordansread]

oh, ok - should we create another issue for this issue?

[jordansread]

Thank you for the detective work. To clarify, it returns a 100% but not a

$statusType
[1] "ProcessSucceeded"

so it checks again later?

[jordansread]

@gjahansen2 as a quick fix, I bet

gconfig('sleep.time' = 45, 'wait' = TRUE)

will solve your issue for the time being. Default is for geoknife to check w/ the GDP every 5 seconds. We know your job is slower than that, so let's delay for 45 seconds in between checks. Haven't had time to test this though...

[gjahansen2]

OK, so I would run that code prior to the loop and it will set the configuration for all subsequent geoknife jobs? Or do I put that within the geoknife call? In other news, everything is trucking along just fine for ACCESS years 2045-2055. Best run yet.

…

On Fri, Sep 21, 2018 at 1:34 PM Jordan S Read ***@***.***> wrote: @gjahansen2 <https://github.com/gjahansen2> as a quick fix, I bet gconfig('sleep.time' = 45, 'wait' = TRUE) will solve your issue for the time being. Default is for geoknife to check w/ the GDP every 5 seconds. We know your job is slower than that, so let's delay for 45 seconds in between checks. Haven't had time to test this though... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#366 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASiU6oebf2TnFFpLipBqxJfge0FY9Roaks5udTFAgaJpZM4RgZyF> .

-- Gretchen Hansen https://gretchenhansen.squarespace.com

[wdwatkins]

@jread I'm rerunning to check. The code doesn't check statusType though: https://github.com/USGS-R/geoknife/blob/master/R/waitUntilFinished.R#L45

Adding that may be the fix.

@gjahansen2 Yes, run it prior to the loop. That will set that option for the rest of your R session.

[wdwatkins]

Yep, looks like that is the issue. This is just printing parts of what check returns:

89"
[1] "ProcessStarted"
[============================================================================================================>-------------]  89%[1] "94"
[1] "ProcessStarted"
[==================================================================================================================>-------]  94%[1] "100"
[1] "ProcessStarted"
[==========================================================================================================================] 100%
[1] "100"
[1] "ProcessSucceeded"
 Hide Traceback
 
 Rerun with Debug
 Error in pb_update(self, private, ratio, tokens) : 
  !self$finished is not TRUE 

[gjahansen2]

@wdwatkins so you are saying Jordan's fix should work? I will give it a try. Right after I said everything was working it failed again.

[wdwatkins]

Yes, it should prevent it for now. We should be able to have a real fix out by Monday though.

[wdwatkins]

@gjahansen2 We just merged a fix in the package that should solve your issue. You can install the latest version with the fix with devtools::install_github('USGS-R/geoknife'). (you may need to install the devtools package if you don't have it already).

[lindsayplatt]

Seeing this issue while pulling the new debiased GCMs down (http://gdp-netcdfdev.cr.usgs.gov:8080/thredds/dodsC/notaro_debias_access) for 24 out of my 54 queries. I can bypass this for now by adding gconfig('sleep.time' = 45, 'wait' = TRUE) but that is not an ideal solution.

[dblodgett-usgs]

Can you share a reprex? I'll see if I can get it fixed.

[lindsayplatt]

I was struggling to do this because the geom I am using has 69 points and choosing just 1 doesn't recreate the issue. I've attached a small CSV that is used in this code.

lplatt_query_geom.csv

library(tidyverse)
library(sf)
library(geoknife)

# Reproject query cell centroids to WGS84
sf_pts_to_simplegeom <- function(sf_obj) {
  sf_obj %>%
    st_coordinates() %>%
    t() %>% as.data.frame() %>%
    # Include the cell numbers in the GDP geom so that the
    # results can be linked to the correct cells
    setNames(sf_obj$cell_no) %>%
    simplegeom()
}
query_simplegeom <- readr::read_csv('lplatt_query_geom.csv') %>% 
  st_as_sf(coords = c("X", "Y")) %>% 
  sf_pts_to_simplegeom()

gcm_job <- geoknife(
  stencil = query_simplegeom,
  fabric = webdata(
    url = "http://gdp-netcdfdev.cr.usgs.gov:8080/thredds/dodsC/notaro_debias_access",
    variables = c(
      'windspeed_debias', 
      'tas_debias', # air temp
      'rsds_debias', #shortwave radiation
      'rsdl_debias', # longwave radiation
      'rh_debias', # relative humidity
      'prcp_debias'), # precip
    times = c('1980-01-01', '2100-01-01')
  )
  # The default knife algorithm is appropriate
)

# This throws an error
wait(gcm_job)

# But when you check on this, it succeeded
check(gcm_job)$statusType

[jordansread]

Hope this example is adequate. I wonder if you can trigger the error to by setting sleep.time in the gonfig down to 0.2 or something really small to cause a higher likelihood of hitting the error even with a smaller request.

[dblodgett-usgs]

Pending the changes in #406 I think you will be good to go @lindsayplatt

I don't actually think this was a status > 100% issue. It seems like we were trying to set the progress bar to 100% more than once but it was ending up in a state where it didn't want to update anymore after hitting it's maximum already.

What I did will also handle the case where progress goes over 100% by just not updating.

[dblodgett-usgs]

Side note, that csv has some funky characters in the first an last lines... not sure how you generated it, but I had to strip some characters...

[lindsayplatt]

Oh hmm sorry about the file funkiness. Not sure what that would have been from. I made the CSV by exporting the st_coordinates() of a geom I had loaded into R so that I could attach a file to a GitHub comment. Shouldn't impact me at all since I will not be using the CSV. This doesn't indicate that there are funky lines though? Hmm wonder what happened to it

readLines('lplatt_query_geom.csv')
 [1] "X,Y,cell_no"                                
 [2] "-96.21941006058987,43.5205760557457,13782"  
 [3] "-95.9072056134748,43.51798718988345,13783"  
 [4] "-95.59503268846021,43.514535543265445,13784"

[lindsayplatt]

Just checked with the development version and this did fix the issue I was having (which sounds different than what was noted in this issue originally). Thanks!