-
I have 0 chance to post a reproducible example here
-
What happened:
-
In a large benchmark, for some reason, loading a dependency failed somewhere.
-
When we call getStatus() (or anything else), bt fails with
Syncing 3677 files ...
Error in rbindlist(updates) :
Item 22 has 5 columns, inconsistent with item 1 which has 6 columns. If instead you need to fill missing columns, use set argument 'fill' to TRUE.
-
this destroys all measures to further use results or status
-
I have the registry locally on my machine and can debug, and reproduce there
-
it failed with bt version 0.9.3, but I guess this is still present in the current GH version
- where do we fail:
> traceback()
7: rbindlist(updates) at syncRegistry.R#32
6: syncRegistry(reg = reg) at loadRegistry.R#78
5: loadRegistry("thresholdTuning-benchmark-files", work.dir = "~/cos/tune_threshold_benchmark/src",
update.paths = TRUE) at testbt.R#3
4: eval(ei, envir)
3: eval(ei, envir)
2: withVisible(eval(ei, envir))
1: source("testbt.R")
the code is
updates = lapply(fns, function(fn) {
x = try(readRDS(fn), silent = TRUE)
if (is.error(x)) NULL else x
})
failed = vlapply(updates, is.null)
updates = rbindlist(updates)
here is what is in "updates"
> uu[[1]]
job.id started done error memory written
1: 2316 1.502e+09 1.502e+09 NA NA FALSE
> uu[[22]]
job.id started done
1: 1827 1.502e+09 1.502e+09
error
1: Error loading registry dependencies: Error in loadRegistryDependencies(jc, switch.wd = FALSE) : \n Failed to load packages: OpenML
memory
1: NA
you can see the problem: the element 22 lacks column "written"
- what might be wrong - I am reading the code and changing bt src code, but I need to guess here to some extent
#' @export
doJobCollection.JobCollection = function(jc, output = NULL) {
now = function() strftime(Sys.time())
error = function(msg, ...) {
now = ustamp()
updates = data.table(job.id = jc$jobs$job.id, started = now, done = now,
error = stri_trunc(stri_trim_both(sprintf(msg, ...)), 500L, " [truncated]"),
memory = NA_real_, key = "job.id")
writeRDS(updates, file = file.path(jc$file.dir, "updates", sprintf("%s.rds", jc$job.hash)))
invisible(NULL)
}
in the above code, we never set "written" to anything.
later we seem to exit here
# load registry dependencies: packages, source files, ...
# note that this should happen _before_ parallelMap is initialized
ok = try(loadRegistryDependencies(jc, must.work = TRUE), silent = TRUE)
if (is.error(ok))
return(error("Error loading registry dependencies: %s", as.character(ok)))
if this only happens in some jobs, but not all --> we are fucked
I have 0 chance to post a reproducible example here
What happened:
In a large benchmark, for some reason, loading a dependency failed somewhere.
When we call getStatus() (or anything else), bt fails with
this destroys all measures to further use results or status
I have the registry locally on my machine and can debug, and reproduce there
it failed with bt version 0.9.3, but I guess this is still present in the current GH version
the code is
here is what is in "updates"
you can see the problem: the element 22 lacks column "written"
in the above code, we never set "written" to anything.
later we seem to exit here
if this only happens in some jobs, but not all --> we are fucked