Add timeout option

[joeHickson]

Provide a (crude) mechanism for ending individual location processing after a timeout.
https://www.rdocumentation.org/packages/R.utils/versions/2.9.2/topics/withTimeout

Note (re: stan code keeping running):
"More precisely, if a function is implemented in native code (e.g. C) and the developer of that function does not check for user interrupts, then you cannot interrupt that function neither via a user interrupt (e.g. Ctrl-C) nor via the built-in time out mechanism. To change this, you need to contact the developer of that piece of code and ask them to check for R user interrupts in their native code."
I have found reference to Rstan checking for interrupts every iteration so we'll have to see what happens.

[joeHickson]

To give visibility of processes being killed this needs a mechanism to return the identity of any failures

[joeHickson]

good news! Rstan does indeed seem to be killed and die.
bad news - funnily enough the summary processing doesn't handle missing data very well!

@seabbs do you have any feeling about how we should represent No Data Available / timeout in the estimate summary? I think the regional epinow route is using the out object to hold the data so it doesn't re-load from disk, and it also has the full list of locations in the region.

[seabbs]

That is great news and yes I can imagine.

So at the moment, the expectation is that when read in from the out object subregions that have failed will have NULL. These then get thrown away in regional_summary and everything continues as normal. When using regional_summary and reading in from disk I think it may be less fault-tolerant (with this bit of code:

EpiNow2/R/summarise.R

Line 54 in fafb618

estimates <- purrr::map(regions, ~ load_data(file = "summary.rds", region = .))

potentially being an issue but this bit:

EpiNow2/R/get.R

Line 113 in fafb618

load_data <- purrr::safely(EpiNow2::get_raw_result)

being fault-tolerant). Similarly, a NULL in the output indicates any failure.

Could continue this for subregions that have been timed out with an informative log entry or add a message to the out object/into the results folder but this would then add a bit of a headache for results processing.

When saving it would make sense to save a killed due to time-out flag definitely.

When outputting results (we stick with this duality, by the way, to stay being CRAN compliant + acting like a normal package whilst also supporting saving to folders in order to make producing regional results more independent) we could add another list entry on the top level to the out object. At the moment it has $regional and $summary. Could also add $timeout and list subregions that have been timed out (could do the same thing for regions that have errored). If going with this approach I guess it would make sense to save this information into the summary folder (though that adds complexity in passing objects into regional_summary whilst trying to maintain interoperability between reading in from disk and passing in objects).

Sorry - a bit of ramble - happy to clarify any of the above that is unclear.

[joeHickson]

I have only put the timeout in the regional_epinow function which always returns a result set from the epinow call - this might mean we can sidestep the re-loading from disk issue.

[joeHickson]

@seabbs Please could you add @JAllen42 so he can put a bit of work in on this? Thanks

[seabbs]

Sorry yes, I missed that before. Done