Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

atoc2gtfs fails because makeCalendar downloads all data in each thread #61

Open
mpadge opened this issue Oct 18, 2023 · 1 comment
Open

Comments

@mpadge
Copy link
Contributor

mpadge commented Oct 18, 2023

This package is cleverly set up to download all of the release data each time it is loaded. But that means these lines can fail:

UK2GTFS/R/atoc_export.R

Lines 386 to 388 in 5094f1f

parallel::clusterEvalQ(cl, {
loadNamespace("UK2GTFS")
})

In my case, I've got 36 threads, and those lines cause each thread to download the full dataset, which then maxes out my available bandwidth, causing failure of threads and therefore the whole routine. (I can get around this by running fewer threads, but would prefer a proper solution to this.)


Edit: Actually, that whole call to checkrows(res.calendar.days) takes hardly any time even in single threaded, so maybe the easiest solution would be just removing the multi-core version there? Everything else then works fine.

@mem48
Copy link
Contributor

mem48 commented May 6, 2024

This is lazy coding on my part, it is not necessary to load the whole package so this can be fixed by specifying the relevant functions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants