Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

280 remove file io #281

Merged
merged 11 commits into from
Nov 3, 2019
Merged

280 remove file io #281

merged 11 commits into from
Nov 3, 2019

Conversation

ethanbsmith
Copy link
Contributor

@ethanbsmith ethanbsmith commented Oct 17, 2019

removed disk i/o from critical path of functions that fetch data form the internet. these functions now operate entirely in memory

main drivers for this:

  1. the original process created significant contention with other disk i/o operations. this was particularly apparent on large batch downloads with older, slower disks and database servers
  2. the implementation pattern was inconsistent. some functions used an entirely in-memory path, while others used disk i/o with temporary files. all functions now consistently use the in-memory pattern

Fixes #280

ethanbsmith and others added 6 commits September 23, 2018 09:58
This reverts commit 9cf41d9, reversing
changes made to b0ab966.
repalced all for download.file and curl_fetch_disk. left  calls in loadSymbol.r alone
@ethanbsmith ethanbsmith marked this pull request as ready for review October 17, 2019 20:00
@ethanbsmith ethanbsmith reopened this Oct 17, 2019
@joshuaulrich
Copy link
Owner

Hi @ethanbsmith, thanks for working on this PR! Is it ready for review?

@ethanbsmith
Copy link
Contributor Author

ethanbsmith commented Oct 26, 2019 via email

@joshuaulrich joshuaulrich merged commit 1739987 into joshuaulrich:master Nov 3, 2019
@joshuaulrich joshuaulrich added this to the Release 0.4-16 milestone Feb 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

remove disk i/o from getSymbols and getQuote functions
2 participants