Skip to content

Commit

Permalink
Avoid cached response from proxy
Browse files Browse the repository at this point in the history
Trying to fetch the Yahoo Finance! homepage multiple times can result
in getting a cached response from a proxy. The cached response does
not have a "Set-cookie" header, so the curl handle will still not have
a cookie and the connection will be retried.

This StackOverflow Q/A has two approaches to avoid a cache hit:
https://stackoverflow.com/q/31653271/271616
1) Add a "Cache-control: no-cache" header
2) Add a random query to the URL.

The first approach didn't work because it appears the Yahoo proxies
simply ignore the request. The second approach did work though.

Fixes #166.
  • Loading branch information
joshuaulrich committed Jun 19, 2017
1 parent a7c57e2 commit 4aa0eee
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion R/getSymbols.R
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,10 @@ formals(loadSymbols) <- loadSymbols.formals

for (i in 1:5) {
h <- curl::new_handle()
curl::curl_download("https://finance.yahoo.com", tmp, handle = h)
# random query to avoid cache
ru <- paste(sample(c(letters, 0:9), 4), collapse = "")
cu <- paste0("https://finance.yahoo.com?", ru)
curl::curl_download(cu, tmp, handle = h)
if (NROW(curl::handle_cookies(h)) > 0)
break;
Sys.sleep(0.1)
Expand Down

0 comments on commit 4aa0eee

Please sign in to comment.