Fetch data from the Tiingo API with a getSymbols.tiingo() method.

[SteveBronder]

Commit Message:

This enables package users to download historical OHLC, Adjusted OHLC,
Dividend, and Split information from the Tiingo API. Tiingo's free
offering allows access to 60K global securities for 30+ years, though with some rate limits. Tiingo
provides daily, weekly, monthly, and annual frequencies.

Details of the service can be found at the following links
https://www.tiingo.com/pricing
https://api.tiingo.com/docs/tiingo/daily

Tiingo also provides fundamentals, mutual fund information, aggregated news feeds, crypto, and a 5 minute delay quote system, though this implementation includes no provision for downloading them.

Me:

I've been searching for good free data online and Tiingo appears to be not bad! Possibly, good! A big thing for me was that they clearly lay out the methodology for adjustments and give you the data to verify the adjustment. See the bottom of this page
https://api.tiingo.com/docs/tiingo/daily

The rate limits are reasonable for good free data. If I need more then I'll be fine with paying $10 bucks a month.

There is also a websocket API which would be cool to use, though I didn't feel like learning how to use that.
https://api.tiingo.com/docs/general/connecting

I was making this for myself and thought to submit a PR in case you wanted it. It's essentially just a copy/paste of Paul's Alpha Vantage code

[SteveBronder]

So after pulling about 6K stocks from Tiingo I can say the data quality seems pretty good! Running some basic tests there were only about 50 OTC penny stocks which had oddities. Two stocks had negative adjusted values though the unadjusted looked fine. I still need to do a deeper dive, but as of now I think the quality is nice and would be good for users of quantmod.

[tiingo]

Hi All,

This is Rishi - founder of Tiingo. The way our system works is that we combine 3-5 data providers for each ticker. We've done this because we've found every vendor misses splits, dividends, or gets the closing values incorrect. We also do this because some vendors have outages. We have a system that runs suites of tests on every data point to help track errors. The additional benefit is that we can keep track of which vendors are good/not-so-good and swap them out.

In the case of Steve's checks - yes we found one vendor was giving us negative values and removed them awhile ago for other reasons. Our systems were able to squash most of the negative values, but a few got through - so 19 of our 60,000 tickers had these values. We've since applied a check and removed those values, so going forward it will not be an issue.

Sometimes we feel like historians - tracking down a merger that happened 10-15 years ago and correcting for issues. And when we do manually override points, we keep an audit log with all the notes from our calculation.

The point of Tiingo is to build the data quality mechanism we always wanted when quant trading. It's taking time but because of the quant community we're rolling through :) Thanks all and thanks @SteveBronder for the PR!

[joshuaulrich]

Thanks for the contribution; this is a great start! I need to get an update to CRAN within a couple weeks, and I plan to include this. I'm going to make a few changes, including making @SteveBronder the author instead of Paul.

I'm also going to add an adjust (TRUE / FALSE) argument, and return only adjusted or unadjusted OHLCV values. Returning both will cause trouble for the extractor functions (e.g Cl(), OHLC(), etc.).

@tiingo, is there a way request only the unadjusted values, or only the adjusted values? Adding that functionality would be useful for me, and would lower user bandwidth. It would also allow users to calculate the adjusted prices themselves, which should allow them to get greater accuracy with lower bandwidth. Also, is there a documented example of an "over-the-limit" response? I couldn't find one on your website, and I would like to provide users an informative error if they hit a limit. Thanks!

[tiingo]

Thanks all for the hard work. @joshuaulrich - would you mind if we implemented this via allowing users to pass the columns they would like? That way we could make this use case more general, and to get just unadjusted prices we could pass the following:

Unadjusted:
&columns='open','high','low','close','volume', 'divCash', 'splitFactor'

Adjusted:
&columns = 'adjOpen', 'adjHigh', 'adjLow', 'adjClose', 'adjVolume', 'divCash', 'splitFactor'

I'm partial to this method as it can allow a more general use case.

Let me know if this would work!

I made an API Token that you can use which will return a usage error:
d116c846835e633aacedb1a31959dd2724cd67b8

JSON: https://api.tiingo.com/tiingo/daily/spy/prices?startDate=2017-1-1&token=d116c846835e633aacedb1a31959dd2724cd67b8
CSV: https://api.tiingo.com/tiingo/daily/spy/prices?startDate=2017-1-1&format=csv&token=d116c846835e633aacedb1a31959dd2724cd67b8

Thanks Joshua

Rishi

[joshuaulrich]

Thanks for the prompt reply, and the API token to generate a rate limit response! I also prefer more general functionality, so allowing the user to specify all the columns to return would be great!

[tiingo]

Of course - the community building this helps us tremendously. Please keep letting us know what we can do.

The ?columns parameter has been added.

Unadjusted values:
https://api.tiingo.com/tiingo/daily/spy/prices?startDate=2017-1-1&columns=open,high,low,close,volume,divCash,splitFactor

Adjusted values
https://api.tiingo.com/tiingo/daily/spy/prices?startDate=2017-1-1&columns=adjOpen,adjHigh,adjLow,adjClose,adjVolume,divCash,splitFactor

To get the data as CSV, append &format=csv, e.g.
https://api.tiingo.com/tiingo/daily/spy/prices?startDate=2017-1-1&columns=open,high,low,close,volume,divCash,splitFactor&format=csv

Hope this helps!
Rishi

[SteveBronder]

Thanks @tiingo and @joshuaulrich !

Josh if you want to make the changes then by all means, else I can do this later in the week

[SteveBronder]

Josh we can throw this out if you don't like it. I wanted some form of argument assertion, though I usually use the checkmate package. The function above, such as the check on adjusted, returns an error if the argument value is not in the set. The error looks like

adjusted must be a value of TRUE, FALSE, both

[joshuaulrich]

What's the benefit of this over match.args()?

[SteveBronder]

I reinvented the wheel here. Good catch will fix

[joshuaulrich]

I'm actually in the middle of rebasing and cleaning up. I can take care of it.

[SteveBronder]

Ah, there is one edgecase

match.arg(TRUE, c(TRUE, FALSE, "both"))
# Error in match.arg(TRUE, c(TRUE, FALSE, "both")) : 
#  'arg' must be NULL or a character vector

but I think I would rather have

adjusted = match.arg(as.character(adjusted), c("TRUE", "FALSE", "both"))
# arg returns a char so we do conversion back to logical
if (adjusted == "TRUE" | adjusted == "FALSE") {
    adjusted = as.logical(adjusted)
}

or

_ = adjusted %in% c(TRUE, FALSE, "both") || 
    stop(paste0("Periodicity must a value of TRUE, FALSE, or 'both'"))

than my function. I'm going to do (1) unless you have a preference or alternative

[joshuaulrich]

My preference is that adjust is logical only. No support for "both" because that's not provided by any other getSymbols() functions and I don't believe it's a common use case.

I would also prefer to take care of it myself. I am amending some of your additions because they use different style/convention than the rest of the code (e.g. = for assignment, underscores in names, etc). Some of your commits include changes from prior commits in master that you didn't have because your branch is behind upstream/master. You also have merge commits, and I have a strong preference for rebasing versus including multiple merges of upstream/master into a feature branch.

[SteveBronder]

My preference is that adjust is logical only. No support for "both" because that's not provided by any other getSymbols() functions and I don't believe it's a common use case.

Yeah I'm cool with that

I would also prefer to take care of it myself.

Totes! If you need anything from me lmk.

I am amending some of your additions because they use different style/convention than the rest of the code (e.g. = for assignment, underscores in names, etc).

I'd rather not waste your time and would be happy to fix these things up myself.

Some of your commits include changes from prior commits in master that you didn't have because your branch is behind upstream/master. You also have merge commits, and I have a strong preference for rebasing versus including multiple merges of upstream/master into a feature branch.

Yes I saw that on the contributor guide, rebased at first but then was waiting for you to look things over before I rebased pre-merge to master. I just rebased / squashed everything

[SteveBronder]

Looking at the url from the test's call it looks like google finance is down

http://finance.google.com/finance/historical?q=SPY&startdate=Mar+27,+2018&enddate=Apr+06,+2018&output=csv

Edit: fixed

[joshuaulrich]

@tiingo, I wrote some tests for the rate-limit error. I receive a HTTP 429 status with the JSON API, but 200 with the CSV API. Both payloads contain the error message. Shouldn't the CSV response status be 429 also?

[tiingo]

@joshuaulrich Yes, absolutely it should be a non-200 error code; however, we found that for some platforms, the data may not render in CSV format if an error response is received, so the user could be left with an error with no descriptive message. It was a difficult decision we had to make, and ultimately we decided to keep the message as a 200 response, even with error, so the user could receive some feedback on corrective action. I'm hesitant to change the behavior any time soon as I don't want to break anybody's code without sufficient notice or a LTS migration plan.

I understand the solution is not technically ideal, but generally, we made it a policy that JSON is our preferred method to deliver data, but many people wanted the data in CSV format so we obliged. For some of our APIs, like the crypto one, CSV loses data because we cannot nest or relate data in together. As we are expanding data sources, JSON will be our preferred method because of the flexibility.

Thanks to you and Steve for working on this.

[joshuaulrich]

@tiingo That's completely understandable, and I assumed something like that was happening. I wanted to verify, just in case. I also share your reservations about potentially breaking user code, so I'll figure out a work-around.

I would like to talk with you about a few ideas I have. If you're interested, you can reach me via email. My address is in the DESCRIPTION file of this repo.

[tiingo]

Emailed! Look forward to hearing them

[Aks1623]

Joshua, adjusted close price is different when i get it via different methods.. in getsymbols, adjusted close is actually adjusted open..

riingo <- riingo_prices(c("IBM"), start_date = "2018-04-21", end_date = Sys.Date(), resample_frequency = "daily")
getSymbols("IBM", src= "tiingo",from= "2018-04-21", to= Sys.Date(),adjust =TRUE)

thanks

[SteveBronder]

EDIT: Apologies I was using an older branch

Yes seems there is a bad match. I'll make an issue and PR for this. Thanks for the catch!

[PCWin12]

hello,

just downloaded quantmod and registered at tiingo, how do i get an api.key?
the default value for getFin is set to google and reports an error.
Thank you