New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Version 2 #403

Open
joshuablackburn-iex opened this Issue Jun 28, 2018 · 55 comments

Comments

Projects
None yet
@joshuablackburn-iex
Contributor

joshuablackburn-iex commented Jun 28, 2018

All,

While we continue to fix issues and improve version 1 of our API, we have been hard at work on version 2. We've learned a lot since we launched last year, and we plan to take those lessons and make an even better API. There are some very exciting developments in version 2, some we cannot disclose at this time, but hopefully we will have more announcements soon. Thank you to those that have stuck with us and have patiently waited for improvements to the API. We are committed to making a sustainable and reliable platform that you can depend on.

Some of the focus for v2 that we can disclose now:

  • Broader data sets
  • More accurate 3rd party data
  • Improved WebSocket support
  • SSE
  • Official SDKs
  • Normalizing data structures
  • Better API versioning
  • More organized documentation
  • Improved latency

Things we are interested in and investigating:

  • GraphQL
  • Binary format in addition to JSON and CSV
  • Widgets
  • Better support for Excel / Google Sheets

One thing we are changing for version 2 is a requirement to use an API key. This should help us provide a better experience for all users.

What are your thoughts about the API key requirement? Will that negatively impact your workflow?

More to follow. Stay tuned.

@McQgit

This comment has been minimized.

McQgit commented Jun 28, 2018

I don't mind an API key. Other fin services (Quandl, for one) require them as well. But, it does make a few things more difficult. Posting url's to github requires care to redact the key, for example. Another is sample links in your docs will need a valid key so the example works. Or, be clever and dynamically set your sample pages to use my logon key in your sample url's (like Quandl). I found it convenient to quickly copy your doc url to a test program but that might need an extra step to add syntax for my key. These are all fairly minor, but you asked :)

I only use programmatic api but I'm curious how you'd handle the Excel interface for keys. Will you require the key in the cell formula directly or have it in a config? What about people sending an xlsx to a friend? I don't care which answer - just thinking of possible future uses.

@kapitan-k

This comment has been minimized.

kapitan-k commented Jun 29, 2018

I’m increasingly using your API and I do not oppose an API key.

About format (I know you did not ask for opinions about it): Personally I’m a big proponent of protocol-buffers (used it in many projects). Clear format, easy versioning, multi language support, high performance. You could simply provide .proto files and everyone could generate the code in the preferred language. = No guessing game about returned structures.

@flashover

This comment has been minimized.

flashover commented Jun 29, 2018

Hi Joshua,

I've built a wrapper to query your API so it won't be a problem for me to integrate the API-key.

I'm mostly looking forward to more accurate data, as I've seen some errors with EPS / Dividend information on some (even popular) stocks. Speed of the API won't be an issue for me, as I'm caching entries for a little while before refreshing them.

Cheers,

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jun 29, 2018

@kapitan-k I do like protocol-buffers. I also like the idea of returning a schema dynamically. I've considered adding JSON Schema.

Another thing being considered is bringing our API docs to this repo so contributions can be made.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jun 29, 2018

@McQgit Thanks for the feedback. My main goal is to keep the platform simple and easy to use. You make valid points for each use case of an API key, which is one of the main reasons we currently do not have them. Going forward, the ambition we have for the platform will require keys to both ensure stability for all users, and to enable personalized services. We will make sure to consider cases where a developer needs to test an endpoint. We are also working on solutions for Excel.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jun 29, 2018

@flashover Data accuracy is very important to us. I think everyone will be more than pleased with what we are working on.

@SeamusBoyle

This comment has been minimized.

SeamusBoyle commented Jun 29, 2018

@joshuablackburn-iex I'm not opposed to an API key requirement, particularly for personalisation purposes (e.g. Saving a watchlist to IEX servers).

The demo version of an app I have developed is available for users to try before buying. So I would like to see the API permit requests without an API key, otherwise, it is very time-consuming for my apps users just to run the demo. Maybe the API could support non-API key requests limited by requests-per-minute or CPU-time-per-hour (counted against users IP).

For binary format, I prefer msgpack over protobuf. Switching a client app from JSON to MsgPack is normally a single line of code change. Protobuf requires additional compilation, and code generated from proto files is large. I think msgpack+JSON+CSV would also be easier to maintain server-side compared to proto+JSON+CSV.

I'm currently writing a client SDK for a time series database that uses msgpack, it embeds raw binary data as columns, one column per msgpack field so you read the entire array, not each element.

RE one-minute candles. I like how the one-minute data is returned one day at a time. It makes it super easy to cache and resample. I also like how null bars are included (for when no trades occurred in the minute), these bars are impossible to insert client-side without an accurate calendar. When designing v2/candles please don't change these two things. If v2 can return minute data spanning multiple days please divide them by trading day in the response. The trading day can sometimes span midnight (e.g Forex, Crypto data from Asian exchanges).

RE websocket (non-socket.io). If not using msgpack or protobuff, please consider gzip+json. This could be an option in the query string when connecting. This request is mainly if the new websocket can be used to make requests that return large responses (e.g. candles, batch quotes).

@WojciechZankowski

This comment has been minimized.

WojciechZankowski commented Jun 30, 2018

@joshuablackburn-iex

Official SDKs

Looks like I am out of the business 😄 Will be sad moment because I engaged a bit already.

But will v1 be still there or you plan to remove it?

@bkcollection

This comment has been minimized.

bkcollection commented Jul 1, 2018

Joshua,

How the api key affected a website that use your API in version 1?
Will there be a limitation of usage for v2 once the API key is implemented? v1 allow developer to use the API freely even for distribution

Does it mean distribution is not allowed in v2 anymore? Please clarify

Besides, what is the roadmap for v2? When will it be officially launched and will v1 remained there?

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 2, 2018

@bkcollection We do not have an official release date for V2 yet. Some things need to get finalized first, then we will announce. The plan is to keep v1 around for 6 months after the launch of v2. We also have an approach to handle client side API use through the use of domain specific client keys, but we will put out more detail later. The API data will remain free to distribute.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 2, 2018

@WojciechZankowski Not necessarily. We could pick an existing library to be officially promoted.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 2, 2018

@SeamusBoyle Great feedback. Thank you.

@yccheok

This comment has been minimized.

yccheok commented Jul 2, 2018

I hope that old API will stay around longer. As, for a small development team, we may not have adequate resource to perform the migration. Thank you.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 3, 2018

@yccheok Is 6 months not enough time to switch to v2?

@bkcollection

This comment has been minimized.

bkcollection commented Jul 3, 2018

Hi Joshua,

is the v2 different from v1 as below?
https://api.iextrading.com/1.0/stock/aapl/financials
https://api.iextrading.com/2.0/stock/aapl/financials?API_KEY=xxxx

If this is the only difference, it should be not hard to migrate

@metbril

This comment has been minimized.

metbril commented Jul 7, 2018

@bkcollection I don't like my API key in the query string. I prefer in an HTTP header.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 11, 2018

@metbril Sending in the header is definitely a common practice, and we will support it. We do intend to continue support GET requests, and a key can be passed that way. More details of how that will work will come later.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 11, 2018

We receive a lot of questions about supporting additional data sets. I thought I'd share some of the data we are investigating and hoping to bring to version 2. None of these are final, but should give a sense for what we would like to have:

Full day coverage of 15 minute delayed US equities
International stocks
Currencies (FOREX)
Mutual fund data
Options data
Commodities data
Index data
OTC data
Treasuries data
Analyst recommendations, ratings, upgrades/downgrades
Expanded fundamentals

Hopefully some additional interesting alternative data sets.

@justin-shaw

This comment has been minimized.

justin-shaw commented Jul 12, 2018

@joshuablackburn-iex
Any plans to have pre-post market data in the chart endpoints for version 2? Also curious about technical indicators being added in(ema, sma, rsi, psar etc..), are any of those being considered? New to the IEX API as well, excellent work btw on v1 .. can't wait to see what comes from v2.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 12, 2018

@justin-shaw Pre/Post is definitely something that will be available. Technical indicators are being looked at. If not at launch, it'll eventually make it in.

@jjttjj

This comment has been minimized.

jjttjj commented Jul 13, 2018

Some sort of machine readable schema such as OpenAPI would be great to have. Does anyone have one for V1? I'm starting to attempt to put one together.

@bkcollection

This comment has been minimized.

bkcollection commented Jul 13, 2018

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 13, 2018

@jjttjj I agree. We are looking to support a schema.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 13, 2018

@bkcollection Full day coverage for US equities means 4am-8pm ET. Right now our systems at IEX are only online from 8am-5pm ET, so we only publish prices during those hours.

@bkcollection

This comment has been minimized.

bkcollection commented Jul 15, 2018

josh,

Is there any plan for full day real time quote instead of 15 minutes delay?

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Jul 16, 2018

@bkcollection IEX will continue to provide real time pricing from 8-5pm. For 4am-8am and 5pm-8pm, we have to stick to delayed for now. Very few places display real time pricing after regular market hours due to very thin trading. Real time pricing can also get very expensive not only for us to distribute, but for you the user if you need to do anything with the data. We are looking into as many options as possible while trying to keep the cost and legal paperwork to a minimum for our users.

@lovettchris

This comment has been minimized.

lovettchris commented Jul 21, 2018

API key is fine by me, pretty standard these days. Just put it in the HTTP header though so it is encrypted with https payload. I look forward to your v2, especially mutual fund data. I'd be happy to be a guinea pig if you need one :-) Cheers!

@jjttjj

This comment has been minimized.

jjttjj commented Jul 24, 2018

Would you consider streamlining the date field on /chart/ responses?

For example, date only refers to a day, and when intraday data is requested a separate minute field is given. A single date string is able to precisely encompass both of these in a standardized way. I'm struggling to think of a real world situation where a client would use one of these without the other and without some sort of parsing of at least one of the strings into some sort of date type, so why not allow it to be done in a single parse call rather then putting the burden on the client to combine two fields and parse them?

Additionally, the date field itself itself is in different formats for daily and intraday data, the former is delineated by dashes while the latter is not.

@bwong

This comment has been minimized.

bwong commented Jul 25, 2018

Will there be support for the annual financials in addition to the last four quarters?

@sasoftwarerl

This comment has been minimized.

sasoftwarerl commented Jul 30, 2018

Is there an estimated release date for API Version 2.0?

@calvinballhero

This comment has been minimized.

calvinballhero commented Jul 31, 2018

Very much looking forward to this next release, mostly the expanded data sets. Happy to beta test if needed.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Aug 1, 2018

@sasoftwarerl Targeting early Q4

@JoeSalmeri

This comment has been minimized.

JoeSalmeri commented Aug 1, 2018

Is API Version 2.0 backward compatible with V 1.0?

Meaning I can just change the URL from /1.0 to /2.0 and expect existing code to work or will additional changes be required?

Is a preview of the 2.0 API docs available?

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Aug 1, 2018

@JoeSalmeri Version 2 will include some breaking changes. I wouldn't count on it being as simple as changing the version. There are still a number of things that need to be finalized. We will try to release a doc preview as soon as we can.

@JoeSalmeri

This comment has been minimized.

JoeSalmeri commented Aug 1, 2018

Thanks Joshua, I will be on the lookout for the doc preview.

@sasoftwarerl

This comment has been minimized.

sasoftwarerl commented Aug 1, 2018

@joshuablackburn-iex
Sounds good Joshua, good luck, I'll be on the lookout for V2.0

@roblav96

This comment has been minimized.

roblav96 commented Aug 9, 2018

Dear @joshuablackburn-iex,

Preface:
The ~5,000 yearly github contributions were from commits to an automated trading platform for Robinhood which heavily used the IEX API. robinhood.tools (currently mid-refactor) 🤒


I started using the IEX API in March of last year and immediately fell in love with the simplicity, please don't complicate things with a free API key. I'm sure as you know, socket.io is terribly inefficient/sloppy and needs to be replaced with the standard WebSocket protocol. I highly recommend uWebSockets npm install uws@10.148.1. Currently the project is being refactored to use a rewritten C library with uWebSockets-node on the way. You can scale uWebSockets to infinity with ClusterWS.

And that's it! I can't think of anything else to criticize.
=]

Seriously though, huge props and thanks for doing such an incredibly good job on this API project! Everything from batch requests, to lightening fast responses, working with this API has been the most seamless experience in my entire career.

Please don't hesitate to ask for help with anything! I would be more than happy to submit some PRs if needed. Thanks again! :D

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Aug 10, 2018

@roblav96 Thank you for your feedback. I agree with you on socket.io. I'm not a fan either, and I've been looking at uWS for a while. The plan is to integrate uWS. socket.io was a way to quickly offer Websocket support at the time. We've also been running SSE in beta for a while, and it will roll out with version 2.

I appreciate the comments regarding the simplicity of the API. We aim to make this as simple to use as possible. In order to deliver the pipeline of services we've been working on, and upgrade our data to the highest quality possible, we have to introduce API keys. Our goal is to keep this as easy to use as possible.

@bartclaeys

This comment has been minimized.

bartclaeys commented Aug 10, 2018

I have been using GraphQL to access Github API and it has been an extremely frustrating experience. I am totally against GraphQL, it overcomplicates a lot of things, and it's yet another weird 'language' to learn. I know it's all the hype, but if you look at the support forums of Github you'll quickly realize GraphQL is very hard to understand, plus implementation of GraphQL by Github is poor and lacking in comparison to their old style API.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Aug 10, 2018

@bartclaeys Great feedback. I think between our batch calls and filter parameters we are able to offer a great deal of flexibility, and may not need to offer something like GraphQL at this time. I'd rather spend that time elsewhere than add complexity.

@yay yay referenced this issue Aug 12, 2018

Open

Stock Float Data #247

@appagg

This comment has been minimized.

appagg commented Aug 16, 2018

Looking forward to the fund or ETF data supports in v2.0.

@timkpaine

This comment has been minimized.

timkpaine commented Sep 11, 2018

@joshuablackburn-iex any chance for early access? Doing the socket-io api from python was particularly annoying, I don't want to be caught out.

@chris831

This comment has been minimized.

chris831 commented Sep 27, 2018

Has this initiative been abandoned?

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Sep 27, 2018

It is still on track. The plan is a Q4 release. We are finalizing contracts with partners.

@yccheok

This comment has been minimized.

yccheok commented Sep 27, 2018

@yccheok Is 6 months not enough time to switch to v2?

@joshuablackburn-iex Sorry, I away for other tasks. Yes, 6 months is reasonable. But, should we start migrating now, as I just go through IEX documentation just now. It seems v1 is still being used. Currently, we're already happy with v1, for fundamental data retrieval.

We also using websocket - https://iextrading.com/developer/docs/#websockets . Will v1 -> v2 migration affect that too?

Thank you.

@xufab

This comment has been minimized.

xufab commented Sep 29, 2018

Will /stock/{symbol}/logo be incorporated in /stock/{symbol}/company in v2?
It seems illogical to me to not have the logo along with the company details.

By the way, is there any plans to have a base64 encoded logo endpoint? This will allow making fewer requests to retrieve all the company details.

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Oct 3, 2018

@xufab The logo endpoint will change with the intent of returning base64 encoded image. We can also return the url in the company endpoint.

@WojciechZankowski

This comment has been minimized.

WojciechZankowski commented Oct 3, 2018

Is there an updated document for the current TOPS file format?
Is there a more appropriate venue to address differences in the TOPS spec and the actual TOPS files?

This is what you want? https://iextrading.com/trading/market-data/#specifications

@WojciechZankowski

This comment has been minimized.

WojciechZankowski commented Oct 3, 2018

I have followed the format in the TOPS spec from this link and it would appear to be out of date...looking for a current specification of file and message structures. Trial and error lead me to pcap-ng format rather than pcap...but still the message formats are not consistent in size or organization to the descriptions provided...

Am I missing something? Is there an updated doc for the file structure?

I run TOPS file from yesterday and it seems to be working. I have library updated to the newest specification from the link I gave previously. There must be something wrong in your implementation. Indeed messages are not constant in size.

You can check my implementation for further details: https://github.com/WojciechZankowski/iextrading4j-hist Start with TOPSSample class and go deeper.

@myichimoku

This comment has been minimized.

myichimoku commented Oct 4, 2018

@joshuablackburn-iex Thanks for all the great data and simplicity thus far. I have started to use your API for a personal project recently. A key is fine, but would access to data remain free ?

@aokellermann

This comment has been minimized.

aokellermann commented Oct 4, 2018

@joshuablackburn-iex Is there any chance of getting an early set of endpoint documentation before it is released? Additionally, an api key would be a very small price to pay for free and accurate data.

@jingyao97

This comment has been minimized.

jingyao97 commented Oct 4, 2018

Thanks for the effort developing and running this api free for all these moment.
I am currently using it to build my Bachelor Degree final year project and really looking forward to previewing the v2 documentation in order to adapt to the changes.
Thanks again for your hardwork and wish you and the team all the best!

@rajivkbhatia

This comment has been minimized.

rajivkbhatia commented Oct 9, 2018

Thanks for all the great data so far

I have a few questions
-Will the api limits be the same? Will they be determined by IP address or key?

Thanks

@chris831

This comment has been minimized.

chris831 commented Oct 18, 2018

I have followed the format in the TOPS spec from this link and it would appear to be out of date...looking for a current specification of file and message structures. Trial and error lead me to pcap-ng format rather than pcap...but still the message formats are not consistent in size or organization to the descriptions provided...
Am I missing something? Is there an updated doc for the file structure?

I run TOPS file from yesterday and it seems to be working. I have library updated to the newest specification from the link I gave previously. There must be something wrong in your implementation. Indeed messages are not constant in size.

You can check my implementation for further details: https://github.com/WojciechZankowski/iextrading4j-hist Start with TOPSSample class and go deeper.

Turns out the issue was with the PcapReader I was using; created one myself and everything fell in line...also needed to install npcap rather than libpcap to get the project you referenced working...

Hope this saves someone some time...

@joshuablackburn-iex

This comment has been minimized.

Contributor

joshuablackburn-iex commented Nov 7, 2018

Update #557

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment