Megathread: Streamer issues, DXFeed licensing, missing event types & symbols

[Graeme22]

Hi folks, I wanted to let everyone know about some developments that will affect a lot of users of the SDK.

Recently, a lot of users have been having issues with the streamer (#135, #137, #141)
It turns out this is due to some changes that have been implemented upstream in the API. For a bit of background reading see the Javascript SDK's page: tastytrade/tastytrade-api-js#7, tastytrade/tastytrade-api-js#9, tastytrade/tastytrade-api-js#40.

Long story short, there are were two types of streamers supported in this SDK, the DXFeed and the DXLink streamer. DXFeed was an older protocol which now appears to be completely defunct. DXLink is the new protocol, and most users had already migrated to it. Not too big of a deal up to this point.

However, it appears that Tastytrade is now providing two different websockets for use with DXLink. The first, obtained by hitting the /quote-streamer-tokens endpoint, has been around for a long time. It is the endpoint used by official Tastytrade applications (I confirmed it's still being used by the Tasty web platform with a MITM just now), and doesn't have many restrictions with respect to event types, instrument types, etc. The second, newer one, /api-quote-tokens, is designed specifically for us API users.

Apparently, the /quote-streamer-tokens endpoint is now reserved for internal usage, and it is considered a violation of DXFeed TOS to use it. According to Devin Moss from TT:

"Distributing quote data is something that puts tastytrade at some risk of breeching [sic] licensing agreements, especially if they were to make it available to all users with API access. tastytrade has no control over what API users do with those quotes, which poses a real possibility that abusers could use the data to source their own proprietary projects."

Well, there's definitely plenty of people using the data "to source their own proprietary projects," though I thought that was kind of the whole purpose of the API... It certainly appears most users are using it alongside Tastytrade in a way that seems to fit within the API's intended purpose.

So where does that leave us? Well, to start off, some "normal" users will be unaffected. Those who are not trading indices nor futures and don't use greeks will hopefully be able to use the new websocket without problems. It's already mostly implemented in #140.

However, there are many users who are relying on the streamer for other kinds of instruments (especially SPX afaict) or events, like greeks. These users could continue to use the old DXLink websocket; however, Devin has also said:

"For the time being, please only use the [/api-quote-tokens] endpoint. API users who rely on the /quote-streamer-tokens endpoint will be flagged and put on delayed quotes."

So this is pretty bad news for a lot of users. I guess it was too good to be true that simply creating a Tastytrade account was enough to get real-time data for any instrument you could want? Anyways, that's pretty much where we're at for now. It doesn't seem like Tastytrade will change this, but maybe if we get enough people to email them something can be worked out.

[Graeme22]

So here's the plan. The main branch and the version v8.0+ releases for this repository will follow the DXFeed TOS. However, I've created a branch advanced-streamer for people who need the old functionality, which I'll also release periodically as v7.1+. I'll do my best to keep it working if possible.

[some-person-0]

Thanks for all your hard work and the update. I view the loss of the ability to get greeks as an absolute disaster!

[OperationalFallacy]

This is a good overview, @Graeme22
By the way, some users are completely fine with the delayed quotes for the automated trading via API.

There is also a technical problem with using dxfeed streaming quotes within API services on a backend: client.connect('wss://demo.dxfeed.com/dxlink-ws') - unlike previous implementation, this is strictly designed for the web and app clients. It is still possible to use it in the backend, but it looks akward.

I hope TT will figure out some pricing for API streaming. Of course, there is an avenue to sign up directly with Dxfeed; the problem is that the whole industry and Dxfeed are not particularly interested in catering to small API development shops.

I'm also considering abstracting quote service in the backend, so any provider can be plugged in if needed.

The problem with that TT backend is too tied up to the formats and structures of dxfeed, I suspect it will be very hard to separate them. I haven't looked into that too much yet. It's interesting to hear what others are planning to do.

[joshuakoh7]

So here's the plan. The main branch and the version v8.0+ releases for this repository will follow the DXFeed TOS. However, I've created a branch advanced-streamer for people who need the old functionality, which I'll also release periodically as v7.1+. I'll do my best to keep it working if possible.

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

Also, consider sending a different user-agent header instead of python requests

[Graeme22]

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

So that's actually not in my control at all. Personally I haven't had problems with whole SPX chain, but you could consider stitching together two separate streamer if it's not working for you.

Also, consider sending a different user-agent header instead of python requests

The v7.1 release is already using a random, realistic user agent!

[joshuakoh7]

Please consider a larger buffer for the advanced streamer because subscribing TAS for the whole SPX chain is unstable with too many subscribe calls

So that's actually not in my control at all. Personally I haven't had problems with whole SPX chain, but you could consider stitching together two separate streamer if it's not working for you.

Are you subscribing to TAS of the full chain? I was using the old repo under the old tastyworks package before switching to DXLinkStreamer here and I was able to chunk 10k symbols per subscribe call with the old streamer. With the new streamer I can do max of 2k before I hit the buffer size. Not sure if this is the reason for the streamer dropping more often (sending too many subscriptions calls).

Also, consider sending a different user-agent header instead of python requests

The v7.1 release is already using a random, realistic user agent!

Not for the logging in to tastytrade though (TT could be flagging via that)

[Graeme22]

Are you subscribing to TAS of the full chain?

Nope, I was doing Greeks and quotes.

I was using the old repo under the old tastyworks package before switching to DXLinkStreamer here and I was able to chunk 10k symbols per subscribe call with the old streamer. With the new streamer I can do max of 2k before I hit the buffer size. Not sure if this is the reason for the streamer dropping more often (sending too many subscriptions calls).

Not much that can be done here, though it seems like as it's something you only have to do once it shouldn't be too big of a deal? Granted I don't know your use case.

Not for the logging in to tastytrade though (TT could be flagging via that)

Ha, good point! Would you be able to do a PR to fix that?

[kaidaniel82]

@Graeme22
Hi all, can someone pls confirm if VXX, UVIX, and VIXY are working? I've been facing issues with an older version of the API (6.4) that has become very unreliable in receiving those quotes for the past few days. It was working fine for many month. I like to know if this could be related to the issue you are describing.

I connect to the DxLink service up to four times simultaneously through parallel processes, e.g. each process is independently connected to Tasty. It has always worked until the last few days.

[Graeme22]

That's probably the issue. Could you see if the latest (7.2) fixes it? Those are ETFs so no reason they shouldn't work normally.

[kaidaniel82]

Thank you for your note,

I'm looking into it now. I need to perform a minor surgery on it, as I had forked the repository to implement a mechanism that inserts an SSL certificate into the Streamer class to connect with the websocket. By the way, this would be a great, fundamental feature.

Additionally, it's not so easy to test since it sometimes works and sometimes doesn't over the past few days.

[Graeme22]

You're welcome to open a PR, though I'm not sure why we'd need to secure quotes streams, maybe you could enlighten me on that?

[kaidaniel82]

I had to set up a complete RnD project on Windows Server 2022 to figure out why the websocket for live data was delivering no data or unreliable data. The same code ran fine on my Windows 11 computer, Linux , and Mac. As soon as I switched to Win Server 2022, it didn't work.

I received Python error messages indicating that there was something wrong with the SSL certificate, or perhaps they weren't properly found on the server. Consequently, I assumed that the upstream library needed them.

After I added these cert.pem files to the websocket.connect, it worked flawlessly for months for me and many of my colleagues.

[joshuakoh7]

Got an email from TT, seems like the new API should now support indices and futures? Can anyone confirm?