Getting rid of web browser

[francoisruty]

Describe the feature
Given the fact that the gateway exposes a web page to enter credentials, I guess this webpage must use some sort of API to send the credentials. The idea would be to reverse engineer this API so that we could send the credentials directly through this API and avoid using a browser for performing heavy crawling operations

Expected interaction
How would users interact with the feature. Include code if possible.

Possible implications
Describe which parts of the existing system could this feature affect and in what way.

The downside if that if the underlying API changes, this system would break, and it seems the CP gateway is not really versioned so this makes evolutions hard to track

Additional context
Add any other context about the feature here.

I'd like to potentially operate a big number of client portal gateways, hence the reason to make this docker container as lightweight as possible

[Voyz]

Hi @francoisruty! Thanks for submitting this great suggestion! It sounds like something that could reasonably work on paper, and it would make both the Docker image lighter and the system as a whole simpler.

Would you know how to approach such reverse-engineering of the API? I wouldn't really know where to start, other than to track the outgoing HTTP in the browser and try to learn something from this.

[francoisruty]

well I just took a look (I used chrome dev tools network tab) and unfortunately it seems the data is transformed by the front web app, I don't see a POST request with username/password, it's something else
The web app javascript source code is present in the client portal gateway but it's "compiled" with all dependencies so it's not really readable... Someone would need to read this source code to figure out how the data from the form is transformed exactly before being sent to the API

[Voyz]

Great, thanks for looking into this a little more @francoisruty. It's useful to know what needs to be done next. I'll leave this issue open for now as a future suggestion for myself and other users to look into.

[gmcmicken]

Hey guys, I did a lot of testing last week to determine why my authentication is lost every few hours (hence how I found iBeam in the first place) and I uncovered a lot of complexity. There is a system to handle elevation of permission to perform trades and there is a series of timeouts and client-verification calls to determine which system has what permissions.

My opinion/guess is there was a bug introduced and IB dev team doesn't even know where it is yet - their beta of the client portal gateway attempts to disallow a user from logging in with another session and overtaking the trade permissions. In my testing this mostly worked as they described but if you're fast enough you can still juggle between them and perform trades. It's really messy and I think you're going to run into a lot a difficulty keeping up with whatever is going on under the hood.

Just my two cents.

[typeless]

@gmcmicken
I encountered the same problem. The trading session cannot sustain itself for more than 4 hours.

attempts to disallow a user from logging in with another session and overtaking the trade permissions.

Does it solve the problem? I doubt it's due to another simultaneous login. In my case, I always use a dedicated account for my trading bot and do manual trading/monitoring with another account.
Anyway, I'll check out the beta gateway. Thanks for your information.

[gmcmicken]

@typeless No it doesn't solve the problem, and like you I also didn't have multiple sessions while testing but it's something IB dev team seems to think is the cause.

[Voyz]

Just to add my subjective observation to above discussion - I'm not surprised at all things are not working as expected and you guys are running into these issues. I've been reading the front end JS code of the Gateway more thoroughly recently and to me it seems like if it was written by someone / some team that has little experience in the subject, although likely had the best intentions and were grossly undermined by the management or whatever other circumstances. Really bad programming practices all across the codebase (eg. global variables everywhere!), confusing names, poor testability (and likely few or no automated tests), essentially spaghetti code full of bodges, hacks and bugs that has been written just to push something out. If the Gateway itself has been written at least half as badly as that JavaScript front end then I think the safest route - rather than to hope for the Gateway to fulfill its promises - would be for us to be prepared for random connection drops and unexpected behaviour. The Gateway receives hardly any updates despite claims that 'New gateway versions will be released frequently', last beta release being on 25/09/2020, so unless there's something happening behind the scenes at IB, I would assume we won't get much real support anytime soon - as much as I wished it was otherwise.

We can however aid each other in understanding these erroneous behaviours and attempt to patch or prepare for them on our side - so thanks for your reports so far, any input on the subject is valuable 👍

[francoisruty]

I agree with your analysis. I was counting on the REST API but seeing that the client portal gateway has no update since august 2020, that it is not versionned and that the source code is not really available, I drew the conclusion that I can't rely on it and I'm probably going to use the TWS API, unfortunately..

[Voyz]

@francoisruty fair play! I used TWS API before that and the way it handles asynchronicity was just unbearable. CPW Gateway at least has a possibility of being done 'the right way' at some point in the future; I don't think anyone could salvage the TWS API in a reasonable way without a ground-up rework. You'd think with all that money they could afford building something proper eh!

[francoisruty]

agreed, our only hope is that CP Gateway not evolving anymore means they're working on making it unnecessary to access the REST API, that would be so great

[Voyz]

I'm going to close this issue due to lack of activity - let me know if anyone would be interested in reopening this issue 👍 Thanks for the discussion so far everyone!