Timeout error in 0.4.4 version #140
Comments
elderanakain
changed the title
|
same issue, it worked yesterday though |
|
Thanks for reporting @elderanakain @eswark18 and for the detailed issue description 👍 It seems that there's an issue with the selenium installation. Please try using |
|
@Voyz Thank you for the quick reply! logs |
|
Thanks for reporting back @elderanakain! Snap, that's an annoying error. It seems to just pop up randomly, and I can't figure out what causes it. Can you give it a shot to launch it a few more times and see if it is persistent in your setup? |
|
@Voyz, I have tried to run it several times on my local machine, and it is very persistent. I happened every time after each restart, and if I leave it running to get to the next maintenance cycle. I have used the Not sure if it is helpful, but here are logs from the IBKR GW: logsCould you please provide some direction on where the issue is, so I will try to look into this as well? Is it somehow related to a selenium version, |
|
Thanks for testing that for me. I must admit, I'm kinda blind at this point. I've found some resources metioning adding Also, can you test that version on your server as well as on your dev machine and see if it persists there? While this isn't going to make it any better for you at this moment, I'm observing that there isn't more reports coming in of this issue. I'm imagining this may affect only a small amount of users. |
|
Thank you for the details. I will see what I can do to make it work on my local machine. However, the new RC has worked just fine on my Linux server 🎉 thank you for the quick fix! Still have the same issue on my macOS machine, but I agree that it looks like it affects a small number of users, and macOS is definitely not the primary target platform :) logs |
|
Gotcha, thanks for confirming. I'm still gutted it's breaking up on your dev station. Let me know if I can help you brainstorm about this. Just FYI - I've pushed master branch, so it now has those most recent fixes in case you'd be interested in pulling and fixing things first hand. |
|
I'm also seeing what appears to be a timeout of some sort using the 0.4.4 release (not the latest fixes) After that, it seems to get stuck in a loop... |
I came across ibeam a few days ago and ran into the timeout issue. FWIW On linux, your change for 0.4.5-rc2 (in addition to increasing startup and page load timeouts) allowed me to authenticate (including IB key.. experience). This resolved the first error bfoz posted above, not sure about the second log. |
|
I just pulled 0.4.5-rc2 and the timeout seems to be solved, but now it's stuck in a "success" loop: The loop restarts after 60 seconds. |
|
Eventually it worked 🤷♂️ |
Nice! This was where I cursed and wished TOTP was a valid MFA method for IBKR. And increased my |
|
@Voyz hi there, I just tried 0.4.4-rc6 and I'm getting following error (running on Mac M1): 0.4.3 works fine for me, but I can see substantial differences between the API gateway versions, I believe 0.4.3 has gateway version from 2018 and 0.4.4 has one from 2023. Would it be possible to do a re-release of 0.4.3 with a newer API gateway? (or is that what's going on with 0.4.4 and I'm asking stupid questions?) Also, for some reason, on 0.4.3 I don't have to do manual login, but on 0.4.4 onwards I have to open the authentication page and log in - is that because of the newer version of IBKR Api Gateway or because of the chrome issue? |
|
A quick update, took the code from main, updated Dockerfile to: As bullseye supports newer versions of chrome and chromedriver (the jump was significant, I believe it was from version 80-something to a hundred something), it was possible to use the newest version and that fixed the I also downloaded the newest version of API gateway, but I don't think that made a difference. And as someone mentioned in one of the other issues, I changed IBKR API endpoint in conf.yaml file. Anyway, now the API gateway only took 2 attempts to log me in, and after spending hours trying different versions of chrome, selenium, api gateway and base docker image, I'm going to call it a success. Hopefully it helps someone. @Voyz someone mentioned it elsewhere that it would be great if ibeam could automagically try various API endpoints to see which will work. You mentioned that it would bring a risk of having to change conf.yaml file on the fly. Could you elaborate more? What is the risk? |
|
Hey @vldmarton superb digging, thanks for helping push this issue forward 👏🥳 I'm quite surprised that updating the Python version would work. Nevertheless, I've just pushed the |
My thoughts are as follows:
Therefore, it is an added risk and complexity, hence trying out other methods first sounds like a better idea to me. We could have a 'source' version of the All in all though, it sounds like a bit of a project to take on. If someone would like to, feel free to contribute a PR with this addition. |
|
Thank you, but we should really be thanking you instead, you did an incredible amount of work here for us.
Unfortunately that doesn't work for me. I don't think it was the python version that did the trick. Other than the python version, I also changed the base image from buster (debian 10) to bullseye (debian 11). That allowed the newer versions of chrome and chromedriver to be downloaded. Did you make that change in 0.4.5-rc3 as well? If yes, perhaps the fact that I'm running an amd64 image on darwin is causing the issue...
Either that, or we could rename
It really sucks that IBKR cannot fix this issue themselves (we can only implement workarounds, and that's far from ideal). I will try to put some code together tomorrow but I'm not as good with Python as you, so don't expect much 😆 |
|
Aw thanks for the kind words 🤭
Yes, I've changed that too. The current version is
Perhaps, yeah. The only other thing I can think of is the version of the Gateway that got upgraded to the July 2022 version in 0.4.4. Wanna see if an older version before that commit would help solve the issue? 49d2bf1
Cool idea! Some things to consider:
Amazing, thanks! And I'm here to help! Feel free to submit a WIP PR as you're working and we can iterate on ideas together 🤜🤛 |
Yep, that's correct, I also checked the container image and it matches the Dockerfile, so no issue there.
We don't even get to the point where gateway could be the problem, and apologies for not mentioning what is the current issue now. I'm just reading my previous answers and I said that it's not working for me, but I didn't say WHAT is not working 😞 The issue is still I'm not sure which one of these is even correct, but 1.) the
Yep, I don't like it very much either. My worry is that we will implement conf file rotation for 8 API endpoints and in the next release they will implement a new, load balanced API endpoint, which will make our solution redundant. 🤣 Maybe a support ticket would be the number one thing to do, and in case we don't manage to "beat" a solution out of them (and not just a recommendation), we could move forward here. I'll get on that first. |
|
@Voyz I created the IBKR support ticket, but it's going to take some time to get a response. In the meantime, I looked on the internet and found other github project with the same aim and with the same problem - it's an interesting read for sure: TLDR: Do not rely on the
|
Yes, that seems to be fine. We don't use chrome anymore, we use chromium. And when you try to launch it from the container it complains about not being able to connect to a display, which sounds fine to me. root@ffea9bd21020:/srv/ibeam# chromium --no-sandbox
[167:167:0524/155236.915464:ERROR:browser_main_loop.cc(1386)] Unable to open X display.
I'm getting a similar output, but I also am getting similar output on I'm seeing the following on root@ffea9bd21020:/srv/ibeam# chromium --version
Chromium 90.0.4430.212 built on Debian 10.9, running on Debian 10.4
root@ffea9bd21020:/srv/ibeam# chromedriver --version
ChromeDriver 90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429})And on root@32e8f901ac2e:/srv/ibeam# chromium --version
find: ‘/root/.config/chromium/Crash Reports/pending/’: No such file or directory
Chromium 113.0.5672.126 built on Debian 11.7, running on Debian 11.7
root@32e8f901ac2e:/srv/ibeam# chromedriver --version
ChromeDriver 113.0.5672.126 (c541687b21a73452ab403e2dced7033ddc97ee9d-refs/branch-heads/5672@{#1202})I suppose one indication that things aren't right is that on your machine the
Great to see you tried this. Let us know what they say.
Right, I can see that, great digging 👍 Seeing that this is from an unofficial source and quite speculative, I'm not sure whether this would be a methodology we'd want to follow. My thoughts are:
I'd feel that prompting IBKR to fix things would be the right way to go here. |
|
Hello, just pulled latest image today, and hit a similar problem on Windows 11. Error is: Also, my docker compose is: If there are any suggestions that can be tried, please throw one in my direction. Cheers, |
|
@andreimcristof thanks for joining in and sharing the details. Can you try |
|
@Voyz Thank you very much for your speedy help. Tried it, and... spot on - seeing later edit - please ignore the questions. Made everything work :) |
|
@vldmarton many thanks for sharing this here. This bit got my curiosity
I don't quite understand it, I've got two interpretations of this. Give a 👍 or 👎 if you believe it's the former or the latter. 👍 Does it mean that logging in twice will somehow log us out?
Only
In either case, not sure how calling validate would help. Am I missing something where this could be useful? I'm thinking that when things go badly, the result of
Yeah... I wouldn't really have much to comment on this 🤷♂️😔 |
|
I've just published |
|
By the way, just out of curiosity, I added an extra piece of logging to confirm if we really depend on the So, unfortunately, looks like we do have to get
What he says has actually happened to me multiple times when running API gateway directly on my computer. I started the gateway and I was logged into paper account on my phone as well. I had to open browser, log in once, which returned me to the login page and I had to log in agan. After the second attempt, it went fine. Similar situation is happening when I open client portal from my laptop browser and log in, and then log into the same acc from my phone. The browser on my laptop will say that there is another session and that my actions are limited. There is a Sign in button next to it. After clicking on Sign in, it repeats the warning and wants me to click the "Sign in" button again. After I do that, I'm allowed to view my portfolio, open trades, etc. Therefore, it's starting to seem to me that the 2-attempt login is a normal thing in IBKR. I'm not sure though why in some cases it's necessary to do it more than twice.
I'm mixing two different topics here I'm afraid, and I forgot to describe the latter. This is what happens in my container all the time - I'm dotting out logs to save space and it's also important to point out that it happens randomly (sometimes there isn't a problem for 4+ hours and sometimes it happens twice within 30 minutes): So, as you can see, there is some issue with the Gateway because after approx. an hour it decided that the session is logged out for some reason. So, perhaps calling both If there is an issue with the session, I want to know about it ASAP, can't wait 60 seconds for the maintenance window. So, here's an idea:
Looks very stable buddy 🥳 not sure what you did but no more crashes. Although, it had to go through the auth process 3 times to do a successful login, but tbh, at this point I don't care anymore, I'm just happy it succeeded - I guess I have to appreciate the little things 🤣 I'll keep it running at night and see what happened during the night. For now, I'm attaching logs of the login attemps: ibeam-045.log |
|
@vldmarton Thanks for sharing your results 👍 And thanks for casting light on that 2-log in, I get your point now. Okay, well I guess IBeam will do that automatically. It does correlate with some results I'm seeing.
In all fairness, let's give it a shot, we've got the code to call
😬 I feel for you, yeah, reliability is key in this case. Possibly request secondary account (or even a tertiary one), and keep several instances of IBeam active and open. Then hide it behind a load balancer and switch between them depending on the health status.
Just in case you haven't considered it, you can change
Note that the whole reauth process can take longer than 5 seconds, so you may be stuck in a re-auth loop if you do this check too often.
Very cool, thanks for confirming! 🚀 |
Thanks for the new version, I will run it now and get back to you in a few hours.
That's actually a good idea, thanks, but the problem is that I would have to split my capital to three accounts and that would reduce my profit by the number of accounts. I was thinking yesterday about running 3 instances of IBEAM a storing authentication state somewhere, so that there is always at least 1 instance which is successfully authenticated and as soon as things go wrong with it, it removes the state file and one of the other two take over. For now, I keep it in my head, because this might become a very time-consuming project and again, it's just a workaround for something that should work without it.
Thanks, will give that a go with the 0.4.6-rc1. I'm also attaching a continuation of the log file from yesterday - it's version 0.4.5, started at 20:34 and still running until now. Unfortunately there is a bunch of weird things happening. 😢 ibeam-455-2.log |
|
@Voyz 0.4.6 gives me 
I'm not sure what the release process is (I've never pushed images to dockerhub), but looks like the rc versions are not built for multi-platform. |
|
IBKR having ANOTHER auth issue on their end w/ client portal API this morning, just off the phone w/ them |
|
@Voyz Starting this morning i had timeout issues Didn't look deeper in what you changed but thanks, works for me now. |
A small update, it's not clear to me how it was possible to get blocked by the scheduler for such a long time, it must be some frozen process that blocks the whole script (it would be interesting to see what blocked it). So, in the meanwhile, I'm testing with |
|
Okay, so it looks like the random skipped scheduler runs are not caused by the ibeam code, but rather by my Mac. I don't know what's going on in the background, but it seems like it was putting processes to sleep even though the system is not sleeping and screen is on. I will test it on a linux server some time in the near future, but I'm currently busy with other things. |
As far as I understand these are separate logins to the same account. I'd double check it, but I think you wouldn't need to split it into three accounts.
Right are you running it on arm chip? Yeah the |
Can you share what was the issue an what was their response? If it was unrelated to the topics described in this issue it may be a good idea to open a separate issue |
Interesting! Can you share how did you figure that out?
That's a good idea, keep us posted |
|
@Voyz Sorry for the long no reply. I can confirm that the latest version works fine on both machines. Thanks a lot for the fix 🎉 |
|
If authenticate: false, why not use /iserver/reauthenticate and wait about 15s? In my remote server, I use IBeam to keep gateway alive. |
After it has happened multiple times during the same test run, I kind of just noticed that it happens when I don't use my computer for a longer period of time (20-30 minutes). I have sleep mode (and all other optimizations) turned off, I did that specifically to do this test, but still, it was happening, and it was only happening when I was away from my laptop.
I've been extremely busy lately so I still didn't have time for it yet, but I hope I can get back to it as soon as possible. |
The fact that this is an official statement from IBKR sounds like there is some very strange design with the Authentication procedure. I find it amusing, or half amusing / half concerning that a company that manages people's money, actually advises that several attempts are required in order to receive Authentication should have 100% predictable outcome. But I might not understand the whole complexity of their system, it might be a legit solution to a much deeper problem. I have also had major issues logging into Client Portal itself (the web app, so nothing API related) where, after entering 2FA code, it just flat out freezes on browser. I have to click back and retry and hope it works. At some point this just stops working entirely, so I have to do Reading through this thread in detail it seems like IBeam elegantly works around and solves some of the shortcomings of IBKR, but those shortcomings should not exist in the first place. |
|
You're absolutely right. IBKR has multiple products (TWS, API, mobile app, client portal). Judging by the appearance of TWS, it's very possible that the legacy code (the absolute core - session handling included) is about a 100 years old and would be too hard to rewrite (modernize) in such a way where all of those products would still work. At the same time, creating a modernized version of the core and a modernized version of all of the products would be equally hard (and expensive) and clearly IBKR is holding onto the "if it ain't broke don't fix it" methodology. Although, I've read somewhere that they're developing a new desktop application, which should replace TWS in the future. So there's little hope. Of course, it depends on what they decide to do, because they can still build it on the top of the same, old legacy code. |
|
I'm using 0.4.5 and still seeing lots of errors. It seems to be stuck in a successful login loop again. |
|
@migoohao short answer: we never knew I tried that endpoint in the past but it never seemed to have solved our problems so I mainly ignored it. I just gave it a shot to implement it - check out See the logs below: logs |
|
@andreimcristof you're making a fair point. I think it is fair to say that a lot of IBKR users find the APIs, both CP Web API and TWS API frustrating to use. I have read some IBKR code from the Gateway front-end (anyone can read it too if you wanna check my observations) and I must admit that if its quality is any indication of the overall quality of their code internally, I'm not surprised we're encountering all these errors and pain points. At the same time, I also try to err on the side of hoping that they're doing the best they can with the system they have. And to your last point: IBeam should not exist in the first place 😅 |
|
@bfoz thanks for sharing it seems to be a different problem, not the authentication loop. The website loads but fails to find the right triggers. I'd suggest you run it in debug mode and save screenshots and open a new issue. |
|
@Voyz I noticed you triggered reauthenticate, but you triggered logout immediately. IB Gateway needs about 10-15s to make "authenticated:true".
|
|
Thanks for sharing these thoughts @migoohao - I've just pushed |
|
@elderanakain I think this issue can be closed now? Or would there be anything else we'd want to discuss regarding the timeout error in 0.4.4? |
|
@Voyz Yes, definitely, the issue has been solved. Thank you! |
|
hey @elderanakain @bfoz @stephas @vldmarton @andreimcristof @JackD111 and others - there is a new WIP version Run the most recent See #147 for more details and report any observations directly there. Thanks 🙏 |
Describe the bug
This morning, after the update to
voyz/ibeam:0.4.4, a local watchtower service restarted theIBeam, and it stopped authenticating with an error, which looks like a timeout. Please, take a look at the stack trace below, as I don't know Python very well.To Reproduce
Steps to reproduce the behavior:
0.4.4docker imageExpected behavior
IBeam docker image works the same as before. I have tested quickly in the same environment with
0.4.3, and it worked just fine.Environment
IBeam version:
0.4.4Docker image or standalone: docker
OS:
Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-71-generic x86_64)as a DigitalOcean droplet, check also on a local macOS machineAdditional context
Logs:
logs
The text was updated successfully, but these errors were encountered: