Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is the recommended way to run IBC in Docker? #67

Open
dvasdekis opened this issue Oct 31, 2019 · 48 comments
Open

What is the recommended way to run IBC in Docker? #67

dvasdekis opened this issue Oct 31, 2019 · 48 comments

Comments

@dvasdekis
Copy link

Hi Richard and others, thanks for providing such great software for free.

I'm wondering what the recommended way to run IBC in Docker would be? There are a few repos, and I stumbled into using this one without understanding the background behind IBC/IBController/QuantConnect etc. It does work for me, albeit CPU usage is quite high for the operations I'm running on it.

In the pursuit of attempting to streamline this image, I've discovered that it's running some quite old software. Is there any up-to-date IB Gateway solution running in docker that I should be using?

Many thanks!

Dimitri

@mcbain
Copy link

mcbain commented Dec 7, 2019

Im currently working on this, but there are some problems with the [ ] Allow connections from localhost only. It needs to be turn off, best via config. The TWS will work then without any client connectivity problems. The Gateway needs the valid trusted IP of the docker gateway the container was started in. (docker inspect container | grep Gateway)

@jspahrsummers
Copy link

FWIW I ran into this same dilemma, so I built my own image here: https://github.com/jspahrsummers/ib-gateway-docker

It's paper-trading and read-only right now, but those are just config settings, so feel free to fork and reuse if it's useful.

@mcbain
Copy link

mcbain commented Dec 8, 2019

@jspahrsummers How did you solve the Allow local connections only issue?

@jspahrsummers
Copy link

AIUI the use of socat will ensure that incoming connections look local to TWS.

@dvasdekis
Copy link
Author

dvasdekis commented Dec 8, 2019 via email

@dvasdekis
Copy link
Author

As a note, after about six hours of usage, the Java process running IBGateway/AWSAPI uses more and more memory to the point where it falls over. Has anyone encountered this previously?

@larroy
Copy link

larroy commented Feb 28, 2020

#79

@omdv
Copy link

omdv commented Feb 3, 2021

@dvasdekis, sorry to resurrect an old thread. I just experienced similar issues with 978 gateway on GCP instance - the memory/CPU usage is growing linearly over time and eventually completely overloads the instance. It was with image I built. Did you manage to find the root of the issue? Do you experience anything like this with your image?

@dvasdekis
Copy link
Author

dvasdekis commented Feb 4, 2021

@dvasdekis, sorry to resurrect an old thread. I just experienced similar issues with 978 gateway on GCP instance - the memory/CPU usage is growing linearly over time and eventually completely overloads the instance. It was with image I built. Did you manage to find the root of the issue? Do you experience anything like this with your image?

Hey @omdv, yes, this was caused by my IB_Insync doing something dumb (in my case, re-subscribing to Market Data repeatedly) without cleaning up the connection (via cancelMktData(), followed by IB.close). This caused the relatively fragile gateway to memory leak and eventually collapse. Once that was fixed, it was stable.

@bfoz
Copy link

bfoz commented May 16, 2022

Any updates on dockerizing IBC? What's the recommended way in 2022?

I'm just getting started with this and I'm wondering how to know which Dockerfile to use. I don't need the GCP version and it seems that @jspahrsummers has archived the linked repository. The one from mvberg hasn't been updated in 3 years.

Has anyone suggested making an official image on Docker Hub?

@rlktradewright
Copy link
Member

No, there is no recommended way of running IBC in Docker.

Actually I began work on a Dockerfile based on @larroy's work (see #79) a couple of years back, with the intention of including it as a sample in the IBC repository (but with no implication that it was the 'right' or 'recommended' way to do it), but it got blown out of the water when I updated my Ubuntu VM to a later release and I could no longer build it. I didn't have the time, or really the inclination, to sort out the problems so I just shelved it (I still have the files though, in case I ever want to get back to it).

To be honest, I don't really see much value in it. If one is just a private individual wanting to run a couple of TWS or Gateway instances, I see no real point in running it under Docker. If you are a commercial organisation wanting to run dozens or hundreds of Gateways, you can do the necessary work yourself and perhaps contribute it back.

Anyone who wants to make an IBC image on Docker Hub is of course free to do so, as long as they don't try to claim that it's in any way 'official'.

I would also be willing to consider including a sample Docker along the lines mentioned above if someone wants to do the work and offer it via a pull request, provided they take account of the points I raised in #79 (where relevant) and they're willing to commit to maintaining it going forward.

@bfoz
Copy link

bfoz commented May 17, 2022

For a private user, I think the the Docker value-add is mostly along the lines of "It Just Works". That is, the image pulls in all of the necessary dependencies, keeps them from interfering with other things that might be installed, and takes care of a lot of the grunt work. Oftentimes Docker (or Docker Compose, more likely) can be used to automate a lot of the boilerplate configuration and set up tasks, which makes life much easier for less technical users. It also makes life easier for those of us who are technical users, but long ago grew weary of manual configuration and set up.

In my particular case, I use a Synology NAS for running lightweight applications, dashboards, etc. Containerization makes it easier for those various utilities to avoid stepping on each other, or the base system. Even though Synology is just Linux under the hood, and I could go in and install everything directly, I'd run the risk of interfering with the base system. Even if I did want to take that risk, I'd still have to worry about my customizations being blown away by the next firmware update, or the next time I update the hardware. Containerizing simply avoids an awful lot of pain points that don't need to exist.

Even if I were to go back to running a "real" server, I think I'd probably keep using containers, just to avoid issues with various packages needing 10 different versions of 3 different dependencies (it happens). Or packages that want to install crazy things like x11vnc, or even Java. 😃

It would actually be very helpful if there were a recommended way of running in a container. Even if there are multiple ways to do it, a default way that's good enough to work "out of the box" for most people would be very useful for people who are either non-developers, or just new to the project. As it stands now, a new user has to sort through a pile of equally unofficial options with no clear way to decide between them. Sorting through the mess adds a lot of unnecessary pain and confusion to the new user experience.

I'm happy to contribute a Dockerfile to the project, and an example Docker Compose file, but only if the project is willing to push it to Docker Hub. Publishing an image from a personal account would only add to the confusion, which is the thing that I'd be trying to fix by contributing a Dockerfile.

@rlktradewright
Copy link
Member

Ok, thanks, nice write-up.

First let me say I have nothing at all against Docker: it's a wonderful technology, and you give a good summary of (some of) its many benefits.

My position here is that IBC is such a tiny little program, trivial to set up, rarely needing any change to keep it operating smoothly, no dependencies (except Java, which is anyway installed with TWS/Gateway), that I don't see any 'pain points' that containerising would solve for IBC. That's not to say a Docker container isn't of value in some circumstances (and your Synology NAS is a good example, though you could alternatively run a Linux VM on the Synology Virtual Machine manager and run IBC in that). And as I alluded to in my previous post, a Dockerised IBC is probably essential for a commercial organisation serving many customers with an IBC instance for each, so they can take advantage of the management facilities provided by Kubernetes, etc.

So in the case of IBC, installing, learning, and maintaining a Docker installation is much more complex than simply running IBC directly. Of course most people wanting to run IBC under Docker will already have been through this learning curve, so perhaps that's not a real issue, just an observation.

And so, I repeat, I'm willing to consider a sample Docker implementation that can be used either 'out of the box' or for further customisation by users. But at this stage I'm nowhere near being in a position to agree to pushing it to Docker Hub.

Part of the reason I'm a bit cagey about all this is the additional support load that such a thing would entail. As soon as there is a Docker image or Compose file in the repo, there will be a flood of new questions from users. For example:

  • every time I run the IBC Docker image, it has reset any configuration changes I last time. How can I prevent this?
  • can I store my IB username and password in Kubernetes secrets?
  • how do I change the network address to be used for API connections?
  • can I run IBC/TWS under Docker (rather than IBC/Gateway)?

There may be simple answers to all of these, and the many other questions that users will come up with, but nevertheless they all need to be responded to. Some may result in calls for IBC to be modified in some way to make it fit better into the Docker environment.

To be frank, I don't want this extra hassle, which is why I say that if I am to accept a sample Docker file into the repo, the contributor must commit to documenting, maintaining and supporting it for the long haul.

If you already have a Docker compose file that works well, I'd be very interested to see it and assess it myself. You don't need to make a pull request at this stage, just email the files or stick them somewhere accessible. I have to say, though, that I no longer have a Docker setup at the moment, so I'll have to start from scratch again, and I really don't have much time (ie none!) for this.

@goodboy
Copy link

goodboy commented May 19, 2022

As a total outsider who wants to now automate ib-gw for some more modern *nix headless deployments, i would say it's pretty critical to have a template Dockerfiler in this project.

In fact, because there isn't one, it's actually greatly turned me off wanting to use this project for a while.

But wait, why?

  • as a minimalist nix user if i could avoid a java install entirely it's a huge win

  • everyone else that's hacking their own, is doing mostly a junk job, their code doesn't work, I'm now relearning why DSLs for supervision is absolute madness:

  • as mentioned, the whole point of docker-like-things is you don't have to think about it

    • also it works cross OS (usually), .. yah yah jvm but let me tell you about peeps on windows who glaze-over**2 when you mention such things..
    • it's easier to hand off to a modern dev-opsy type peep
  • the experts on the source should know better then anyone how to keep the container (or equivalent file-DSL-thing) working for the most basic case

    • Part of the reason I'm a bit cagey about all this is the additional support load that such a thing would entail. As soon as there is a Docker image or Compose file in the repo, there will be a flood of new questions from users. For example: - every time I run the IBC Docker image, it has reset any configuration changes I last time. How can I prevent this? ... blah blah blah

    There may be simple answers to all of these

    yeah and y'all already seem pretty well versed in writing readmes 😉

  • If you already have a Docker compose file that works well

    • uhhhhhh, works well eh?...
    • well this one is supposed to but for me it hasn't.. and this is one of the better ones i've found.
    • would be lovely to not have to use socat (and other sh related malarky) to make this whole thing just work...
    • i would put in much effort to also contrib fully working Python docker-py automation for first class support for "python-as-a-DSL" 😱 )
    • 10 million bonus points for having the container be known to work with ib_insync (the most pop py client on gh) work out of the box with it

    Anyway, hopefully not too ranty nor drive by, I just want to not have to uses the abs junk ib puts out as "aps"; y'all seem to be in the business of that right?

@mfrener
Copy link

mfrener commented May 20, 2022

Part of the reason I'm a bit cagey about all this is the additional support load that such a thing would entail. As soon as there is a Docker image or Compose file in the repo, there will be a flood of new questions from users.

I did https://github.com/waytrade/ib-gateway-docker (has been arborted) and I can confirm it will bring additional support effort, however the type of question is different to what you assume.
It's not really about docker, but about everything else, as this is "your" package that doesn't work. People starting with this often don't even know IB gateway app exists. It's not "I want to automate IB Gateway with IBC and this my problem: .." but "I run docker-compose up an it does work".
A Dockerfile will extend it this project from "we delvier an utility and need you to find out how to use" to "we delvier a ready to run package and you type a command to run it". If that command does not work, you will be aksed, even if IB servers are down and the connection issue has nothing to do with the Dockerfile at all ("docker-compose up" doesn't work anymore, it's obviosly a problem of the docker container).

If you want to add it to this probject, feel free to have a look at https://github.com/waytrade/ib-gateway-docker/blob/master/Dockerfile to have something to start with. Think you should be able to simply strip out the build-automation stuff (checksums, donwloading gateway from gh releases) and change it so it copies a local IBC and Gateway. And maybe find a better soltuion to using socat, caused troube for some users (randomd hangs because of stall connection between socat and gateway, we never figured the actual reason for, either problem disapeared or users stopped trying and went plan b (run your app on same container and connect to gateway directly)).

https://github.com/waytrade/ib-gateway-docker (waytrade/ib-gateway-docker#3 - love how this get's closed even though nothing has been solved 😂)

I can name about 100 people on the other side that use it successfully.

@goodboy
Copy link

goodboy commented May 20, 2022

https://github.com/waytrade/ib-gateway-docker (waytrade/ib-gateway-docker#3 - love how this get's closed even though nothing has been solved joy)

Woops, also linked to wrong issue (edited it), should have been waytrade/ib-gateway-docker#4
The solution being we'll need the beta from IBC (which afaict at that time wasn't yet released) doesn't really mean it's "solved". Further I'm trying to say it still isn't really working and I can prove it with an automated test if you want.

I can name about 100 people on the other side that use it successfully.

Fair enough, but either way there should be a test in CI to prove this all works with an automated client (bc why else are we automating all this if not to make API oriented trading work 😉).

As an example though, why is socat even really used. If you were to launch ib-gw and remove the local host filter, you should just be able to make connections over the docker net bridge; it's this kind of thing that i think IBC could help resolve and provide recommendations for.

Personally, i'd much rather collab here with the experts then do what I always end up having to do: writing this all from scratch myself, doing it correctly such that it's proven to work in CI, and then having to constantly worry about upstream changes which I don't have as much time for these days (i already am doing this for far too many projects 😂).

I would highly recommend a coordinated effort here in this repo at the very least with a recipe or super basic container setup script with automation to show it all coming up and working with CI runs we can point to when stuff doesn't work or a user claims it doesn't. When you have this, and can link to the workflow runs, i find that "users problems" tend to disappear when they can see the execution that proves they're problem isn't ours.

@rlktradewright doesn't even have to maintain the container part, that's what collabing with foss is all about; we can have certain people try to take care of certain things 🏄🏼

As an example of what my crew would be willing to contrib:

  • pytest automation to verify any container recipe works
  • a wrapper API to be able to control the IBC cmd server for things like restarting to work around the auto-logoff stuff
  • further magic to do things like data resets and re-connection via the secret key combos

I'll restate again, as a total outsider to this "space" for ib app "integrations", it's looking difficult to navigate I'd much rather see and get involved in something more collaborative:

  • less container repos
  • more tips / tricks then ad-hoc docker files
  • more CI testing bringing it all together for as many languages as possible

@mfrener
Copy link

mfrener commented May 20, 2022

Woops, also linked to wrong issue (edited it), should have been waytrade/ib-gateway-docker#4

That is just the perfect example of what I was talking about in term of support.
IB updates Gateway, IBC stops working because of it and the problem is the reported to the dockerfile project.

As an example though, why is socat even really used. If you were to launch ib-gw and remove the local host filter,

Try it. It's there for a reason.

I would highly recommend a coordinated effort here in this repo at the very least with a recipe or super basic container setup script with automation

Than please go ahead, @rlktradewright seem to be open for PRs.
My project was not made to serve the public with a basic container setup script with automation, it was something I implemented because I needed it and made it opensourcce so other can use it too.

@goodboy
Copy link

goodboy commented May 22, 2022

Try it. It's there for a reason.

I did, just run the network stack in "host" mode (network_mode: 'host'), done.
Makes the gw look exactly like if you were running it locally with no apparent net/sec downside afaict.
I don't know why you'd want to use docker's default internal networking settings anyway, it's generally awful to use. I would have done a bridge setup over default settings if you wanted to avoid host mode but 🤷🏼

Than please go ahead, @rlktradewright seem to be open for PRs.

@rlktradewright if you are indeed open for this I would consider putting in the effort since the crew i'm with also have other "workarounds" that y'all don't seem to be implementing here. As I said it would be mostly python oriented. Either way happy to keep eyes on this thread and contribute what we can.

@mfrener
Copy link

mfrener commented May 22, 2022

I don't know why you'd want to use docker's default internal networking settings anyway,

Because I develop on windows (there is no host network driver for Windows or Mac) and live system run in a VM as VPS on the internet. Just running Gateway container on host network would make it free for everyone on internet to use. If you want to go the linux-only way using host network driver, make sure to put a big fat warning somewhere so ppl don't start to expose their IBGateways on public networks (you need a firewall additionally, or block the ports otherwise, I woudn't rely on the "ip filter" of gatway - you don't want leave localhost with IB API protocol, see https://github.com/waytrade/ib-gateway-docker#leaving-localhost)

@rlktradewright
Copy link
Member

Sorry for the delay in getting involved in this issue again.

Can we please stop discussing @mfrener's project here: the discussion raises important issues that will need to be taken care of in any new Docker work here, but the details of such discussions are irrelevant just now (pursue it on the relevant repository if need be).

Yes, I am indeed open to PR submissions. There are two possibilities that interest me:

  • a sample Docker setup that others can build on or customise (there could be more than one of these, with different characteristics)
  • a full-fledged 'professional' productised version that could be submitted to Docker Hub as the 'official' IBC Docker image

Let me try to explain what I see as the difference between these two things.

A 'sample' Docker setup:

S1. Does (at least) the minimum necessary to get IBC/Gateway running under Docker.
S2. With the relevant username and password.
S3. With API connections enabled to localhost.
S4. It should honour any settings specified in config.ini.
S5. It should include the ability for the user to view the Gateway UI via VNC Viewer (or similar).
S6. It would be acceptable that any configuration changes made to Gateway via the UI would be for the current session only and not persisted between sessions.
S7. It would be acceptable that the user's API client application needs to be built into the Docker image.
S8. To use such a setup, the user would have to customise the DockerFile to meet their requirements, and then build the Docker image.

A 'professional' docker setup:

P1. Provides Docker capabilities for both Gateway and TWS (separate Docker images).
P2. Docker images are provided for different combinations of TWS/Gateway/IBC versions. Release of a new version of IBC, Gateway or TWS should require only that the the user run a different Docker image.
P3. A running Docker image behaves in all respects like a locally installed IBC/Gateway or IBC/TWS, to the maximum extent possible. An example of where this might not be possible is a change to the API port number in TWS/Gateways API settings: this might require restarting the Docker image with a different port mapping.
P4. No need for the user to modify the relevant DockerFile in any way.

As @mfrener pointed out above, his project might well be a suitable starting point for a sample DockerFile. I'll try to take a closer look at this some time, but I'd be happier if someone else did this and submitted either it or something similar as a PR, in line with the sample description given above.

@rlktradewright
Copy link
Member

@goodboy

Just responding to some points you mentioned earlier.

Regarding Python, I don't see what relevance Python has to creating a Docker image. Please clarify.

Regarding ib_insync, Ewald has done a fantastic job with this and it has dramatically eased the Python developer's experience with the API. I haven't looked in detail at ib_insync recently, but I believe it invokes IBC if required to get TWS/Gateway running on the local machine. Clearly if we provide a Docker image for IBC, Ewald might want to make use of that and this would presumably require some re-engineering within ib_insync. But it's not clear to me that this would have any impact on our Docker capability (except perhaps in the special case of a Docker image containing the user application and the ib_insync API). At this stage, I don't think we need to concern ourselves with that - in any case, Ewald might prefer that such a Docker image be under his control in his repository.

@goodboy
Copy link

goodboy commented Jun 6, 2022

Regarding Python, I don't see what relevance Python has to creating a Docker image. Please clarify.

Mostly that the most popular API client (afaict, and of which I know you're well aware of) is the ib_insync project, so it feels natural to me to test such a container recipe against the client that you'd expect most FOSS users to be trying to integrate. Maybe this is a bad assumption?

Further, python is often used via the docker package to manage containers from python code (this is what my crew is planning on doing and where the docker-compose project originated from) so I was suggesting a full system test suite for such a setup that at the very least would provide an example for other users wanting to use other languages for such things. I don't think it's too too abnormal to suggest that most trading peeps are probably at least going to building a prototype in python first before moving to something, more enterprise 😂.

but I believe it invokes IBC if required to get TWS/Gateway running on the local machine. Clearly if we provide a Docker image for IBC, Ewald might want to make use of that and this would presumably require some re-engineering within ib_insync.

Definitely! There is both existing ibc module code and an ongoing "fyi" issue regarding a container solution, plus another container issue that got closed: erdewit/ib_insync#261.

Ewald might prefer that such a Docker image be under his control in his repository.

I don't think that's true if you look at the above issues. I think erdewit is probably approaching this the way I would, ib_insync is only an api client lib, not a full system solution.

Also note, I'm not suggesting we put any theoretical consensus container recipe inside any particular code base like IBC or ib_insync. I'm just trying to get all the people who are experts in the things ib to maybe have their minds aware of a project which is trying to unify these projects for the benefit of the overall hacker-trader community so that we can all avoid having to deal with the headaches that are ib's frankly, annoying to use apps for anyone trying to do automatic trading. I would much prefer not to have to audit 5 container repos all of which i have to tweak to get working with ib_insync and all of which have no input (or [subconscious] attention) from configuration experts (such as yourself and Ewald).

I guess for me it'd be nice to have a unified place to start getting everyone working together to create a least a template of a solution for this whole thing. I much prefer all the minds making things better through PRs then disparate forks of slight DOCKERFILE diffs all over github..

But hey maybe that's an unpopular opinion 😂

@rlktradewright
Copy link
Member

I think we're not on the same page.

What I'm looking to create is a simple Docker image that runs IBC to start Gateway or TWS. Nothing else at all. When using such an image, the API application (and hence the API itself) will be in a different process, which may be another Docker image, perhaps starting both using Docker Compose, or that process could be on another computer anywhere in the known universe (provided it has an internet service and a reasonable ping time).

So who cares what API or application you use to test it? As long as the correct API messages arrive at the running Gateway, nothing else about the client matters.

I'm not going to speculate about what Ewald is or isn't going to do regarding Docker. He's more than capable of doing whatever fits with his way of doing things, and if he wants to contribute to this effort here, he's very welcome.

@goodboy
Copy link

goodboy commented Jun 7, 2022

What I'm looking to create is a simple Docker image that runs IBC to start Gateway or TWS. Nothing else at all.

@rlktradewright how do you feel about nixOS?

I much prefer something that avoids as much DOCKERFILE as possible 😂

package for the old project that i'd like to update here:
https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/office/ib

When using such an image, the API application (and hence the API itself) will be in a different process, which may be another Docker image, perhaps starting both using Docker Compose, or that process could be on another computer anywhere in the known universe (provided it has an internet service and a reasonable ping time).

Right, and end-to-end system tests for all of these scenarios are not only super easy to write these days but also pretty commonplace especially with tools like docker, pytest, docker-py.

will be in a different process, which may be another Docker image,

Or just a different process on the test suite host.
I mean if you really wanted to could write test in multiple langs as well, there is normally a plethora of lang support in github actions for example.

the main thing I think is important is that you can't easily test client code/projects for the ib API without the automation from IBC, and the most sane way to put that all together to test a client in CI is to do it with containers..

I'm not proposing that IBC build out any such integration tests, just saying if we had an official image, then those of us wishing to use and improve it could implement such tests in our repo image (fork) CIs and then report issues that much easier when things change in the official image.

Anyway just an idea, nothing to do with ib_insync or IBC doing test work or stuff with a specific client.

@rlktradewright
Copy link
Member

Please can you just stick to the point.

NixOS has no bearing whatever on this thread.

We've already established the desire for an official Docker image. You said you wanted to contribute a pull request. Well then, contribute it instead of wasting time on peripheral matters.

@goodboy
Copy link

goodboy commented Jun 10, 2022

NixOS has no bearing whatever on this thread.

It does if i make the docker image based on a nixos image?
Not really "peripheral", I'm just asking if you'd be ok with that 😂

I'm asking if we could do an image based on nix so that the image build/configuration can be almost entirely in nix's config language instead of having to hack to together shell scripts for everything and shipping them alongside the build.
I also think you can "export" docker images from a nix config, but gotta dig into that since I've never tried it before.

Most IBC docker images on github are based on unbuntu, which i personally try to avoid.
I'd also be open to using alpine or something on an actual "container oriented" distribution; i.e. something super minimal and ideally fast to build.

Anyway, I'm not trying to be "off topic" i'm just trying to test the temperature for a "non-mainstream" distro as the basis for an image.

@rlktradewright
Copy link
Member

Ok, I've spent quite a bit of time looking into Nix (which I wasn't aware of before you mentioned it), and it looks like a very sensible approach to creating packages.

I don't have any objection to you using Nix to create an IBC package, and IF a sensible Docker image can be automatically generated/exported from the Nix package then you're killing two birds with one stone, since some might prefer to use the Nix package directly. Which is obviously a good thing.

So if you want to follow that route then fair enough. However the thrust of this thread is to end up with a Docker image, because that's what people are asking for, so it would be a shame if you spent a lot of time on this approach and then found that it wasn't going to fly for some reason.

But I'm encouraged to see that, for example, MySQL has a Nix image (and an impressively huge dependency tree!), which indicates that it should have no trouble coping with IBC which is trivial by comparison. It all hinges on the Docker export capability.

@goodboy
Copy link

goodboy commented Jun 14, 2022

Ok, I've spent quite a bit of time looking into Nix (which I wasn't aware of before you mentioned it), and it looks like a very sensible approach to creating packages.

🥳 i agree very much!

, and IF a sensible Docker image can be automatically generated/exported from the Nix package then you're killing two birds with one stone, since some might prefer to use the Nix package directly. Which is obviously a good thing.

Slick! Yeah, this was one of the reasons I've started moving infra over to this "distro", package-manager, whatever 😂

So if you want to follow that route then fair enough. However the thrust of this thread is to end up with a Docker image, because that's what people are asking for, so it would be a shame if you spent a lot of time on this approach and then found that it wasn't going to fly for some reason.

Agreed. I haven't actually personally done any docker export stuff yet so will report back once of our crew (or myself) has a working POC.

@rlktradewright again appreciate all the patience with this back and forth 😺

@goodboy
Copy link

goodboy commented Jun 21, 2022

Bleh, ran into an additional "fun" issue with docker and the current modified waytrade image setup which is documented in this comment.

I wonder has anyone tried to solve this problem before where a user would like to run a paper and live account simultaneously on the same host but one or the other in docker and the other as a native app?

It seems as though the ib app blobs profile the "host ip" (it's definitely not doing that since this whole issue is seen on a single host 😂 ) as different even when running the app inside a docker container in --net=host mode..

Further, some follow up questions (maybe for rlktradewright):

  • If you want to run 2 or more IBC instances on the same host what are the restrictions?
    • i noticed the config file's ExistingSessionDetectedAction setting and don't entirely grok it: I don't get if the reasons for this setting apply if you are using multiple sign ins, different creds for paper and live accounts.
    • does the command server settings have to be set to independent sockets manually or does that take care of itself? ( i noticed you only set the host part and the port part to 0 which seems to disable the cmd server by default?)

@rlktradewright
Copy link
Member

I don't think there's going to be a way around this (short of persuading IB to somehow make it possible).

I run several virtual machines on the same host, and I have to run both live and paper sessions in the same virtual machine to avoid the problem - the fact that they share a common virtualisation host doesn't help.

As far as I can see, if you want to run live and paper-trading at the same time, and either is in a Docker image, then it will be necessary to use two different logins and pay the price of the separate market data subscriptions.

A completely bizarre idea that occurred to me is the possibility of running more than one instance of Gateway/TWS within the same JVM. It would be straightforward for IBC to call the Gateway/TWS entrypoint more than once, but I guess it would be very surprising if there were no shared data areas or whatever within the programs that would need to be separated between the different instances for them to work properly. And IBC would somehow have to know which instances any particular dialog was initiated by. There are just too many ways such a scheme could, and almost certainly would fail. But the idea is preying on my mind!...

Regarding the ExistingSessionDetectedAction setting, here are two scenarios where you might appreciate its value:

  1. You run your live TWS/Gateway in your office automatically, starting it early every morning, and it shuts down late at night. One evening, you want to try fiddling with some of the settings, so you run it up at home after it has finished in the office, and you try out whatever you're interested in, and then you forget to shut it down (you know, the baby wakes up or you get plastered, whatever). Next morning your office instance can't start because the account is already logged in at home. But if you set your office instance to ExistingSessionDetectedAction=primary, it will start properly and the home session will be terminated (this, if I remember correctly, is something like the scenario described by a user that led to development of ExistingSessionDetectedAction, though he didn't mention babies...).

  2. You run your live TWS/Gateway somewhere, starting it automatically and restating periodically if it has stopped running. You set it to ExistingSessionDetectedAction=primaryoverride. When you're out and about you can run the IBKR Mobile app on your phone, and log in to the live account (for example to place a trade manually), and this ends the TWS/Gateway instance. If TWS/Gateway tries to restart while you're still using the IBK Mobile app, it won't be able to. But as soon as you stop using the mobile app, the next restart will succeed. [Disclaimer: I've only recently acquired a non-Microsoft phone, and haven't got round to explicitly testing this sort of scenario, so use with care.]

I didn't quite understand your question about the command server settings. Please clarify.

@goodboy
Copy link

goodboy commented Aug 18, 2022

First sorry about the delay and thanks for the response!

As far as I can see, if you want to run live and paper-trading at the same time, and either is in a Docker image, then it will be necessary to use two different logins and pay the price of the separate market data subscriptions.

I actually only need one data feed profile (since you can share live feeds with paper accounts) and the way we use it is by normally having the paper be the data feed and the live is only for order mgmt. This seems to actually work just fine except when trying to pull historical data oddly, which throws the "something something another client on another IP" (which it's not) error.

I didn't quite understand your question about the command server settings. Please clarify.

I'm more or less trying to figure out how to run 2 instances of IBC on the same host such that i can run both a paper and live gateway simultaneously. So I'm asking which settings do I need to make this possible?

My theory is that if you run both gw blobs in the same container net env then this "different IP address" error might go away; figure it's a least worth a shot.

@goodboy
Copy link

goodboy commented Aug 18, 2022

Yeah so reading ExistingSessionDetectedAction more thoroughly I don't think I'm hitting this. Currently when i try to start 2 containers one of the internal IBC processes will always terminate; so i'm trying to not have that happen.

@rlktradewright
Copy link
Member

I think maybe you're not fully understanding the restriction on sharing the data feed between paper and live instances on different hosts. You can certainly start them both and they'll run, and one of them (presumably the first one to make a request) will be able to use market data, historical data, etc, but as soon as you try to use any data-returning function on the other instance you'll get the "something something another client on another IP" message. I don't believe you'll be able to get round this, no matter what you do at the networking level.

To illustrate this, I normally run both live and paper accounts under the same host username on a Windows server. If I need to look at the GUI, I use a remote desktop (RDP) session. Now Windows Server allows two simultaneous RDP sessions running under different users. But if I do this, with live and paper instances running under different host usernames, I get the data sharing problem. So these are two instances running in the same physical host, using exactly the same network adaptor, and yet it still gets caught out.

I suspect that everything has to be the same for both instances: physical host, virtual host, IP address, MAC address, operating system username: in other words, effectively the same desktop: if you can't actually 'see' both instances on the same desktop, then you can't share the data.

So I think you're wasting your time trying to defeat this restriction.

By the way, you don't need to make any different settings to run two instances on the same host, except for the IB username and password, and of course setting the paper trading mode for one of them. Note in particular that ExistingSessionDetectedAction is nothing whatever to do with running different instances under different IB credentials: it's only relevant when you try to run multiple instances under the same user.

@goodboy
Copy link

goodboy commented Aug 18, 2022

You can certainly start them both and they'll run, and one of them (presumably the first one to make a request) will be able to use market data, historical data, etc, but as soon as you try to use any data-returning function on the other instance you'll get the "something something another client on another IP" message. I don't believe you'll be able to get round this, no matter what you do at the networking level.

I already am doing this, have been for more then a year and it works fine presuming I run one account as "native" and not under IBC and the other inside the container + IBC with the only caveat is that I have to use the live instance running native for the data feeds when running both, but I can still manage orders in the paper instance no problem. If I run both the gw instances native then there is no problem on which is used for the data feeds iirc (but maybe i'm wrong on this gotta double check).

So that's not the issue. I'm trying to run 2 instances with IBC (containerzed) but can't seem to avoid one instance always terminating when I start the other in a 2nd container. I'm currently running containers in docker's "host networking mode" (not supported on windows and i link to it below) merely to avoid doing bridged networking config and only because it's no really for my purposes yet.

I suspect that everything has to be the same for both instances: physical host, virtual host, IP address, MAC address, operating system username: in other words, effectively the same desktop: if you can't actually 'see' both instances on the same desktop, then you can't share the data.

As I said, no this isn't required. I can run one of a paper or live in container + IBC and the other as native (aka I run ./ib-gw from terminal on linux) and everything works just fine. And it's not MAC nor username (afaict) as I've already tried hacking all that (at least I think though I should probably double check this).

effectively the same desktop

With docker's host mode this should be true minus the user.

So I think you're wasting your time trying to defeat this restriction.

I don't think so 😂
Being able to run paper and live together is immensely useful for forward testing as well as many other auto-strat auditing tasks and market conditions analysis.

is nothing whatever to do with running different instances under different IB credentials: it's only relevant when you try to run multiple instances under the same user.

Gotcha, ok then maybe something else is up that needs diggin.
Heck it could be something docker related I'm not thinking off.

Thanks again for the response 👍🏼

@goodboy
Copy link

goodboy commented Aug 18, 2022

Andd don't worry it was all just me being an idiot with docker-compose 😂

So might be closer to something useful 🏄🏼

Heh, yup got it workin both in containers but still needs the api client to talk to the live account one.

Really not sure what that's all about since if you direct run the gws you can do the paper case which I just double checked.

@Mikkel84
Copy link

Mikkel84 commented Sep 4, 2022

Just to contribute to this interesting debate: https://github.com/manhinhang/ib-gateway-docker

@dvasdekis
Copy link
Author

dvasdekis commented Oct 11, 2022 via email

@goodboy
Copy link

goodboy commented Feb 15, 2023

Just to contribute to this interesting debate: https://github.com/manhinhang/ib-gateway-docker

This is cool but one of the benefits of using the VNC stuff is being able to hack historical query throttles.

I might take a look at extending with some options for running with and without the VNC bit.

@rlktradewright
Copy link
Member

@goodboy

Are you still progressing this? If so, how's it going? If not, please say so, so that other options can be pursued.

@extrange
Copy link

https://github.com/extrange/ibkr-docker

Fully containerized TWS/IB Gateway, no external dependencies
TWS API access automatically configured and forwarded
Viewable in a browser (via noVNC)
Auto-restart, auto-login TWS/IB Gateway automatically via IBC Alpha

@goodboy
Copy link

goodboy commented Apr 26, 2023

@rlktradewright hey! sorry yes we are, but i don't yet have anything to publish unfortunately..

we will likely host the nixconfig in our org and i will link to it here to start.
if the IBC community ends up liking it the we can move wherever you'd like and maintain from there.

@rlktradewright
Copy link
Member

@goodboy Thanks for that. I'll look forward to seeing whatever you come up with.

@gnzsnz
Copy link

gnzsnz commented Oct 2, 2023

in case anybody finds it usefull, i'm maintaining a docker image with IB gateway and IBC ib-gateway-docker

it includes IB gateway, IBC, socat to route IB gateway port and optionally VNC.

The build is triggered by every new IB gateway release. And it pulls latest IBC as well.

@goodboy
Copy link

goodboy commented Oct 11, 2023

@goodboy Thanks for that. I'll look forward to seeing whatever you come up with.

@rlktradewright yaahh super sorry I've let this slide again. literally stuck in one of them life-task mud puddles (spraying slop everywhere) the past month getting a buncha stuff out of the way that has been on the backburn wayy too long 😂

I'm still pushing on nixos stuff and will definitely be working on this at some point, ideally in the next month - literally right now i have no working container (which is frankly a nightmare) since IB just deprecated the last one supported by the (now abandoned) project that i mentioned using prior (i think in thread?).

On the flip side i've been able to compile a list of of nix-ops related projects that I want to try and that will likely relate to the particular container rendering sys we decide on.


@gnzsnz dope. literally gonna check this out tonight to see if i can get it workin 😎

thanks for linking it 👍🏼

@gnzsnz
Copy link

gnzsnz commented Nov 3, 2023

All,

As mentioned on my previous comment, I've been working on a docker image to run IB gateway & IBC. The image is a fork from UnusualAlpha/ib-gateway-docker, which in turn is based on waytrade/ib-gateway-docker. I have added my own sauce on top.

This thread is really useful as it has allowed me to consolidate requirements for an ib gateway image.

The current image version is covering most of the requirements. I understand that the objective of this thread is to have a "recommended" Dockerfile, but after working on the docker image I can say that a "Dockerfile" only is not going to work. You need a Dockerfile and a script to orchestrate the execution within the container. I'm not sure that this has been considered/discussed. Plus provide settings/config to the container, which is better suited for a tool like docker compose or more complex tools (do we really need kubernetes for this?). On top of that IB gateway gets obsolete quite fast, specially the latest branch. Which i have solved using github actions. I think this has not been discussed, and the implication is that more than a "Dockerfile" is needed.

Please feel free to used the Dockerfile and scripts required to run the container, plus extra things like config files pre-tunned for a container, and a sample docker-compose.yml. I'm doing some heavy refactoring at the moment, so you might want to check the sshclient branch which has changes to support ssh tunnels.

Based on the requirements described on this tread I have done a match with the image capabilities. I would love to get the community feedback.

A sample Docker setup:

S1. Does (at least) the minimum necessary to get IBC/Gateway running under Docker.

A1. It starts IBC/gateway, in headless mode with Xvfb and x11vnc.

S2. With the relevant username and password.

A2. username and password passed as environment variables. and dynamically inserted into config.ini. this is to avoid using ibctart.sh parameters [--user=userid] [--pw=password] which can expose sensitive information to the host.

S3. With API connections enabled to localhost.

A3. The API port can be exposed in different ways.

  • host network(insecure). using socat
  • host loopback(127.0.0.1), more secure than host network. but still not secure.using socat
  • container network. containers in the same network can use it. more secure than previous options, but host root can still access it. using socat
  • through an ssh tunnel, not yet released. but been tested at the moment. most secure option. ssh tunnel available for VNC as well (as VNC is not a secure protocol)

S4. It should honor any settings specified in config.ini.

A4. This is achieved in different ways

  • common settings can be pass through environment variables
  • as an alternative config.ini and Jts.ini can be mounted from the host to the container.
  • ideally a sed script should parse all options as environment variables and update config.ini dynamically. I have put this aside, as the first option is tackling all my current needs and in case is needed option 2 is available

S5. It should include the ability for the user to view the Gateway UI via VNC Viewer (or similar).

A5. VNC is available, and can be accessed through ssh tunnel.

S6. It would be acceptable that any configuration changes made to Gateway via the UI would be for the current session only and not persisted between sessions.

A6. by default settings are not persistent. they are stored in .env file and passed when container start. optionally a ib gateway settings volume can be used, this has the benefit of allowing changes in ibgateway that are not supported by IBC, as they can be done through GUI/VNC.

S7. It would be acceptable that the user's API client application needs to be built into the Docker image.
A6. I have taken a "separation of concerns" approach. this is an IB gateway end point, any extra is there to support this main purpose (IBC, VNC, socat, SSH tunnel, etc).


S8. To use such a setup, the user would have to customize the DockerFile to meet their requirements, and then build the Docker image.
A8. This remains as an option. But please don't open issues on my repository for this type of setups.

A 'professional' docker setup:
P1. Provides Docker capabilities for both Gateway and TWS (separate Docker images).

AP1. not possible, this is just an ib gateway image. I found that linuxserver/rdesktop image works great with TWS, and with the option of entering credentials once per week I haven't even installed IBC(sorry about that). it has firefox, libreoffice, and ssh so i can tunnel API port. and TWS persists containers as it's installed in /config folder, it can even upgrade. I think that the effort to have this working would be too big, when a simple solution that only requires a login once per week covers 99.99% of the needs.

In addition, one of my objectives is to keep the ib-gateway-docker image as small and lean as possible. Resources should go to the algos, not to the broker bridge.

P2. Docker images are provided for different combinations of TWS/Gateway/IBC versions. Release of a new version of IBC, Gateway or TWS should require only that the the user run a different Docker image.

AP2. Every IB gateway release is stored as a release and an image build for it, this is automated through github actions. Even new IBC releases are monitored, if a new IBC release is available a PR is created to update the Dockerfile.

P3. A running Docker image behaves in all respects like a locally installed IBC/Gateway or IBC/TWS, to the maximum extent possible. An example of where this might not be possible is a change to the API port number in TWS/Gateways API settings: this might require restarting the Docker image with a different port mapping.

AP3 My objective has been to expose IB gateway API port "as a service". IB gateway uses standard ports and ports are exposed to the outside where and how it's needed, but using docker-compose.yml or docker cli. ex socat or ssh tunnel. for ex with an ssh tunnel you can point at any port on the remote host.

P4. No need for the user to modify the relevant Dockerfile in any way.
AP4 For 99% of the cases this is already the case. when the default settings do not work see A4. Regarding S7/S8, the philosophy that I have followed is that the images does one thing and only one thing, expose the API port.

Sorry for the wall of text. Hope you find this useful.

@bfoz
Copy link

bfoz commented Nov 3, 2023

@gnzsnz Have you looked at voyz/ibeam? They're already going down the path that you outlined above.

@gnzsnz
Copy link

gnzsnz commented Nov 3, 2023

@bfoz thanks for raising it. yes i did see it some time ago. The thing with ibeam is that it works with web api gateway, which at the moment does not have all the features than TWS/IB gateway. At some point i expect IBKR to unify their APIs.

I hope that web api gateway is not going to remain in it's current form. I would prefer to avoid any gateway to be honest. A REST api would be great, but we are not there yet.

@bfoz
Copy link

bfoz commented Nov 3, 2023

I would prefer to avoid any gateway to be honest. A REST api would be great, but we are not there yet.

🤞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests