Overcome two factor authentication

[vishwajeetio]

Describe the feature
I am trying to access the API using a cash account but it's not working and giving the login failed message while it's working perfectly fine with a paper account. Can you tell me how can I fix that or if possible can we add that authentication feature?

Expected interaction
Using a two-factor authentication device

Possible implications
Only cash account authentications

Additional context
Please suggest to me if there's any workaround to that

[Voyz]

Thanks for the suggested enhancement 😊 Am I understanding correctly you're trying to figure out how to overcome the two-factor authentication with IBeam?

[vishwajeetio]

yeah, I tried all the way but I am still unable to use my cash account on the container. can you suggest to me how can I use my main account with that? Also, I am willing to put effort into it to make the two-factor auth possible somehow using the IBKR app. Please help me know if there's something I missed out on.

[Voyz]

Right, I see. Unfortunately, IBeam does not handle 2FA automatically, therefore if this is what's stopping the authentication on your cash account then there is nothing that can currently be done by default. You'd need to complete the 2FA each time using your 2FA method or find another method to complete it automatically.

If however it isn't the 2FA then I'd suggest you check that you don't have any other places - such as TWS or web services - where you use the cash account credentials.

Could you share the IBeam output log in case there's something there that could indicate the problem? Please run it with IBEAM_LOG_LEVEL environment variable set to DEBUG.

[jembl]

I have solved this problem, contact me to help you out.

[vishwajeetio]

I have solved this problem, contact me to help you out.

that sounds great.
Can you please explain to me how can I do that?
thank you

[Voyz]

hey @jembl super cool you found a way around it! Could I ask you to share your solution here so others can use your findings in the future too? Thanks! 😊

[jembl]

yes, so here is how I did it:

1. I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.

2. I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

3. use this code to parse the SMS on the server side and upload it to a storage of my choice
   `public String receive(String message) {
   StringBuilder sb = new StringBuilder();
   ResourceSet messages = Message.reader().limit(10).read();

    for(Message record : messages) {
        if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
            String messageSid = record.getSid();
            String messageBody = record.getBody();
            try {
                log.info("received message with SID " + record.getSid());
                log.info("received message with BODY " + record.getBody());
                String[] splits = messageBody.split(":");
                if (messageBody.startsWith("Your requested authentication code: ")) {
                    String code = splits[1].trim();
                    twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
                }
                sb.append(messageSid + "\n");
            } catch (Exception e) {
                log.error("Failed to process the message " + messageBody, e);
            } finally {
                Message.deleter(messageSid).delete();
            }
        }
    }
   
    return sb.toString();
   

   }`

• On the container side i read the code from storage after login inside gateway_client.py:
  ` sf_select_present = EC.presence_of_element_located((By.ID, _SF_SELECT_EL_ID))
  WebDriverWait(driver, 15).until(sf_select_present)
  _LOGGER.info("sf_select has appeared")
  sf_select_el = Select(driver.find_element_by_id(_SF_SELECT_EL_ID))
  sf_select_el.select_by_value('4.2')
  _LOGGER.info("sf_select one time passcode")

    two_factor_input_present = EC.presence_of_element_located((By.ID, _TWO_FAC_EL_ID))
    WebDriverWait(driver, 15).until(two_factor_input_present)
    time.sleep(45)
    two_factor_el = driver.find_element_by_id(_TWO_FAC_EL_ID)
    _LOGGER.info(two_factor_el.text)
    two_factor_el.send_keys(get_latest_authentication_code())
  
    _LOGGER.info('Submitting the form for two fac')
    submit_form_el = driver.find_element_by_id(_SUBMIT_EL_ID)
    submit_form_el.click()
  
    success_present = EC.text_to_be_present_in_element((By.TAG_NAME, 'pre'), _SUCCESS_EL_TEXT)
    WebDriverWait(driver, _OAUTH_TIMEOUT).until(success_present)
    _LOGGER.info(driver.page_source.encode("utf-8"))
  
    _LOGGER.debug('Client login succeeds')`
  

and the code to get SMS from storage:
def get_latest_authentication_code(): connection_string = "YAYAYAYAYYA" blob_client = BlobClient.from_connection_string( connection_string, container_name="sms", blob_name="code.text") # [START download_a_blob] _LOGGER.info("downloading the code from blob") download_stream = blob_client.download_blob() code = download_stream.content_as_text() _LOGGER.info("done downloading the code from blob ") _LOGGER.info(str(code)) return code

Hope this helps, ask me if you have any questions!

[Voyz]

That is fantastic @jembl! 👏

Thank you so much for following up with such a detailed explanation! I was hypothesising that such an approach would be doable but I haven't looked that direction yet. Super happy to see this method worked for you and seriously congrats on devising it on your own!

Hey would you be interested in contributing some of these changes to IBeam? So that conditionally IBeam could be set to download the code from a URL (or some other way) and paste it to the login page. This way IBeam's users would only need to set up the SMS client and a pipeline to store the code. I could help you get this done, although no worries if you're busy or not interested, this answer is already a massive help 😊

[vishwajeetio]

thank you so much @jembl
I understand what you are trying to implement here and got the Twilio part working now working on adding this code to the container. I am still in doubt on exactly at what part do I need to add these lines.
thank you

[Voyz]

Hey @odssh and @jembl 👋 I've got some good news! There's a new version of IBeam currently in testing that should facilitate implementing this pattern. In this new version you will have an opportunity to set up IBeam in a way that will attempt to pull the 2FA code from an external URL.

You can try it out by using the image with tag 0.3.0-rc1, eg:

docker pull voyz/ibeam:0.3.0-rc1

Then when running it, do the following:

• Set IBEAM_TWO_FA_HANDLER env var to EXTERNAL_REQUEST
• Set IBEAM_EXTERNAL_REQUEST_URL env var to the URL you'd like to call
• Set other env variables according to your requirements, see the external_request_handler.py

---

Alternatively, if you require full customisation in how you acquire the code you can do the following:

• Provide the Inputs Directory with a script containing a class inheriting the TwoFaHandler and implementing the get_two_fa_code method - synchronously returning the 2FA code upon completion.
• When running IBeam, set:
  • IBEAM_TWO_FA_HANDLER to CUSTOM_HANDLER
  • IBEAM_CUSTOM_TWO_FA_HANDLER to your_module_name.YourHandlerClass, corresponding to the script you provided in the Inputs Directory. Eg. for a file called my_handler.py and a class MyTwoFaHandler you'd set it as my_handler.MyTwoFaHandler.

Let me know if you'd have any feedback! 😊

[vishwajeetio]

Hi,
I did test that and put it to run. It ran fine for a few hours so I put it on run for two days and I just checked on it and I got about more than a thousand messages. Something is really wrong.
I am trying to identify it. Please let me know if you can suggest to me where it could break or any potential reason for that bug.
thank you

[Voyz]

Hi @odssh thanks for testing it! Sorry to hear it started breaking after a while. Could you copy the output log over to here so I could give it a go at figuring out what's wrong? Thanks!

[vishwajeetio]

Hi @Voyz ,
Here's the log file for the whole session. It worked almost fine for the first day and then I got locked out of my account because of the excessive number of login trials as it kept making multiple login attempts each time it needed to log in.
In the log file, the line below is printed by me:
{"log":"Your requested authentication code: 433235\n","stream":"stdout","time":"2021-02-21T23:22:47.664576083Z"}
Please let me know if there's any fix for these multiple login attempts as it causes the account to get locked out from the Interactive broker's side.
thank you

[Voyz]

That's fantastic @odssh thanks for passing this over and for the explanation! I'm really sorry you hit the lock out from IB - it happened to me too when I was testing this new method. I think it may be a good idea to implement some safety mechanism that will stop IBeam from attempting after a number of times as to prevent the lock-out from happening.

As for the output log, I'd have a number of questions:

1. Are you using IBeam as a Docker image, or standalone?

2. How are you printing this line? Your requested authentication code: 433235? This must come from some custom code, yet IBeam doesn't support that other than through CustomTwoFaHandler.

3. How is this message printed: Did not receive the authentication code? I cannot find it anywhere in IBeam. Have you modified it in some way? Could you let me see your fork if that's the case?

4. What IBEAM_TWO_FA_HANDLER are you using? The log seems to suggest that you're using the EXTERNAL_REQUEST TwoFaHandler, however I'm surprised to see that when it prints in the log it does not contain any URL field, ie. url=None:

   2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from:  ExternalRequestTwoFaHandler(method=GET, url=None, timeout=300, params=None, headers=None)
   

   If so, can you confirm that you're setting the URL correctly using IBEAM_EXTERNAL_REQUEST_URL?

5. It seems the 2FA code acquisition you perform is not acting fast enough. In multiple places in the output log I can see that IBeam requests the code, then waits and eventually times out. Only once that happens, your custom code seems to print the 2FA code. Could you share how your 2FA acquisition code looks like just to verify it isn't the issue?
   (example of that behaviour):

   2021-02-21T23:22:28.233255303Z -- 2021-02-21 23:22:28,232|I| Credentials correct, but Gateway requires two-factor authentication.
   2021-02-21T23:22:28.233567968Z -- 2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from: ExternalRequestTwoFaHandler(   method=GET, url=None, timeout=300, params=None, headers=None)
   2021-02-21T23:22:32.279821579Z -- 2021-02-21 23:22:32,279|E| Did not receive the authentication code
   2021-02-21T23:22:32.280179965Z -- 2021-02-21 23:22:32,279|I| No 2FA code returned.
   2021-02-21T23:22:47.55094453Z -- 2021-02-21 23:22:47,549|E| Error encountered during authentication
   2021-02-21T23:22:47.550985101Z -- Traceback (most recent call last):
   2021-02-21T23:22:47.55099196Z --   File \"/srv/ibeam/src/authenticate.py\", line 164, in authenticate_gateway
   2021-02-21T23:22:47.551019941Z --     trigger = WebDriverWait(driver, var.OAUTH_TIMEOUT).until(any_of(success_present,    error_displayed))
   2021-02-21T23:22:47.551026047Z --   File \"/opt/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py\", line 80, in    until
   2021-02-21T23:22:47.551031437Z --     raise TimeoutException(message, screen, stacktrace)
   2021-02-21T23:22:47.551036576Z -- selenium.common.exceptions.TimeoutException: Message: 
   2021-02-21T23:22:47.551041561Z -- 
   2021-02-21T23:22:47.551046234Z -- 
   2021-02-21T23:22:47.551050865Z -- The above exception was the direct cause of the following exception:
   2021-02-21T23:22:47.551055927Z -- 
   2021-02-21T23:22:47.551060554Z -- Traceback (most recent call last):
   2021-02-21T23:22:47.551065397Z --   File \"/srv/ibeam/src/authenticate.py\", line 176, in authenticate_gateway
   2021-02-21T23:22:47.551070461Z --     raise RuntimeError('Error encountered during authentication') from e
   2021-02-21T23:22:47.551075534Z -- RuntimeError: Error encountered during authentication
   2021-02-21T23:22:47.664339484Z -- 2021-02-21 23:22:47,663|I| Authentication process failed
   2021-02-21T23:22:47.664509274Z -- Your requested authentication code: 433235
   

   Note how the authentication fails due to no code being returned at 23:22:32, while your message is printed at 23:22:47, over 10 seconds later. It seems that the code is received a little too late. It would be great if we could figure out why that hold happens.

If you could feed me in on as many details regarding these questions as possible, I should be able to understand what is causing the issue on your end. Appreciate your help in the matter!

[vishwajeetio]

thank you for your quick response @Voyz.
Here's the python code that I have modified. In this file, I have changed only lines 12, 13, and 60-118. I know I should have gone with two_fa_handler.py but at the moment external_request_handler.py seemed easier.
Here are the answers to your question:

1. Docker
2. line 116
3. line 85
4. EXTERNAL_REQUEST
5. the code in the file attached checks for verification messages constantly and returns it as soon as it gets it.
   Right now I am not sure what causing the issue.
   Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".
   thank you

[Voyz]

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials.

import json
import logging
import os
from pathlib import Path
from typing import Union

import requests

from ibeam.src.two_fa_handlers.two_fa_handler import TwoFaHandler


from datetime import datetime, timezone
from twilio.rest import Client




_LOGGER = logging.getLogger('ibeam.' + Path(__file__).stem)

_EXTERNAL_REQUEST_METHOD = os.environ.get('IBEAM_EXTERNAL_REQUEST_METHOD', 'GET')
"""Method to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_URL = os.environ.get('IBEAM_EXTERNAL_REQUEST_URL')
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_TIMEOUT = int(os.environ.get('IBEAM_EXTERNAL_REQUEST_TIMEOUT', 300))
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_PARAMS = os.environ.get('IBEAM_EXTERNAL_REQUEST_PARAMS')
"""Params to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_HEADERS = os.environ.get('IBEAM_EXTERNAL_REQUEST_HEADERS')
"""Headers to use by the external request 2FA handler."""


def parse_json(s):
    if s is None:
        return None
    try:
        return json.loads(s)
    except Exception as e:
        _LOGGER.exception(e)
        return None


class ExternalRequestTwoFaHandler(TwoFaHandler):

    def __init__(self, method: str = None,
                 url: str = None,
                 timeout: int = None,
                 params=None,
                 headers=None):

        self.method = method if method is not None else _EXTERNAL_REQUEST_METHOD
        self.url = url if url is not None else _EXTERNAL_REQUEST_URL
        self.timeout = timeout if timeout is not None else _EXTERNAL_REQUEST_TIMEOUT
        self.params = params if params is not None else parse_json(_EXTERNAL_REQUEST_PARAMS)
        self.headers = headers if headers is not None else parse_json(_EXTERNAL_REQUEST_HEADERS)

    def get_two_fa_code(self) -> Union[str, None]:
        try:
            # response = requests.request(method=self.method,
            #                             url=self.url,
            #                             timeout=self.timeout,
            #                             params=self.params,
            #                             headers=self.headers, )
            # response.raise_for_status()
            # return response.content
            return self.twilioAuthToken()

        except requests.exceptions.HTTPError as err:
            _LOGGER.error(err)
            return None


    def twilioAuthToken(self):
        vCode = None
        for i in range(20):
            vCode = self.getNewMessage()
            if vCode != None:
                break
            else:
                pass
        if vCode == None:
            _LOGGER.error("Did not receive the authentication code")
        print("got it", vCode)

        return vCode

    def getNewMessage(self):
        vcode = None
        account_sid = 'ACCOUNT_SID'
        auth_token = 'AUTH_TOKEN'
        dtf = '%Y-%m-%d %H:%M:%S'
        client = Client(account_sid, auth_token)

        messages = client.messages.list(
            limit=1,
            to='PHONE_NUMBER'
        )
        for record in messages:
            message = client.messages(record.sid).fetch()

            currentTime = datetime.now().timestamp() # current time in utc
            dts = str(message.date_created).split('+')[0]
            dtf = '%Y-%m-%d %H:%M:%S'
            dtSend = datetime.strptime(dts, dtf).replace(tzinfo = timezone.utc).timestamp() # message send time in utc

            if currentTime - dtSend <= 10:
                vcode2 = message.body
                try:
                    vcode1 = vcode2.split(':')[1]
                    vcode = int(vcode1.strip(' '))
                except:
                    vcode = None
            print(message.body)
    
        return vcode 



    def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

[Voyz]

Now, thanks for your answers! That's all very helpful to know 😊

Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".

This is due to Gateway's config.yml not having allowed IPs set up. Have a look at Troubleshooting to learn about this issue and how to solve it - let me know if you'd have any troubles doing it and I'll be happy to help you.

---

Digging into the code I can recognise a few areas that may cause the issues, leading to the unexpected behaviour we're observing in your logs:

1. Firstly, you run 20 attempts to acquire the code:

   for i in range(20):
       vCode = self.getNewMessage()

   That 20 seems like an arbitrary number. There also isn't any time delay between these attempts, so what seems to be happening is that all 20 happen at roughly the same time. The logs seem to confirm this, as 20 prints from line 116 appear within milliseconds of each other. I would suggest that you base that loop on a timeout, rather than on an arbitrary number, introducing a time delay between attempts, eg.:

    def twilioAuthToken(self):
        vCode = None
        start_time = time.time() 
        end_time = start_time + _TWILIO_TIMEOUT
        while time.time() < end_time:
            vCode = self.getNewMessage()
            if vCode != None:
                break
            time.sleep(1) # 1 second delay between requests
   
            _LOGGER.info(f"No code received, retrying for another {round(end_time - time.time())} seconds")
   
            
        if vCode == None:
            _LOGGER.error(f"Did not receive the authentication code after {round(time.time()-start_time)} seconds.")
        else:
            _LOGGER.info(f'Received code: {vCode}')
   
        return vCode

   I also increased verbosity of your logs to get more information. I suggest you set IBEAM_LOG_LEVEL env variable to 'DEBUG' in order to see even more information in your logs during this development stage.

2. I cannot comment much on the way you're interacting with Twilio as I haven't used them - I'd double check that you're doing everything correctly there too. However, you seem to only look up the messages from the previous 10 seconds if I'm not mistaken:
   python if currentTime - dtSend <= 10:
   I'd suggest to try to increase that buffer, as this may simply not be enough for a SMS message to deliver.

3. Furthermore, I'd suggest to alter the way you parse the messages received:

if currentTime - dtSend <= 10:
      vcode2 = message.body
      if 'Your requested authentication code' not in vcode2 or not vcode2[-6:].isdigit():
          _LOGGER.error(f'Unrecognised message body: {vcode2}')
      else:
         
          try:
              vcode1 = vcode2.split(':')[1]
              vcode = int(vcode1.strip(' '))
          except Exception as e:
              _LOGGER.exception(e)
              vcode = None

This way we can be more sure that the right messages are being parsed, and that we don't return invalid codes (eg. in your existing code a message that reads Error : Couldn't access code would return Couldn't access code as the 2FA code)

Lastly, I suggest you remove the following lines:

 def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

As they are misleading, and replace them with:

 def __str__(self):
        return f"TwilioHandler()"

---

I hope these changes help you ensure the right code is returned. We could certainly introduce some changes on IBeam to make things more robust in case wrong code is returned too many times - I'll try to look into this soon. Let me know how it goes! 👍

[vishwajeetio]

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials.
...

Hi @Voyz
I am sorry for the confusion. Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

[vishwajeetio]

Thank you so much @Voyz
I think it will definitely solve the issue as mow I am realizing that it might be the loop of random number 20.
I will put the code on run and will let you know what I get.
About the point
2. 10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue.
thank you so much for the time. It really helps a lot and will save a lot of time.
I will let you know how it goes.
thank you

[Voyz]

Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

Ah glad to hear that! Sorry I stepped over then! 😅

10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue.

Right, this may be a tricky one to figure out. It would be helpful to mark messages as 'read' and 'unread' on Twilio side - have a look if this is possible. A rigid 10 seconds limit may be a cause of bugs in the future.

I will let you know how it goes.

Great, looking forward to it! 😊

[Voyz]

@odssh I've just released a new version -rc2, ie.: voyz/ibeam:0.3.0-rc2. It contains additional security measures, checking that the 2FA code provided is valid (ie. contains 6 digits). It can be disabled by setting IBEAM_STRICT_TWO_FA_CODE to False, and it is based on the assumption that all 2FA codes are 6-digit. Let me know if this is not the case for you. Additionally, the new version does a better job at handling cases where there is no 2FA code provided by the handler.

Looking forward to hearing how the improvements worked out for you 👍

[vishwajeetio]

Thank you for the detailed update @Voyz
I think the verification codes are 6 digit all the time. I will definitely help me a lot.
I am still working on previous fixes. I will update you on the status once i get it all running.
Thank you 👍

[vishwajeetio]

Hi @Voyz
I was running the container for the last two days and it broke yesterday.
As you can see in this log file was authenticated for a long time but then all of a sudden it started authenticating every single minute and was able to authenticate but still kept doing it until it got locked out and then it still kept trying. I tried to search for the potential cause but unable to find it.
Please let me know if there's any way to fix that.
thank you

[Voyz]

hey @odssh I'm really sorry to hear about the second lock-out, that must be incredibly annoying and I'm going to try my best to help you avoid it in the future. Thanks for submitting the detailed log too. From that I can conclude the following:

• IBeam worked fine for you from 2021-03-03 22:43:16 till 2021-03-04 22:01:33
• During that time, it encountered some authentication drops and handled all of them well, except for one which returned 500 Internal Server Error. I haven't seen this error at all during development and I'd understand it as Gateway having some internal issues. Given that this happened only once and that authentication completed for another few hours after that I'd assume that it was not the cause for the later issues.
• At 2021-03-04 22:01:33 Gateway started losing authentication and IBeam couldn't complete authentication since then.
• From 2021-03-04 22:02:33 till 2021-03-04 23:22:39 - so for another 80 authentication attempts - IBeam was successfully retrieving and inputting the 2FA code along with the credentials. I conclude that by seeing the Authentication process succeeded message printed a lot in this time. At the same time, the error message Error displayed by the login webpage: failed appeared 9 times in that period. After the 9th time that error message appeared your account got locked out, although bear in mind this error also appeared earlier in the log once. We could assume 10 unsuccessful 2FA authentications cause a lock-out.
• There are two problems at hand here:

1. These 10 failed attempts would signify that the 2FA code returned is not correct. If I remember correctly, your 2FA auth logic tries to pull the most recent messages and then uses the most recent one. Could there be a flaw in the logic somewhere there? I'd suggest you double check for every of these 10 error 2FA attempts that IBeam received the correct 2FA code. Look at the timestamps of the messages received and double check if they correspond to the times IBeam would have requested these - and see if indeed the right codes were returned to IBeam by looking at the Received code: logs. If there's a discrepancy here we at least have one culprit. If you could share the messages log from that period of time it would be useful too.
2. Despite the majority of successful attempts, Gateway didn't seem to be authenticated, as from that point every maintenance recognised an active unauthenticated session.

• This second point is probably the core of the issue here. Why was the session not authenticated despite so many successful attempts? We can try to answer that by looking at the following:
  • Have you used these credentials to authenticate manually at that time? For instance to TWS, client portal or mobile app? If you could double check that, it would verify that lead.
  • Assuming you're running IBeam as a Docker image, is there any way your orchestration (or other system) would start two instances of IBeam? Could you find a way to confirm that only one IBeam instance was running at the time?
  • If both of the above leads don't provide any indications, we may try to read the Gateway logs. You may access these by looking in the clientportal.gw\logs directory on the container where IBeam is running. There you'll find two files gw.[DATE].log and gw.message.[DATE].log. If you can, either read through these logs from 2021-03-04 22:02:33 and try to understand what changed, or clean these files of any private information (some credentials are present there) and send them over in a private email to hello@voyzan.com for me to look through these.

These steps should move us forward towards understanding what is wrong. Once again, I'm really sorry you've run into these issues and I'm hoping we'll mange to get to the bottom of this for the benefit of all users of IBeam. Thanks for your patience in the meantime 👍

[vishwajeetio]

Hi @Voyz
thank you for the detailed explanation. I will start testing it now and see where it takes me.
I am still unable to get the port issue working with the docker. I tried the previous solution but it didn't work as it already had port 5000 allowed. I think I need to expose port 5000 from the docker file somewhere but all of my solutions failed.
thank you again.

[Voyz]

hey @odssh thanks for outlining all of that! I'd be interested to see and hear what scheduler for the authenticator would do. Keep us posted on your progress 👍

And thanks for the kind words, I'm happy to be able to support you in your trading 😊

[vishwajeetio]

thank you for all the help and I will definitely make sure to update you on that scheduler thing.
I have run into another issue. So while trying to add a proprietary certificate I got everything done but when I am running the container by mounting the inputs as a volume using the command below:

docker run -v /inputs:/srv/inputs --env-file paper.list -d -p 5000:5000 ib_with_ssl

I am getting the error on making requests. so when I ran the command

curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem

I am getting this error:

curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I followed the exact guide in the page tls
Please let me know if there's anything I have done wrong.
thank you

[Voyz]

Sorry to hear that @odssh!

Which OS are you calling cURL from?

What is the content of your /srv/inputs?

[vishwajeetio]

I am running the whole program on the ubuntu server and running the curl command from the same host on which the container is running.
When I checked the container and went to /srv/inputs directory inside the container it had none of the files that I mounted.
As you can see from the command I used I mounted the whole /inputs directory which contained three files:
cacert.jks cacert.pem conf.yaml
Please let me know if there's any other details you need
thank you

[Voyz]

That's surprising. Can you try mounting that volume to another docker image and seeing if this time the directory contains the files?

[vishwajeetio]

I have tried to mount the inputs directory multiple times in all the ways possible but still the same error and works only when I give -k tag at the end. Should I try manually importing the files inside the container and then restarting the container?
thank you

[Voyz]

So when you mount that directory to another container that uses a different image (not IBeam) you also don't see any files inside of the mounted directory? This would lead me to the following guesses:

• there's something broken with the setup (eg. Docker, host)
• the contents of /inputs are empty
• the command has a typo or is invalid: eg -v /inputs:/svr/inputs

The latter two are easier to verify, but if these aren't the cause of the issue then I'm sorry but I'm not entirely sure how to help you. You could try the setup on a different machine with a fresh installation of Docker and see if the issue exists there too?

[vishwajeetio]

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem
curl: (60) SSL: unable to obtain common name from peer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

That is the error message I am getting now even after copying the ca certificate manually to the destination inside the container.
It might be because of the certificate is invalid or something else.

Also, how can I use the default certificate that comes with a gateway with python instead of that custom one. I went with the custom as the default certificate wasn't working with the python
thank you

[Voyz]

I don't know how IBeam would behave if you copy the certificates manually - this is an unexpected behaviour and isn't really supported. How about the points I brought up in my previous reply?

The default certificate needs to be provided through the Inputs Directory, but since you can't mount the volume correctly this will not be possible until that gets sorted

[vishwajeetio]

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Hi, @Voyz
I got that error when using the beam's official docker image with the paper account. If possible can you please help me with where I am doing wrong?
thank you

[Voyz]

Okay @odssh, I'm trying to help you but I'm feeling there's some miscommunication between us. I need you to address the points that I bring up. It's useful that you provide me with the error messages, but there's a number of questions or observations I replied you with that you haven't really addressed. Please let's do things in the right order and so we can figure out how to fix your setup, okay?

You need to start by discovering what is the reason for not being able to mount the Docker volume correctly. Have a look at this reply I made #8 (comment) and confirm for me which seems to be the case - or update me if this is no longer a problem and you can now mount volumes correctly. Otherwise please provide some screenshots of outputs of ls and pwd commands from your /inputs directory to verify its contents. Thanks 👍

[vishwajeetio]

Hi @Voyz thank you so much for your support. Finally got it to work with the custom SSL certificate. I was providing the wrong path for the input folder and I just kept trying with the same wrong path.
Just another question I want to add, is how can I request the historical market data using that API? From the documentation, it seems like I need to make a get request to https://localhost:5000/v1/api/iserver/marketdata/history but when I do so I get the error message "Error 403 - Access Denied", Any idea on how can I fix that if you have done it before or if there's anything I am missing out?
thank you

[Voyz]

Hey, happy to hear you got that sorted! Issues with Access Denied are described in the Troubleshooting:

https://github.com/Voyz/ibeam/wiki/Troubleshooting#access-denied

This should help you out 👍

[vishwajeetio]

Actually, it's working fine with the endpoint https://localhost:5000/v1/api/one/user but failing with the historical data endpoint. I think this issue is probably because of how I have written my request and have nothing to do with iBeam. Please let me know if there's any other endpoint you have worked with as I think I am providing the conid wrong. Any idea on how can I generate it?
thank you

[Voyz]

You can check the conid's here:

https://contract.ibkr.info/v3.10/index.php?action=Stock%20Search&symbol=DDD&lang=en&ib_entity=&wlId=IB&showEntities=Y

Just type the symbol, click on 'Details' of the right stock and conid will be there.

Paste your full code for performing the request here, including the headers and parameters if you'd like me to see if I can spot something wrong with it.

[michaeljherrmann]

Hey @Voyz! I stumbled across your repo as I've been working on a similar project. I've come up with a solution to two factor authentication in case you're interested in another approach. 😄
https://github.com/michaeljherrmann/ib-gateway-service

[Voyz]

Hey @michaeljherrmann welcome to IBeam mate! 👋 Super cool of you to share your project, fantastic to see your approach!

The direct API requests you do are amazing, great job figuring this out! Just out of curiosity - do you know if your API interaction isn't against their TOC?

Going with Twilio is something that others have suggested in this thread, how do you find it working out for you?

[michaeljherrmann]

Hmm good question about the TOC, not something I've really dug into but you're probably right. I'll look further and perhaps I should rollback what I've publicly published.

In terms of Twilio, it works great! I've personally been using it consistently for about 3 months. It was easy to set up and has been pretty smooth overall, twilio had one outage for a few hours but that was it. The caveat with SMS two factor & IB is that they don't let you change the phone number from what the account was registered with. After talking with customer support, they insisted I set up IBKey auth via mobile app instead and that they would not be able to change the phone number, which is what led me down the IBKey route.

[Voyz]

Not sure if it is against TOC - I've had others suggesting doing exactly what you did in other threads. I'd just double check - maybe even ask them directly after having unpublished the code temporarily - if it's good then put it up mate, it's fantastic.

Glad to hear Twilio works great for you too! We actually expect such usage with IBeam v0.3.0 too, allowing to communicate with an external server on 2FA request. It does sound annoying they forced you to stick with just one phone number.

[kashyappatel7]

Hi Folks,
I wanted to show some Node.js code for implementing the EXTERNAL_REQUEST without needing a external database/datastore:

Basically, you setup a Twilio account and get a phone number and then setup a SMS webhook to https://your-site.com/sms/incoming. Then paste in the following code to your Node.js express web server:

let smsToken;

router.get('/sms/code', async (req, res) => {    
    try {
        if(req.query.token != 'ANY_STRING_VALUE_FOR_SECURITY') {
            console.log('Invalid token specified in query param.');
            res.send({});
            return;
        } else {

            console.log('request to /sms/code');

            smsToken = undefined;
            
            let waitCountMax = 120; // seconds
            let waitCount = 0;
            while(waitCount < waitCountMax) {
                waitCount++;
                if(!smsToken) {
                    await new Promise(done => setTimeout(done, 1000));
                } else {
                    break;
                }
            }

            if(smsToken) {
                console.log(`Responding with verificationCode: ${smsToken.verificationCode} obtained at ${smsToken.dateTime}`);
                res.send(smsToken.verificationCode);
                smsToken = undefined;
            } else {
                console.log('No SMS token received.');
                res.send({});
            }
        }        
    } catch(e) {
        console.error(e);
        res.send({});
    }    
});

router.post('/sms/incoming', async (req, res) => {    
    try {
        // "Your requested authentication code: 289026"
        let parts = req.body.Body.split(':');

        if(parts.length == 2) {
            let codePart = parts[1].trim();
            if(parts[0] == 'Your requested authentication code' && codePart.length == 6) {
                console.log(`Parsed Verification Code: ${codePart}`);
                
                
                smsToken = { 
                    verificationCode: codePart, 
                    dateTime: (new Date()).toISOString()
                };
                
            } else {
                console.log('Invalid verification code.');
                console.log(JSON.stringify(req.body));
            }
        } else {
            console.log('Non-verification code SMS detected.');
            console.log(JSON.stringify(req.body));
        }
    } catch(e) {
        console.error(e);
    } finally {
        res.writeHead(200, { 'Content-Type': 'text/xml' });
        res.end('<Response />');
    }    
});

Next, start your docker container using something like:
docker run --env IBEAM_ACCOUNT=YOUR_PAPER_OR_LIVE_ACCOUNT_ID --env IBEAM_PASSWORD=YOUR_PWD --env IBEAM_TWO_FA_HANDLER=EXTERNAL_REQUEST --env IBEAM_EXTERNAL_REQUEST_URL=https://your-site.com/sms/code --env IBEAM_EXTERNAL_REQUEST_PARAMS={"token":"ANY_STRING_VALUE_FOR_SECURITY"} --env IBEAM_LOG_LEVEL=DEBUG --env IBEAM_PAGE_LOAD_TIMEOUT=500 -p 5000:5000 voyz/ibeam

I hope this helps someone... and also, great work Voyz !!!, you have saved a lot of people a lot of time =)

[Voyz]

That's very useful @kashyappatel7 many thanks for sharing this! I'll add a link to this to the 2FA Wiki. And thanks for the kind words, I'm glad to help you find IBeam useful 😊

[psyfb2]

Hi, I tried to use Twilio to receive 2FA codes from IBKR but the 2FA messages don't even show up on Twilio REST API or their website. Apparently this is because virtual numbers cannot send SMS to other virtual numbers (IBKR sends 2FA from a virtual number). Is the Twilio solution still working for anyone?

[Voyz]

@psyfb2 Sorry, I never used Twilio, couldn't comment on that. Google Messages solution works.