Cannot signin : bot detected

[lasconic]

See https://community.finary.com/t/api-finasync-extension-open-source/10104/20

After a call to signin we get

{
    "errors": [
        {
            "message": "Bot traffic was detected. Please try again or contact support.",
            "code": "bot_detected"
        }
    ]
}

[lasconic]

My first thoughts:

• Could be as simple as a referer or user-agent check ? (I doubt but it's easy to try...)
• Could because there is no delay between the call with username and password and the call with the 2FA code... We could add a delay but it could fail because of the validity period of the 2FA code. Or we change our API, and we ask for the code interactively...

Any other ideas ?

[lasconic]

Actually, the error occurs right away on the first code, we don't even get the chance to submit the 2FA code.

[lasconic]

The following works, not sure why it doesn't work with requests yet.

curl 'https://clerk.finary.com/v1/client/sign_ins' \
  --http1.1 \
  -H 'User-Agent: finary_uapi 0.2.0' \
  -H 'Connection: keep-alive' \
  -H 'Origin: https://app.finary.com' \
  -H 'Referer: https://app.finary.com/' \
  -H 'Accept-Encoding: identity' \
  --data-raw 'password=PASSWORD&identifier=EMAIL' \
  --verbose

Translating this request exactly to requests causes the bot_detected error.
The same call using httpx works. There is something special about requests and clerk or finary detects it.

[lasconic]

@nmathey @MadeInPierre
A possible solution is to switch to httpx. There must be a subtle difference between requests and curl or httpx for requests to be detected as a bot with the configuration by default. I can't find this difference... Using httpx, most of the things seems to work for now... Is it a problem for your respective projects to use httpx instead of requests ?

[lasconic]

Note that even with httpx running from github actions doesn't work... clerk detects a bot.
Locally all the tests but 2 are passing.

[lasconic]

See https://github.com/lasconic/finary_uapi/tree/switch_httpx

[nmathey]

Moving away from requests shouldn't be a problem for me.
So from GitHub it still doesn't but you were able to get it working from your wrapper?

[lasconic]

Yes, the tests don't work from Github Actions server but they run fine from my home computer. You can try to signin with the main branch (and should get the bot detected error) and with this branch and you should get success.

[nmathey]

Got same error

INFO:httpx:HTTP Request: POST https://clerk.finary.com/v1/client/sign_ins "HTTP/1.1 403 Forbidden"
{
    "errors": [
        {
            "message": "Bot traffic was detected. Please try again or contact support.",
            "code": "bot_detected"
        }
    ]
}

Did your branch includes all headers additions you tested with curl?

--http1.1 \
 -H 'User-Agent: finary_uapi 0.2.0' \
 -H 'Connection: keep-alive' \
 -H 'Origin: https://app.finary.com' \
 -H 'Referer: https://app.finary.com/' \
 -H 'Accept-Encoding: identity' \

[lasconic]

See https://community.finary.com/t/api-finasync-extension-open-source/10104/24 for the current official status (by Finary CTO).

The behavior described in this bug is "expected". For me, the HTTPX branch works. I tried several headers but of course, I made sure to use exactly the same one while testing. I had the same header in the main branch (using requests), with curl and with HTTPX (including forcing HTTP 1.1 when possible). Clerk does smart things for their bot detection for sure.

For now, I would like to get more data. Does the main branch work for you ? Does the HTTPX branch work ?

[nmathey]

Sorry I'm bit confuse, and might be a dummy question, but can you confirm headers you successfully tested are hardcoded in your Httpx branch?
Or should I configured them myself somewhere?
Because when I tested your branch, I just did as is.

[lasconic]

@nmathey they aren't hardcoded but the HTTPX branch works for my laptop as it, without configuring anything.

[JulesGuesnon]

Hey!
I've been developing my own client with reqwest to automatically update balances of manual accounts, and I faced the same problem while developing:

• I tried to put the same headers than the browser but I was detected as a bot
• I tried CURL and it works

I was using this repo as a reference for implementation (thanks for the awesome work btw), and I naively tried to set my user-agent to: finary_uapi 0.2.0, and it worked. FYI, I'm also setting the following headers:

• Origin: https://app.finary.com
• Referer: https://app.finary.com
• Connection: keep-alive
• Accept: */*

I haven't tried anything else since then, but my guess is that Clerk detects http clients that try to behave like browsers as they're probably bots most of the times, and allows clients that are transparent and saying that they're an API.

So basically, I don't think it's an issue related to the http client, but it's probably linked to headers. Hope those information can help you!