fix captcha solving fixes #1036

[tadasgedgaudas]

Fix for #1036

• starts browser with devtools
• switches to the correct window because it starts with devtools focused and we are interacting with the devtools page instead of the actual page we want
• removing duplicate driver.get, not necessary when doing simple GET requests, only required when doing POST for CORS

[SmartArray]

tACK

[iliyaZelenko]

I think it’s better to merge this as soon as possible, otherwise at the moment we cannot bypass the CloudFlare.

[ilike2burnthing]

Ideally waiting to hear back from @howwwdi before merging.

[juanfrilla]

Okay, so let's see what @howwwdi thinks about this amazing solution

[howwwdi]

Ideally waiting to hear back from @howwwdi before merging.

I apologise for the wait, can't really make a comit rn. So the only thing i've had noticed, is adding driver = get_correct_window(driver) right after adding cookies to the session in evil_logic func

[tadasgedgaudas]

Ideally waiting to hear back from @howwwdi before merging.

I apologise for the wait, can't really make a comit rn. So the only thing i've had noticed, is adding driver = get_correct_window(driver) right after adding cookies to the session in evil_logic func

good point, added 7416962

[ilike2burnthing]

Is access_page(driver, req.url) not needed for cookies POST? Haven't tested yet, will do so later and merge if it's all good.

[tadasgedgaudas]

Is access_page(driver, req.url) not needed for cookies POST? Haven't tested yet, will do so later and merge if it's all good.

the page has been accessed before, so not needed

[ilike2burnthing]

import requests

url = "http://localhost:8191/v1"
headers = {"Content-Type": "application/json"}
data = {
    "cmd": "request.post",
    "url": "https://leporno.info/tracker.php",
    "maxTimeout": 60000,
    "postData": "nm=720pxyz&submit=Search",
    "proxy": {"url": "http://192.168.0.2:8080"}
}
response = requests.post(url, headers=headers, json=data)
print(response.text)

Well I thought I had fixed POST with proxy, but apparently not. The above sees the devtools panel automatically close and the postData isn't sent. Works fine without the proxy, or with proxy and GET. Current master works with proxy and POST until the open devtools argument is added.

[BakasuraRCE]

@ilike2burnthing I tested this PR with under container 1337x.to #1073

I confirm that it works!

[ilike2burnthing]

The PR won't be merged until #1065 (comment) is resolved.

[tadasgedgaudas]

import requests

url = "http://localhost:8191/v1"
headers = {"Content-Type": "application/json"}
data = {
    "cmd": "request.post",
    "url": "https://leporno.info/tracker.php",
    "maxTimeout": 60000,
    "postData": "nm=720pxyz&submit=Search",
    "proxy": {"url": "http://192.168.0.2:8080"}
}
response = requests.post(url, headers=headers, json=data)
print(response.text)

Well I thought I had fixed POST with proxy, but apparently not. The above sees the devtools panel automatically close and the postData isn't sent. Works fine without the proxy, or with proxy and GET. Current master works with proxy and POST until the open devtools argument is added.

Perhaps there is a specific issue with proxy without authorization? Because it works just fine for me, I was testing like this:

{
    "cmd": "request.post",
    "url": "https://leporno.info/tracker.php",
    "maxTimeout": 60000,
    "postData": "nm=720pxyz&submit=Search",
		"cookies": [
			{
				"domain": "leporno.info",
				"httpOnly": false,
				"name": "__PPU___PPU_SESSION_URL",
				"path": "/",
				"sameSite": "Lax",
				"secure": false,
				"value": "%2Ftracker.php"
			},
			{
				"domain": "leporno.info",
				"expiry": 1708338058,
				"httpOnly": false,
				"name": "__PPU_tuid",
				"path": "/",
				"sameSite": "Lax",
				"secure": false,
				"value": "7337240885443409304"
			},
			{
				"domain": "leporno.info",
				"expiry": 1742804519,
				"httpOnly": false,
				"name": "__PPU_puid",
				"path": "/",
				"sameSite": "Lax",
				"secure": false,
				"value": "7337240889736249891"
			},
			{
				"domain": ".leporno.info",
				"expiry": 1739870514,
				"httpOnly": true,
				"name": "cf_clearance",
				"path": "/",
				"sameSite": "None",
				"secure": true,
				"value": "AWPUYZeid0PiymlpS_oGh27jfWhksyzlN9n7Z3PbO1k-1708334507-1.0-AdGIQAWOxCdYMk+Hd2hW02c3jLEWNuFr+Qw+lAiD/C2VtLzjBA222X7NzuRig+IqJLudtPekfrA4hSsTJpfAbc0="
			}
		],
    "proxy": {"url": "http://REDACTED",
						 "username": "REDACTED",
						 "password": "REDACTED"}
}

[ilike2burnthing]

Perhaps there is a specific issue with proxy without authorization? Because it works just fine for me

I'll take a look to make sure. It had been previously working for me, and I have been using dummy credentials at that point.

[ilike2burnthing]

After a bit of testing, credentials aren't the cause. If you remove the cookie block, in particular cf_clearance (it's just bypassing the problematic CF challenge), you should see my issue. The quick check is that a search for 720pxyz should return no results.

Ideally I'd love to get this PR published ASAP without this issue, as the number of affected sites has jumped. However, if you don't see an easy fix, please let me know; I can open a new issue about this and we can work from there with a new v3.3.15 build.

[tadasgedgaudas]

After a bit of testing, credentials aren't the cause. If you remove the cookie block, in particular cf_clearance (it's just bypassing the problematic CF challenge), you should see my issue. The quick check is that a search for 720pxyz should return no results.

Ideally I'd love to get this PR published ASAP without this issue, as the number of affected sites has jumped. However, if you don't see an easy fix, please let me know; I can open a new issue about this and we can work from there with a new v3.3.15 build.

I'm checking with the removed cookie and only using one cookie this time, completely unrelated and the captcha gets solved :/ Cannot reproduce the issue, so I guess your proposed solution to continue fixing this with another PR would work better...

[howwwdi]

After a bit of testing, credentials aren't the cause. If you remove the cookie block, in particular cf_clearance (it's just bypassing the problematic CF challenge), you should see my issue. The quick check is that a search for 720pxyz should return no results.

Is there any point using exactly POST method, because this website protected by deafault with IUAM. U can just re-use cf_clearance cookie in next request just by using exact same tls+ua+proxy

[ilike2burnthing]

I'm checking with the removed cookie and only using one cookie this time, completely unrelated and the captcha gets solved :/ Cannot reproduce the issue, so I guess your proposed solution to continue fixing this with another PR would work better...

If you use no cookies at all, just copy and pasting my example from #1065 (comment), only changing the proxy details, you get a page showing a search for 720pxyz and returning no results?

[ilike2burnthing]

U can just re-use cf_clearance cookie in next request just by using exact same tls+ua+proxy

That would make FlareSolverr redundant.

[tadasgedgaudas]

I'm checking with the removed cookie and only using one cookie this time, completely unrelated and the captcha gets solved :/ Cannot reproduce the issue, so I guess your proposed solution to continue fixing this with another PR would work better...

If you use no cookies at all, just copy and pasting my example from #1065 (comment), only changing the proxy details, you get a page showing a search for 720pxyz and returning no results?

Yeah, I see it now and I wonder how it worked previously 🤔

It could work if we would solve the captcha first, then make the post request

But now what happens:

• captcha is present
• making post request while captcha is present
• redirects to /tracker.php
• captcha is solved and we get /tracker.php content without the submitted form

[ilike2burnthing]

I wonder how it worked previously 🤔

Glad I'm not the only one who had it weirdly work one time but not another 😛

Fix coming shortly, just doing the last round of tests.

[ilike2burnthing]

Will merge this now, but unfortunately it won't close #1036. In the last few hours this PR (with or without my last commit) stopped being able to bypass the challenges on fanfiction.net and airbusan.com, but works on everything else I tested 🤷 (feel free to double-check, to make sure it's not just me)