IAFD: FlareSolverrException: The cookies provided by FlareSolverr are not valid

[MisterMcDuck]

Hello,

I'm currently running FlareSolverr and FlaresolverrSharp v2.0 released a few days ago. I am able to successfully run the Sample program (FlaresolverrSharp.Sample), but when I try to use the program with another cloudflare-protected URL I receive
FlareSolverrException: The cookies provided by FlareSolverr are not valid

On Flaresolverr's side, it appears that it cannot detect cloudflare:

2021-10-23T16:17:38-04:00 INFO REQ-2 Incoming request => POST /v1 body: {"maxTimeout":60000,"cmd":"request.get","url":"http://www.iafd.com/results.asp?searchtype=comprehensive&searchstring=Stoya"}
2021-10-23T16:17:44-04:00 INFO REQ-2 Cloudflare not detected
2021-10-23T16:17:46-04:00 INFO REQ-2 Response in 8.416 s

The response message from Flaresolverr does have the correct HTML, so it appears Flaresolverr isn't running into the same problem the .NET webclient is.

On FlaresolverrSharp's side it does detect a cloudflare response and forwards the request to Flaresolverr. It receives the correct response, and even moves what appear to be cloudflare cookies across to the subsequent request (e.g. the __cf_bm cookie), but the .NET web request still returns the typical 403 Forbidden cloudflare response.

If you want to reproduce the issue, try changing the URL in the sample program to:

public static string ProtectedUrl = "http://www.iafd.com/results.asp?searchtype=comprehensive&searchstring=Stoya";

I could use Flaresolverr as a proxy for every request, but it would be nice if the captured cookies worked in a normal request.

Thanks for any assistance you can give

[ngosang]

I can't reproduce the issue. It seems the site disabled Cloudflare protection, at least form me.

[MisterMcDuck]

Strange. I receive the following when I try it:

I also have tried it on my Linux machine, and get the same general error with .NET core:

Debug  net5.0  master  ✎  $  dotnet FlareSolverrSharp.Sample.dll
Unhandled exception. System.AggregateException: One or more errors occurred. (The cookies provided by FlareSolverr are not valid)
 ---> FlareSolverrSharp.Exceptions.FlareSolverrException: The cookies provided by FlareSolverr are not valid
   at FlareSolverrSharp.ClearanceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/chris/Projects/FlareSolverrSharp/src/FlareSolverrSharp/ClearanceHandler.cs:line 111
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at FlareSolverrSharp.Sample.ClearanceHandlerSample.SampleGet() in /home/chris/Projects/FlareSolverrSharp/sample/FlareSolverrSharp.Sample/ClearanceHandlerSample.cs:line 25
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at FlareSolverrSharp.Sample.Program.Main() in /home/user/Projects/FlareSolverrSharp/sample/FlareSolverrSharp.Sample/Program.cs:line 8
fish: “dotnet FlareSolverrSharp.Sample…” terminated by signal SIGABRT (Abort)

I wonder if my IP address has any effect. I guess I could spin up an EC2 instance to see if the behavior is any different.

[ngosang]

As I said, I can't reproduce the issue but it could be a false positive. If you take a look that the IsChanllengeDetected method I'm just checking the "server" header and the HTTP response code. There are websites hosted in Cloudflare CDN that can be detected by that method but they doesn't have challenges. Maybe you could try to open the URL in your web browser with DevTools (F12) => Network enabled to see what is happening.

[MisterMcDuck]

I was able to solve the issue by forcing my HttpClient web requests to use 2.0:

client.DefaultRequestVersion = Version.Parse("2.0");

Thanks for the help along the way