Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser or Postman #25

Closed
ghisbo opened this issue Feb 20, 2021 · 1 comment

Comments

@ghisbo
Copy link

ghisbo commented Feb 20, 2021

While in FireFox this site ( = example, there are multiple sites ) opens without problems, Abot is having a problem with it.
The Error message refers to an "invalid or unrecognized response"
After some more digging:
When using Postman, with the exact same request, the site returns and responds normally
It look as if the HttpClient.sendAsync in the PageRequester seems to cause the problem

thanks in advance

Ghislain

Output of Abot2Demo

Did not crawl the links on page https://aanhangwagenspattyn.be/ due to Page has no content
ERR: Crawl of page failed crawledPage.HttpRequestException.InnerException = System.IO.IOException: The server returned an invalid or unrecognized response.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
Page had no content https://aanhangwagenspattyn.be/

@sjdirect
Copy link
Owner

sjdirect commented Feb 22, 2021

This is better homed as an Abot Bug. I'm able to replicate the issue. Looks like a known issue in the HttpClient as discussed here and elsewhere on the net. Not sure that I can do anything about this specifically as Abot relies on the .net HttpClient for its http calls. 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants