Unable to crawl https site

[ArsenShnurkov]

I am trying to run Abot.Demo on
https://focus.kontur.ru

I added site certificate with
yes | certmgr -ssl -v https://focus.kontur.ru

Abot.Demo program gives me "Max. redirections exceeded." exception.
and the following line in log:
[2014-11-15 08:24:41,678] [1] [INFO ] - Page crawl complete, Status:[302] Url:[https://focus.kontur.ru/] Parent:[https://focus.kontur.ru/] - [AbotLogger]

I use mono 3.10.1 on linux

What is the problem, and how to overcome it?

[sjdirect]

The siteis replyingwith an http 302 redirect on every request. I would bet
that site requires a cookie to be sent with each request. If the cookie is
not sent it likely redirects over and over until it reaches the max number
of redirects which is 7 by default.

1. Extend the Abot.Core.PageRequester class and override the MakeRequest()
   method

2. Copy the overridden code as the starting point for your new impl of
   MakeRequest()

3. After you receive the first response from that site, you need to save
   the cookie that you get back and resend it with all future requests in the
   overridden MakeRequest method.

Good luck
Steven

I am trying to run Abot.Demo on
https://focus.kontur.ru

I added site certificate with
yes | certmgr -ssl -v https://focus.kontur.ru

Abot.Demo program gives me "Max. redirections exceeded." exception.
and the following line in log:
[2014-11-15 08:24:41,678] [1] [INFO ] - Page crawl complete, Status:[302]
Url:[https://focus.kontur.ru/] Parent:[https://focus.kontur.ru/] -
[AbotLogger]

What is the problem, and how to overcome it?

—
Reply to this email directly or view it on GitHub
#41.

[ArsenShnurkov]

Yes, works this way.

See https://github.com/sjdirect/abot/pull/42/files