Add handling of cookies to HTTP protocol

[jnioche]

The cookies info could be stored in the metadata (and inherited from the parent URL) and used by the Protocol implementation.

[foromer4]

I would like to implement this. @jnioche, could you please add some info on what classes should be changed to save and read these stored cookies?
from what I can tell. JSoupParserBolt might be the place to detect cookies.
However, I'm not sure how to do it only once for each website?
Does Storm crawler keep info to connect each url to a specific parent url?
ANd does it store infromation for the parent url?
thanks

[jnioche]

Here is how I see it. You should only need to modify things at the protocol level. Basically, the cookies are stored as text in the metadata of the parent URL, transferred to the outlinks which get persisted to the status in the usual ways, see https://github.com/DigitalPebble/storm-crawler/wiki/MetadataTransfer.

When the tuple for an outlink is sent down the topology, we'll need to convert the info from the metadata into cookies in the getProtocolOutput() method so that the low level protocol stuff sends it.

from what I can tell. JSoupParserBolt might be the place to detect cookies.

Nope. Cookies are nothing more than http headers with an attitude.

However, I'm not sure how to do it only once for each website?

We don't. We could have different cookies information for different URLs belonging to the same host or domain.

Does Storm crawler keep info to connect each url to a specific parent url?

Part of the optional path info in the metadatam see Wiki page about MetadataTransfer

ANd does it store information for the parent url?

Like any other URL, nothing more.

Hope this helps

[foromer4]

@jnioche , So you want the crawler to visit a ceratin page and collect from the response the cookies, and then use the same cookies for discovered links? and if it find other cookies in the discovered links add them also to their "child links" and so on?

Might it be beneficial to set the desired cookies once per site , or per regex in the config, instead of passing them from parent to child?

[jnioche]

if it find other cookies in the discovered links add them also to their "child links" and so on?

they would replace the existing ones instead of adding to them I think

Might it be beneficial to set the desired cookies once per site ,

there is no global repository for sites info in SC

or per regex in the config, instead of passing them from parent to child?

you could set them as seed metadata instead of keeping them in the config and have an option in the protocol implementation to NOT replace an existing value. This would be more flexible I think as you could modify it on the fly instead of relaunching the topology

which problem are you trying to solve or to put it differently, any reason you think my suggestion is a bad idea?

Thanks

[foromer4]

Well I definitely see the value of passing cookie from url to discovered link - and will start working on this as you suggested. In addition, I can think of some cases where you would want to set up specific cookies in advance (for example session cookies ,as login is hard to do sometimes in a crawler)

[foromer4]

@jnioche , I started looking at the solution you suggested and ran into an issue.
It seems that in order to get the cookies I need to add a cookie store to the httpClinet.

Now the problem as I see it is that in order to get the cookies, I need to make the HttpClient a member of HttpProtocol, and since HttpProtocol itself is being reused over and over again (stored in a cache , provided by ProtocolFactory) it will be a problem.

Possible solution:
Add flag for "useCookies" , when this flag in on, ProtocolFactory will return a new instance per request.
Of course it might incur a performance penalty (hence the flag).
Please let me know if this seems right to you.

[jnioche]

Hi @foromer4

You don't need a cookie store. The advantage of passing it as metadata is that we don't expect the protocol instance to persist anything or have any prerequisites. The cookies (if any) can simply be added to the header like this:

    List<Cookie> cookies = CookieConverter.getCookies(datum, url);

    for (Cookie c : cookies)
      httpget.setHeader("Cookie", c.getName() + "=" + c.getValue());

as for the CookieConverter.getCookies method, it simply populates a list of BasicClientCookie instances from the metadata key values.

There should be nothing special to do on receiving the cookies, as I said, they are just normal HTTP response metadata.

[foromer4]

Hi @jnioche, sorry for being difficult, I just want to make sure I understand.
If you do not want to persist the cookies, only to use the cookies in the http headers of the response as the cookies to the output link, why can't we use the exiting MetaData field of _response_headers as it is already saved in HttpProtocol.handleResponse?

We can than use MetadataTransfer.getMetaForOutlink as you suggested to set the cookie to the output link.

[jnioche]

Hi @jnioche, sorry for being difficult, I just want to make sure I understand.

don't worry you are not being difficult at all.

If you do not want to persist the cookies, only to use the cookies in the http headers of the response as the cookies to the output link, why can't we use the exiting MetaData field of _response_headers as it is already saved in HttpProtocol.handleResponse?

that field is optional and contains ALL the key/values returned by the server. It can get quite large and would not typically passed on to the outlinks. Also it you want to inject the cookie into on a per seed basis then having a separate key to do it would be cleaner.

[foromer4]

Thanks for clarifying, i will be back from vacation next week, and will do it

…

On Mar 13, 2017 4:52 PM, "Julien Nioche" ***@***.***> wrote: Hi @jnioche <https://github.com/jnioche>, sorry for being difficult, I just want to make sure I understand. don't worry you are not being difficult at all. If you do not want to persist the cookies, only to use the cookies in the http headers of the response as the cookies to the output link, why can't we use the exiting MetaData field of _response_headers as it is already saved in HttpProtocol.handleResponse? that field is optional and contains ALL the key/values returned by the server. It can get quite large and would not typically passed on to the outlinks. Also it you want to inject the cookie into on a per seed basis then having a separate key to do it would be cleaner. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKA76MTMQb1yVuX46R8vIiD8s1ohRM4Oks5rlVgagaJpZM4DEPfS> .

[jnioche]

Here is some code which you could use as a starting point [https://gist.github.com/jnioche/6141308519694b5c57d4fbd45d5990ac]. It was written for Nutch but could easily be adapted to SC.