Override request_fingerprint with meta field?

[jcushman]

Would it make sense to add something to the top of scrapy.utils.request.request_fingerprint like:

if 'fingerprint' in request.meta:
    return request.meta['fingerprint']

This is useful, for example, if you consider two pages to be identical if they share the same productID query parameter:

def parse(self, response):
    ...
    request = Request(url)
    m = re.search(r'productID=([^&]+)', url)
    if m:
        request.meta['fingerprint'] = m.group(1)

I'm currently handling this with a custom duplicate filter, but it seems like it would be broadly useful.

[pablohoffman]

This sounds good to me. I'm happy to review and merge a pull request that implements it.

[dangra]

+1 too

along this changes, what do you think about dropping _request_fingerprint_cache
and store the computed fingerprint in the meta as suggested in this ticket

[dangra]

just figured out a problem with storing fingerprint in request.meta :-(

request.meta is propagated by request.replace() which is commonly used to redirect, retry or simply propagate itermediate results stored in meta to other urls

[pablohoffman]

Yes, just using request.meta['fingerprint'] wouldn't work, but I still like the original proposal of supporting as an additional source for the fingerprint.

@jcushman mind submitting a pull request? ;)

[dangra]

dusty.