Support for HTTP header "If-Modified-Since"

[jetnet]

Hello Pascal,

it seems, the crawler always download documents from web servers to check, if they have been changed, e.g.:

www.zoo-dresden.de: 2019-08-31 05:13:52 INFO -          DOCUMENT_FETCHED: https://www.zoo-dresden.de/daten/1/slider/Roter-Panda.jpg
www.zoo-dresden.de: 2019-08-31 05:13:52 INFO -       REJECTED_UNMODIFIED: https://www.zoo-dresden.de/daten/1/slider/Roter-Panda.jpg

even though the webserver supports the If-Modified-Since header (run the curl twice):

curl -s -v -o Roter-Panda.jpg -z Roter-Panda.jpg https://www.zoo-dresden.de/daten/1/slider/Roter-Panda.jpg

> GET /daten/1/slider/Roter-Panda.jpg HTTP/1.1
> Host: www.zoo-dresden.de
> User-Agent: curl/7.59.0
> Accept: */*
> If-Modified-Since: Tue, 03 Sep 2019 08:40:31 GMT
>
< HTTP/1.1 304 Not Modified

I quickly searched the git repository for If-Modified-Since, but could not find, if the header is used.
Could you please clarify that? And if the header is not used, than could you please add this for future releases in order to minimize download traffic?
Thanks a lot!

[jetnet]

I guess, in the meantime, the Metadata Fetcher can be used to check, if remote documents have been changed:

I'll try it!

[essiembre]

You are correct that adding the HTTP Metadata fetcher will save you from downloading the files as long as the Last-Modified is provided in the HTTP Response (but costs an extra call).

If-Modified-Since is indeed not supported right now. I am all for reducing traffic where we can.
I am marking this as a feature request.