Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

don't crawl files that haven't been modified since the last crawl #17

Open
wants to merge 1 commit into
base: master
from

Conversation

@rpendleton
Copy link

commented Oct 6, 2019

According to RFC 7232 §4.1, when a page is returned with a 304 Not Modified header, it is recommended that headers such as Content-Type should not be sent the the client again. When verbose logging in osg is enabled, this would manifest as not crawling a page because its content type was an empty string.

Instead of doing that, we should just log that the page hasn't been modified and avoid crawling it completely. This is okay since a 304 response isn't permitted to have a response body anyway, so there's no need to crawl it.

According to RFC 7232 §4.1, when a page is returned with a 304 Not
Modified header, it is recommended that headers such as Content-Type
should not be sent the the client again. When verbose logging in osg is
enabled, this would manifest as not crawling a page because its content
type was an empty string.

Instead of doing that, we should just log that the page hasn't been
modified and avoid crawling it completely. This is okay since a 304
response isn't permitted to have a response body anyway, so there's no
need to crawl it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.