Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Record and use HTTP cache information to skip unnecessary crawling and processing #856

Merged
merged 19 commits into from
Feb 7, 2022
Merged

Record and use HTTP cache information to skip unnecessary crawling and processing #856

merged 19 commits into from
Feb 7, 2022

Commits on Feb 4, 2022

  1. Record and use HTTP cache information to skip unnecessary crawling an… 

    …d processing

close #850
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    0d54f81 View commit details
    Browse the repository at this point in the history

  2. Delegate proper transparent caching to fetch-filecache-for-crawling 

    see tidoust/fetch-filecache-for-crawling#6
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    092602a View commit details
    Browse the repository at this point in the history

  3. Remove bogus logCounter usage 

    as pointed in code review https://github.com/w3c/reffy/pull/856/files#r799292847
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    c0bc248 View commit details
    Browse the repository at this point in the history

  4. Simplify code 

    from code review https://github.com/w3c/reffy/pull/856/files#r799290909
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    a7bf290 View commit details
    Browse the repository at this point in the history

  5. Do not use exception for signal processing skip 

    as suggested in code review https://github.com/w3c/reffy/pull/856/files#r799297474
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    dafced8 View commit details
    Browse the repository at this point in the history

  6. Improve caching semantics 

    Only use conditional request headers on the relevant URL
Use Last-Modified in prefere to Etag as ETag tends to provide less reliable caching experience
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    b4585a3 View commit details
    Browse the repository at this point in the history

  7. Remove dead code

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    5b70337 View commit details
    Browse the repository at this point in the history

  8. Format code

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    861f221 View commit details
    Browse the repository at this point in the history

  9. Skip processing also if response is not 304 but cache info have not c… 

    …hanged
    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    521dd86 View commit details
    Browse the repository at this point in the history

  10. Unbreak http error handling

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    077a83c View commit details
    Browse the repository at this point in the history

  11. Avoid adding bogus cache crawl records

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    2933268 View commit details
    Browse the repository at this point in the history

  12. Add test for cache-fallback

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    8090c89 View commit details
    Browse the repository at this point in the history

  13. Fix non-http error reporting

    @dontcallmedom
    dontcallmedom committed Feb 4, 2022
    Configuration menu
    Copy the full SHA
    0025f88 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2022

  1. Bump version of fetch-filecache-for-crawling

    @tidoust
    tidoust committed Feb 5, 2022
    Configuration menu
    Copy the full SHA
    abbfb2c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into http-cache

    @tidoust
    tidoust committed Feb 5, 2022
    Configuration menu
    Copy the full SHA
    4e7e182 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2022

  1. Report not modified status as a status rather than an error 

    as suggested in #856 (review)

Co-authored-by: François Daoust <fd@tidoust.net>
    @dontcallmedom @tidoust
    dontcallmedom and tidoust committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    a00e310 View commit details
    Browse the repository at this point in the history

  2. Use fallback data only if crawler version match 

    per #856 (review)
    @dontcallmedom
    dontcallmedom committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    9097c0d View commit details
    Browse the repository at this point in the history

  3. Use fallback for errors when available, independently of crawler compat 

    per #856 (comment)
    @dontcallmedom
    dontcallmedom committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    83cf692 View commit details
    Browse the repository at this point in the history

  4. Adapt test to annotate crawl with current reffy version

    @dontcallmedom
    dontcallmedom committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    0160c1b View commit details
    Browse the repository at this point in the history