Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qa/replay crawl loading improvements #526

Merged
merged 5 commits into from
Apr 4, 2024
Merged

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Apr 3, 2024

  • use frame.load() to load RWP frame directly instead of waiting for navigation messages
  • retry loading RWP if replay frame is missing
  • support --postLoadDelay in replay crawl
  • support --include / --exclude options in replay crawler, allow excluding and including pages to QA via regex
  • improve --qaDebugImageDiff debug image saving, save images to same dir, using ${counter}-${workerid}-${pageid}-{crawl,replay,vdiff}.png for better sorting
  • when running QA crawl, check and use QA_ARGS instead of CRAWL_ARGS if provided
  • ensure info.warc.gz is closed in closeFiles()

misc:

  • fix typo in --postLoadDelay check!
  • enable 'startEarly' mode for behaviors (autofetch, autoplay)

- use frame.load() to load RWP frame directly instead of waiting for navigation messages
- retry loading RWP if replay frame is missing
- support --postLoadDelay in replay crawler
- support --include / --exclude options in replay crawler, allow excluding and including pages to QA via regex
- improve --qaDebugImageDiff debug image saving, save images to same dir, using ${counter}-${workerid}-${pageid}-{crawl,replay,vdiff}.png for
better sorting
- when running QA crawl, check and use QA_ARGS instead of CRAWL_ARGS if provided

misc:
- fix typo in --postLoadDelay check!
- enable 'startEarly' mode for behaviors (autofetch, autoplay)
@ikreymer ikreymer requested a review from tw4l April 3, 2024 07:33
Copy link
Contributor

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments/suggestions, overall seems to be working well

src/replaycrawler.ts Outdated Show resolved Hide resolved
src/replaycrawler.ts Outdated Show resolved Hide resolved
src/replaycrawler.ts Outdated Show resolved Hide resolved
ikreymer and others added 4 commits April 4, 2024 12:04
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
@ikreymer ikreymer merged commit c247189 into main Apr 4, 2024
4 checks passed
@ikreymer ikreymer deleted the replay-load-improvements branch April 4, 2024 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants