Allow configuration of deduplication policy

[wvengen]

The simplest solution for #331 (whole or partial solution, depending how you look at it), allowing configuration of pywb's dedup_policy.

• Test this actually works.

[ikreymer]

Thanks for adding this! Don't know if this is the best solution to the verification check issue per se, but not opposed to having this option in general, especially if it helps!

[ikreymer]

One issue however, is that we are planning to move away form pywb for recording (see: https://github.com/webrecorder/browsertrix-crawler/tree/recorder-work) so it may be more difficult to support this exact functionality, though probably should have some sort of dedup options..

[wvengen]

Thank you for including this change! 🎉

For reference: maybe it's good to point to the issue from here (#331 (comment)) and note that I took a safe set of accepted option values, which work with redis duplicate detection.

Good to know that a new recording path is being taken, thanks for mentioning. Curious to see how that develops!