Revisit check-url behavior and provide User-Agent a custom default value

[benoit74]

Fix #228
Fix #227
Fix #230

Changes:

• User-Agent now has a default value (based on my Firefox on Mac/OS, not sure this is the best one)
• The same UA is passed to Python requests to check URL and to browsertrix crawler
  • This UA is based on the concatenation of CLI flags --userAgent (always has a value due to default) + optional --userAgentSuffix + optional --adminEmail ; these 2 last values are automatically prefixed by a space, no need to provide one (or if someone does, he will have two spaces which is not an issue)
  • Except when a --mobileDevice is passed, then UA is only used for Python requests
  • If both --mobileDevice and --userAgent are passed, a warning is displayed
  • This UA is always used, even when a --mobileDevice is passed
• In check_url:
  • we do a GET instead of a HEAD
  • we close the connection

NOTA: this won't be totally functional until webrecorder/browsertrix-crawler#419 is fixed (once it is fixed and used by us, there is no change needed except adapt the test case which has been made a little too permissive due to this bug)

[rgaudin]

Looks good but I think custom UA should apply to the crawler even when using --mobileDevice. I don't see a use case for specifying it only for our python check.
Is that a crawler limitation?

[benoit74]

No, it's not a crawler limitation, it is something I even had to "force" somehow with specific code.
Since it is covered by a test case, I thought that the most important thing when passing a --mobileDevice is to set the proper UA.
But you know the business way better than I do, so I do not mind to always use our custom UA even when a mobileDevice is set.
From my culture, aside the UA do you know what else is changed when passing a --mobileDevice to Pupetteer ?

[rgaudin]

Mostly the apparent device sizes (resolution).

That's used mostly to crawl mobile versions of website or to have the appropriate media queries when using it for screenshots.
Modern websites also serve different sized pictures based on media queries

[benoit74]

I have adapted the code to always use the UA, even when mobile device is set. I updated the first comment.
This change allowed me to discover a bug in crawler, see NOTA in first comment (probably not a big concern, at least we can proceed without this fix).

[rgaudin]

LGTM ; please split the long line

[benoit74]

I split the line, I let you merge if everything is OK.