chore: replace safe browsing with web risk

[gweiying]

Context

Replaces Google Safe Browsing with Google Web Risk API, which has no daily quotas on number of links scanned.

Notes

• There seems to be minor differences in the threat types detected by Web Risk and Safe Browsing.
  • Web Risk: MALWARE SOCIAL_ENGINEERING UNWANTED_SOFTWARE SOCIAL_ENGINEERING_EXTENDED_COVERAGE
  • Safe Browsing: MALWARE SOCIAL_ENGINEERING UNWANTED_SOFTWARE POTENTIALLY_HARMFUL_APPLICATION
  • Checking w the Google team right now whether this will cause any adverse effect /regression
• Required abit of refactoring as the return types for Safe Browsing is different from Web Risk
  Safe Browsing return:

  // pass in list of urls, 
  // if threats found, returns list of matched threats
  {"matches": [{  "threatType":  "MALWARE", "platformType":  "WINDOWS", "threatEntryType": "URL", "threat": {"url": "http://www.urltocheck1.org/"}, "threatEntryMetadata": { "entries": [{ "key": "malware_threat_type", "value": "landing" }]}, "cacheDuration": "300.000s"}]}
  // if no threats 
  {}
  

  Web Risk return:

  // only single lookup endpoint available
  // if threat found, returns threat as object
  { "threat": { "threatTypes": ["MALWARE"], "expireTime": "2024-03-20T05:29:41.898456500Z"} }
  // if no threats
  {}
  

Tests

Tests on staging

• Try creating short link to malicious url, should see error "Link is likely malicious, please contact us for help"
• Create short link to safe url, should be able to create short link
• [API] Create short link to safe url, should be able to create short link
• [API] Try creating short link to malicious url, should see error "Link is likely malicious, please contact us for help"
• [Bulk] Try creating short links with bulk upload of 500 links, all safe, should be able to bulk create
• [Bulk] Try creating short links with bulk upload of 500 links, at least one malicious link, should not be able to bulk create
• See usage on Web Risk API on Google console

Deploy Notes

Before deploying,

• Activate Web Risk API on edu prod
• Activate Web Risk API on go prod
• Activate Web Risk API on health prod

After deployment, tests on production, 3 environments

• Try creating short link to malicious url, should see error "Link is likely malicious, please contact us for help"
• Create short link to safe url, should be able to create short link
• [API] Create short link to safe url, should be able to create short link
• [API] Try creating short link to malicious url, should see error "Link is likely malicious, please contact us for help"
• [Bulk] Try creating short links with bulk upload of 500 links, all safe, should be able to bulk create
• [Bulk] Try creating short links with bulk upload of 500 links, at least one malicious link, should not be able to bulk

New environment variables:
None, reusing existing Safe Browsing API keys

[halfwhole]

LGTM!

• agree that it looks ok to just set the cache time to 300s, no real need to follow the given expireTime so closely
• minor question, non-blocking: SOCIAL_ENGINEERING_EXTENDED_COVERAGE wasn't there before and so it looks like we don't need to add it right away, but i'm wondering if it's good to add in eventually?
• previously isThreatBulk could handle bulk requests all at once (up to 500), but now it sends individual requests for each URL, should we test that bulk uploads are okay too?
• in the future, refactoring to use "web risk" instead of "safe browsing" would be nice!

[gweiying]

LGTM!

* agree that it looks ok to just set the cache time to 300s, no real need to follow the given `expireTime` so closely

* minor question, non-blocking: `SOCIAL_ENGINEERING_EXTENDED_COVERAGE` wasn't there before and so it looks like we don't need to add it right away, but i'm wondering if it's good to add in eventually?

* previously `isThreatBulk` could handle bulk requests all at once (up to 500), but now it sends individual requests for each URL, should we test that bulk uploads are okay too?

* in the future, refactoring to use "web risk" instead of "safe browsing" would be nice!

Good points! Thanks for the quick review.

Added in SOCIAL_ENGINEERING_EXTENDED_COVERAGE as part of the threat types covered, I agree that it's good to add and err on the safe side!

Also ran tests on bulk upload and updated the test list above, thanks for flagging out.