File to upload exceeds max size

[say-yawn]

About this PR

As a fix to #104 gracefully exit if the file is bigger than the max size.

Acceptance Criteria

• If the file to upload exceeds the maximum file size that is downloadable by Firefox gracefully exit
• Shavar list creation should continue to upload the rest of the lists

Practical test

Test shavar-list-creationg gracefully exits if the file is bigger than the max size on Firefox

1. Go to shavar_list_creation.ini and add the section:

[google-whitelist]
entity_url=https://raw.githubusercontent.com/mozilla-services/shavar-prod-lists/create-seperate-entitylist-for-google/google-entitylist.json
output=google-trackwhite-digest256
s3_key=entity/google-trackwhite-digest256

2. Go to constants.py and edit the WHITELIST_SECTIONS to include "google-whitelist"
3. Run shavar-list-creation by executing the command python ./lists2safebrowsing.py
4. Check that you get the following message at least once
   !!! google-trackwhite-digest256 NOT UPLOADED. File size is 1434496 which exceeded max size 1048576 !!!
5. Check that the google-trackwhite-digest256 is NOT uploaded to S3

Dependent on chunk size increase on Firefox

[englehardt]

Is there a reason to use print instead of throwing an error? With print we will still upload lists which don't exceed the limit, but what if we may related changes across multiple lists. Let's say we add new domains to the blocklist which should have corresponding entitylist updates, but the entity list exceeds the chunksize. We'd upload the new blocklist but not the new entitylist, which might lead to breakage. Thus. it seems safer to throw an error. Are there any downsides to throwing an error?

[say-yawn]

No downsides I could think of. Though now I think about this, we should probably make the size be dependent on the version as well right? Since Nightly (73) will have the increased chunk size while the Beta and Release will still have the older max chunk size.

[englehardt]

If it's not too much work that would be safest.