Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload images to a cloud-hosting solution #2986

Open
karlcow opened this issue Oct 16, 2019 · 4 comments
Open

Upload images to a cloud-hosting solution #2986

karlcow opened this issue Oct 16, 2019 · 4 comments

Comments

@karlcow
Copy link
Contributor

@karlcow karlcow commented Oct 16, 2019

Currently images which are uploaded through webcompat.com are hosted on webcompat server. This issue is about creating the python code which will upload images to a cloud solution.

People upload images in two ways:

  1. Through GitHub. We have no control on this.
  2. Through webcompat.com. It also generates a thumbnail in addition to the big size version.

The image upload is done in the api/upload

def save(self):
'''Check that the file is allowed, then save to filesystem.'''
save_parameters = {}
if self.file_ext not in self.ALLOWED_FORMATS:
raise TypeError('Image file format not allowed')
# Paths of the images
file_dest = app.config['UPLOADS_DEFAULT_DEST'] + self.image_path
thumb_dest = app.config['UPLOADS_DEFAULT_DEST'] + self.thumb_path
dest_dir = os.path.dirname(file_dest)
if not os.path.exists(dest_dir):
os.makedirs(dest_dir)
# Alpha channels are not supported in JPEG
if (self.image_object.format == 'PNG'):
self.image_object = self.image_object.convert('RGB')
# Optimize further the image compression for these formats
if self.image_object.format in ['JPEG', 'JPG', 'JPE', 'PNG']:
save_parameters['optimize'] = True
# Convert PNG to JPEG. See issue #1051
file_dest = 'jpg'.join(file_dest.rsplit('png', 1))
# If animated GIF, aka duration > 0, add save_all parameter
if (self.image_object.format == 'GIF' and
self.image_object.info['duration'] > 0):
save_parameters['save_all'] = True
# unpacking save_parameters
self.image_object.save(file_dest, **save_parameters)
# Creating the thumbnail
size = (1024, 1024)
self.image_object.thumbnail(size, Image.HAMMING)
self.image_object.save(thumb_dest, **save_parameters)

We need to change the destination and handles oauth for accessing the service where we decide to upload the images.

This is related to mozilla/webcompat-team-okrs#108

@karlcow

This comment has been minimized.

Copy link
Contributor Author

@karlcow karlcow commented Oct 16, 2019

We will probably need to define a provider for image hosting. Some candidates

Digital Ocean Spaces

S3-compatible object storage with a built-in CDN that makes scaling easy, reliable, and affordable. $ 5 per month /mo

  • 250 GB of storage
  • 1 TB of outbound transfer
  • Unlimited uploads
  • Unlimited Spaces
  • $0.02 per additional GB stored
  • $0.01 per additional GB transferred

Amazon AWS S3

Pricing

Microsoft Azure Storage

Pricing

Other candidates Dropbox, Wasabi Cloud Storage (wasabi has a calculator for price comparison between services).

@karlcow

This comment has been minimized.

Copy link
Contributor Author

@karlcow karlcow commented Oct 18, 2019

Wasabi

it has exactly the same API than AWS S3. An example of code which overrides part of the boto3 code. boto3 is a python module used to interact with amazon.

@karlcow

This comment has been minimized.

Copy link
Contributor Author

@karlcow karlcow commented Oct 18, 2019

We currently have ~70000+ images from 2015-08 to 2019-10-17

  • from 0 bytes to 107,551,565 bytes
  • format distribution is
43327 jpg
25879 jpeg
 707 png
 119 gif
  14 bmp

The total weight is 4.9 Go

  • 2019: 1.3Go
  • 2018: 956 Mo
  • 2017: 2.1 Go
  • 2016: 599 Mo
  • 2015: 48 Mo

According to Wasabi:
1TB (we are far from that, only 5%)
with 5GB of download per month

would cost:

wasabi: 72$/year
aws s3: 337.92$/year

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.