Skip to content

[WIP] Feat/cloudflare #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leomp12 merged 20 commits into from
Feb 3, 2022
Merged

[WIP] Feat/cloudflare #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
daaa144
feat: creating cloudflare adapter
Mazurco066 Jan 27, 2022
99d8ca4
feat: cloudflare integration
Mazurco066 Jan 28, 2022
7363dce
feat: cloudflare integration
Mazurco066 Jan 28, 2022
2ef8a2e
feat: cloudflare integration
Mazurco066 Jan 28, 2022
a431341
feat: cloudflare integration
Mazurco066 Jan 28, 2022
814bbb3
fix: ecom review #1
Mazurco066 Jan 31, 2022
1d84453
fix: ecom review #1
Mazurco066 Jan 31, 2022
7f0f8b9
fix: ecom review #3
Mazurco066 Jan 31, 2022
4f9d339
fix: ecom review #4
Mazurco066 Jan 31, 2022
efbc64e
fix: ecom review #4
Mazurco066 Jan 31, 2022
a4a44ef
fix: ecom review #5
Mazurco066 Feb 1, 2022
ff92c4a
fix: ecom review #6
Mazurco066 Feb 1, 2022
da504b6
fix: ecom review #7
Mazurco066 Feb 1, 2022
1d1e711
fix: ecom review #8
Mazurco066 Feb 1, 2022
d7f89aa
fix: ecom review #8
Mazurco066 Feb 1, 2022
ae6f770
fix: ecom review #9
Mazurco066 Feb 1, 2022
03fe3dd
refactor(uploader): cloudflare uploader
Mazurco066 Feb 1, 2022
ff43238
test(logic): testing logic
Mazurco066 Feb 2, 2022
12340cd
refactor: logic
Mazurco066 Feb 2, 2022
47c175d
refactor: logic
Mazurco066 Feb 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
251 changes: 92 additions & 159 deletions bin/web.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ const Express = require('express')
const multer = require('multer')
// body parsing middleware
const bodyParser = require('body-parser')
// UUID Generator
const { v4 } = require('uuid')
Comment on lines +26 to +27
Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// UUID Generator
const { v4 } = require('uuid')


// Redis to store buckets and Kraken async requests IDs
const redis = require('redis')
Expand Down Expand Up @@ -219,10 +221,7 @@ fs.readFile(path.join(__dirname, '../config/config.json'), 'utf8', (err, data) =
// setup multer for file upload
const localUpload = multer({
storage: multer.memoryStorage(),
limits: {
// maximum 2mb
fileSize: 2000000
}
limits: { fileSize: 2000000 }
})

const baseS3Options = {
Expand All @@ -231,181 +230,114 @@ fs.readFile(path.join(__dirname, '../config/config.json'), 'utf8', (err, data) =
}

app.post(urls.upload, (req, res) => {
const { storeId } = req
const { bucket, host } = spaces[0]
logger.log(`${storeId} Uploading...`)
// unique object key
let key = '@v3/'
logger.log(`[storage-api] Upload start...`,)

// Validate file
localUpload.single('file')(req, res, (err) => {
if (err) {
const usrMsg = {
sendError(res, 400, 3001, err.message, {
en_us: 'This file cannot be uploaded, make sure it is a valid image with up to 2mb',
pt_br: 'O arquivo não pôde ser carregado, verifique se é uma imagem válida com até 2mb'
}
sendError(res, 400, 3001, err.message, usrMsg)
})
} else {

// Retrieve request parameters
let key = '@v3/'
const { bucket, host } = spaces[0]

// Retrieve local image data
let dir = req.query.directory
if (typeof dir === 'string' && dir.charAt(0) === '/') {
// remove first char to not duplicate first bar
// normalize, then remove empty paths
dir = dir.substr(1).replace(/[^\w-/]/g, '').replace('//', '')
if (dir.length) {
key += dir.toLowerCase() + '/'
}
}
// keep filename

// Keep filename
const filename = req.file.originalname.replace(/[^\w-.]/g, '').toLowerCase()
key += `${Date.now()}-${filename}`
const { mimetype } = req.file

runMethod('putObject', {
...baseS3Options,
ContentType: mimetype,
Key: `${storeId}/${key}`,
Body: req.file.buffer
})
// S3 Response
.then(() => {
logger.log(`${storeId} ${key} Uploaded to ${bucket}`)
// zoom uploaded
const mountUri = (key, baseUrl = cdnHost || host) => `https://${baseUrl}/${storeId}/${key}`
const uri = mountUri(key)
const picture = { zoom: { url: uri } }
const pictureBytes = {}
// resize/optimize image
let i = -1
let transformedImageBody = null

const respond = () => {
logger.log(`${storeId} ${key} ${bucket} All optimizations done`)
res.json({ bucket, key, uri, picture })
key += `${v4()}-${filename}`
Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
key += `${v4()}-${filename}`
key += `${Date.now()}-${filename}`

acho que o uuid só pra isso aqui é exagero, principalmente porque ainda concatena com o filename original..


// Aux methods
const mountUri = (key, baseUrl = cdnHost || host) => `https://${baseUrl}/${req.storeId}/${key}`
let picture = {}
const respond = (suc = true) => {
if (suc) {
logger.log(`${req.storeId} ${key} ${bucket} All optimizations done`)
res.json({ bucket, key, uri: picture['zoom'] ? picture['zoom'].url : '' , picture })
Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

na verdade se a zoom não existir teremos problemas haha

Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aí a resposta não poderia ser 200

} else {
res.status(500).json({ message: {
en_us: 'An error ocourred whlie uploading your file.',
pt_br: 'Ocorreu um erro ao persistir seu arquivo'
}})
}
}

const callback = err => {
if (!err) {
// next image size
i++
if (i < pictureOptims.length) {
let newKey
const { label, size, webp } = pictureOptims[i]
newKey = `imgs/${label}/${key}`

const imageBuffer = i === 0 ? req.file.buffer : transformedImageBody
const imageBase64 = imageBuffer
? `data:${mimetype};base64,${imageBuffer.toString('base64')}`
: null
// free memory
transformedImageBody = req.file = null

setTimeout(() => {

// Retrieve url
let originUrl
if (picture[label] && webp) {
originUrl = picture[label].url
} else {
originUrl = uri
}

// Transform image updated to cloudflare
const transformImg = (isRetry = false) => {

cloudflare(imageBase64 || originUrl, label === 'small' ? 'w90' : label, (err, data) => {
if (!err && data) {
const { id, url, imageBody } = data
return new Promise(resolve => {
// Cloudinary keeps image as webp, so we using webp as default
const contentType = 'image/webp'
const fileFormat = 'webp'
if (imageBody || id) {
const s3Options = {
...baseS3Options,
ContentType: contentType,
Key: `${storeId}/${newKey}.${fileFormat}`
}
if (imageBody) {
transformedImageBody = imageBody
// PUT new image on S3 bucket
return runMethod('putObject', { ...s3Options, Body: imageBody })
.then(() => resolve(mountUri(newKey)))
.catch((err) => {
logger.error(err)
resolve(url)
})
}
// async handle with callback URL
redisClient.setex(genRedisKey(id, true), 600, JSON.stringify(s3Options))
return resolve(mountUri(newKey))
}
resolve(url)

}).then(url => {
if (url && (!picture[label] || pictureBytes[label] > bytes)) {
picture[label] = { url, size }
pictureBytes[label] = bytes
}
callback()
})
}

if (
err &&
typeof err.message === 'string' &&
(err.message.indexOf('504 Gateway Timeout') > -1 || err.message.indexOf('503 Service Unavailable') > -1)
) {
if (!isRetry) {
return setTimeout(() => transformImg(true), 1000)
} else {
return respond()
}
}
callback(err)
})

}

// Transofrm image
transformImg()

}, imageBase64 ? 50 : 300)
} else {
setTimeout(() => {
// all done
respond()
}, 50)
}
// Upload to cloudiflare
cloudflare(req.file, (err, data) => {

// No errors by the way
if (!err && data) {

// Retrieve converted images
const { convertedImages } = data
let s3attempts = 0

// Map converted images and upload to S3
convertedImages.forEach(({ id, label, imageBody }) => {

const newKey = `imgs/${label}/${key}`

// Upload image to s3
return new Promise((resolve) => {

// Define s3 options
const contentType = label === 'zoom' ? 'image/jpeg' : 'image/webp'
const fileFormat = label === 'zoom' ? 'jpg' : 'webp'
Comment on lines +295 to +296
Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ué mas a imagem original não necessariamente é jpg na verdade

const s3Options = {
...baseS3Options,
ContentType: contentType,
Key: `${req.storeId}/${newKey}.${fileFormat}`
}

// Put s3 object
if (imageBody) {
return runMethod('putObject', { ...s3Options, Body: imageBody })
.then(() => resolve(mountUri(newKey)))
.catch((err) => {
logger.error(err)
resolve(key)
})
}

// async handle with callback URL
redisClient.setex(genRedisKey(id, true), 600, JSON.stringify(s3Options))
Comment on lines +313 to +314
Copy link
Copy Markdown
Member

@leomp12 leomp12 Feb 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isso aqui era só pra callback do cloudinary pra manipulação assíncrona na verdade, o cloudflare não envia esse webhook então isso não tem utilidade mais..

return resolve(mountUri(newKey))
})
.then((url) => {
s3attempts++
if (url && (!picture[label])) {
picture[label] = { url: mountUri(url), size: label }
// pictureBytes[label] = bytes
}
if (s3attempts === 4 && Object.keys(picture).length === 4) {
return respond()
} else if (s3attempts === 4) {
return sendError(res, 500, 3002, 'Internal server error', {
en_us: 'An error ocourred whlie uploading your file.',
pt_br: 'Ocorreu um erro ao persistir seu arquivo'
})
}
})
})

} else if (uri && typeof err.message === 'string' && err.message.indexOf('cloud_name') > -1) {
// image uploaded but not transformed
respond()
logger.error(err)
} else {
// respond with error
const usrMsg = {
en_us: 'Error while handling image, the file may be protected or corrupted',
pt_br: 'Erro ao manipular a imagem, o arquivo pode estar protegido ou corrompido'
}
sendError(res, 415, uri, err.message, usrMsg)
}
//return respond()
}

switch (mimetype) {
case 'image/jpeg':
case 'image/png':
callback()
break
default:
respond()
}
})
// CDN Upload error
.catch((err) => {
const usrMsg = {
en_us: 'This file cannot be uploaded to CDN',
pt_br: 'O arquivo não pôde ser carregado para o CDN'
// Sadness and sorrow
if (err) {
return respond(false)
}
sendError(res, 400, 3002, err.message, usrMsg)
})
}
})
Expand Down Expand Up @@ -481,6 +413,7 @@ fs.readFile(path.join(__dirname, '../config/config.json'), 'utf8', (err, data) =
})

app.listen(port, () => {
console.log('App is running')
logger.log('Storage API running with Express on port ' + port)
})
}
Expand Down
Loading