Skip to content

Commit

Permalink
feat(uploads): add sharp operation example for images ✨
Browse files Browse the repository at this point in the history
  • Loading branch information
PierreBrisorgueil committed Apr 30, 2020
1 parent bab038b commit ad6461a
Show file tree
Hide file tree
Showing 18 changed files with 480 additions and 50 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Our stack node is actually in Beta.
| Linter | [ESLint](https://github.com/eslint/eslint) ecmaVersion 10 (2019)
| Security | JWT Stateless - [passport-jwt](https://github.com/themikenicholson/passport-jwt) <br> Passwords: [bcrypt](https://en.wikipedia.org/wiki/Bcrypt) - [zxcvbn](https://github.com/dropbox/zxcvbn) <br> DataBases options available (auth, ssl ..) <br> [SSL](https://github.com/weareopensource/Node/blob/master/WIKI.md#SSL) Express / Reverse Proxy (must be activated, otherwise => plain text password)
| API | Default answer wrapper (helper) : [jsend](https://github.com/omniti-labs/jsend) like : status, message, data or error <br> Default errors handling (helper) : formatted by the controller, Custom ES6 errors for other layers
| Upload | Example : [Mongo gridfs](https://docs.mongodb.com/manual/core/gridfs/) - [mongoose-gridfs](https://github.com/lykmapipo/mongoose-gridfs) - [Multer](https://github.com/expressjs/multer) <br> Avatar stream example available (could catch all contentType)
| Upload | Example : [Mongo gridfs](https://docs.mongodb.com/manual/core/gridfs/) - [mongoose-gridfs](https://github.com/lykmapipo/mongoose-gridfs) - [Multer](https://github.com/expressjs/multer) - [Sharp](https://github.com/lovell/sharp)<br> Avatar stream example available, with sharp options <br /> example could catch all contentType
| Logs | [winston](https://github.com/winstonjs/winston) [morgan](https://github.com/expressjs/morgan) *custom example available*
| CI | [Travis CI](https://travis-ci.org/weareopensource/Node)
| Developer | [Coveralls](https://coveralls.io/github/weareopensource/Node) - [Code Climate](https://codeclimate.com/github/weareopensource/Node) - [Dependency status](https://david-dm.org/weareopensource/node) - [Dependabot](https://dependabot.com/) - [Snyk](https://snyk.io/test/github/weareopensource/node) <br> [standard-version](https://github.com/conventional-changelog/standard-version) - [commitlint](https://github.com/conventional-changelog/commitlint) - [commitizen](https://github.com/commitizen/cz-cli) - [waos-conventional-changelog](https://github.com/WeAreOpenSourceProjects/waos-conventional-changelog)
Expand All @@ -55,7 +55,7 @@ Our stack node is actually in Beta.
* **User** : classic register / auth or oAuth(microsoft, google) - profile management (update, avatar upload ...) - **data privacy ok** (delete all data, get all data, send all by mail data)
* **Admin** : list users - get user - edit user - delete user
* **Tasks** : list tasks - get task - add tasks - edit tasks - delete tasks - **data privacy ok**
* **Uploads** : get upload - add upload - delete upload **data privacy ok**
* **Uploads** : get upload stream - add upload - delete upload - get image upload stream & sharp operations **data privacy ok**

## Prerequisites

Expand Down
14 changes: 8 additions & 6 deletions config/defaults/development.js
Original file line number Diff line number Diff line change
Expand Up @@ -91,12 +91,14 @@ module.exports = {
},
},
uploads: {
users: {
avatar: {
formats: ['image/png', 'image/jpeg', 'image/jpg'],
limits: {
fileSize: 1 * 1024 * 1024, // Max file size in bytes (1 MB)
},
avatar: {
formats: ['image/png', 'image/jpeg', 'image/jpg', 'image/gif'],
limits: {
fileSize: 1 * 1024 * 1024, // Max file size in bytes (1 MB)
},
sharp: {
sizes: ['128', '256', '512', '1024'],
operations: ['blur', 'bw', 'blur&bw'],
},
},
},
Expand Down
2 changes: 1 addition & 1 deletion lib/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ const startMongoose = async () => {
try {
await mongooseService.loadModels();
const dbConnection = await mongooseService.connect();
await multerService.setStorage(dbConnection);
await multerService.storage();
return dbConnection;
} catch (e) {
throw new Error(e);
Expand Down
5 changes: 3 additions & 2 deletions lib/services/multer.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
* Module dependencies.
*/
const path = require('path');
const _ = require('lodash');
const crypto = require('crypto');
const multer = require('multer');
const { createBucket } = require('mongoose-gridfs');
Expand All @@ -23,7 +24,7 @@ const fileFilter = (formats) => (req, file, callback) => {
/**
* set Strorage
*/
module.exports.setStorage = () => {
module.exports.storage = () => {
storage = createBucket({
bucketName: 'uploads',
model: 'Uploads',
Expand All @@ -38,7 +39,7 @@ module.exports.setStorage = () => {
*/
module.exports.create = (name, config) => async (req, res, next) => {
// set options
const options = config || {};
const options = _.cloneDeep(config) || {};
if (options.formats) {
options.fileFilter = fileFilter(options.formats);
delete options.formats;
Expand Down
6 changes: 3 additions & 3 deletions modules/core/tests/core.config.tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ describe('Configuration Tests:', () => {
let TaskService = null;

beforeAll(() => mongooseService.connect()
.then((dbconnection) => {
multerService.setStorage(dbconnection);
.then(async () => {
await multerService.storage();
mongooseService.loadModels();
UserService = require(path.resolve('./modules/users/services/user.service'));
TaskService = require(path.resolve('./modules/tasks/services/tasks.service'));
Expand Down Expand Up @@ -383,7 +383,7 @@ describe('Configuration Tests:', () => {

describe('Multer', () => {
test('should be able to get multer avatar configuration', () => {
const userAvatarConfig = config.uploads.users.avatar;
const userAvatarConfig = config.uploads.avatar;
expect(userAvatarConfig).toBeDefined();
expect(userAvatarConfig.formats).toBeInstanceOf(Array);
expect(userAvatarConfig.limits.fileSize).toBe(1048576);
Expand Down
4 changes: 2 additions & 2 deletions modules/tasks/tests/tasks.crud.tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ describe('Tasks CRUD Tests :', () => {
beforeAll(async () => {
try {
// init mongo
const dbconnection = await mongooseService.connect();
await multerService.setStorage(dbconnection);
await mongooseService.connect();
await multerService.storage();
await mongooseService.loadModels();
UserService = require(path.resolve('./modules/users/services/user.service'));
// init application
Expand Down
75 changes: 75 additions & 0 deletions modules/uploads/controllers/uploads.controller.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
* Module dependencies
*/
const path = require('path');
const sharp = require('sharp');
const _ = require('lodash');

const config = require(path.resolve('./config'));
const errors = require(path.resolve('./lib/helpers/errors'));
const responses = require(path.resolve('./lib/helpers/responses'));
const UploadsService = require('../services/uploads.service');
Expand All @@ -26,6 +29,37 @@ exports.get = async (req, res) => {
}
};

/**
* @desc Endpoint to get an upload by fileName with sharp options
* @param {Object} req - Express request object
* @param {Object} res - Express response object
*/
exports.getSharp = async (req, res) => {
try {
const stream = await UploadsService.getStream({ _id: req.upload._id });
if (!stream) responses.error(res, 404, 'Not Found', 'No Upload with that identifier can been found')();
stream.on('error', (err) => {
responses.error(res, 422, 'Unprocessable Entity', errors.getMessage(err))(err);
});
res.set('Content-Type', req.upload.contentType);
switch (req.sharpOption) {
case 'blur':
stream.pipe(sharp().resize(req.sharpSize).blur(20)).pipe(res);
break;
case 'bw':
stream.pipe(sharp().resize(req.sharpSize).grayscale()).pipe(res);
break;
case 'blur&bw':
stream.pipe(sharp().resize(req.sharpSize).grayscale().blur(20)).pipe(res);
break;
default:
stream.pipe(sharp().resize(req.sharpSize)).pipe(res);
}
} catch (err) {
responses.error(res, 422, 'Unprocessable Entity', errors.getMessage(err))(err);
}
};

/**
* @desc Endpoint to delete an upload
* @param {Object} req - Express request object
Expand Down Expand Up @@ -60,3 +94,44 @@ exports.uploadByName = async (req, res, next, uploadName) => {
next(err);
}
};

/**
* @desc MiddleWare to ask the service the uppload for this uploadImageName
* @param {Object} req - Express request object
* @param {Object} res - Express response object
* @param {Function} next - Express next middleware function
* @param {String} filename & params - upload filename & eventual params (two max) filename-maxSize-options.png
*/
exports.uploadByImageName = async (req, res, next, uploadImageName) => {
try {
// Name
const imageName = uploadImageName.split('.');
const opts = imageName[0].split('-');
if (imageName.length !== 2) responses.error(res, 404, 'Not Found', 'Wrong name schema')();
else if (opts.length > 3) responses.error(res, 404, 'Not Found', 'Too much params')();
else {
// data work
const upload = await UploadsService.get(`${opts[0]}.${imageName[1]}`);
if (!upload) responses.error(res, 404, 'Not Found', 'No Upload with that name has been found')();
else {
// options
const sharp = _.get(config, `uploads.${upload.metadata.kind}.sharp`);
if (opts[1] && (!sharp || !sharp.sizes)) responses.error(res, 422, 'Unprocessable Entity', 'Size param not available')();
else if (opts[1] && (!/^\d+$/.test(opts[1]) || !sharp.sizes.includes(opts[1]))) responses.error(res, 422, 'Unprocessable Entity', 'Wrong size param')();
else if (opts[2] && (!sharp || !sharp.operations)) responses.error(res, 422, 'Unprocessable Entity', 'Operations param not available')();
else if (opts[2] && !sharp.operations.includes(opts[2])) responses.error(res, 422, 'Unprocessable Entity', 'Operation param not available')();
else {
// return
req.upload = upload;
req.isOwner = upload.metadata.user; // used if we proteck road by isOwner policy
req.sharpSize = parseInt(opts[1], 0) || null;
req.sharpOption = opts[2] || null;
next();
}
}
}
} catch (err) {
console.log('err', err);
next(err);
}
};
1 change: 1 addition & 0 deletions modules/uploads/models/uploads.model.mongoose.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ const UploadsMongoose = new Schema({
type: Schema.ObjectId,
ref: 'User',
},
kind: String,
},
}, { strict: false });

Expand Down
3 changes: 3 additions & 0 deletions modules/uploads/policies/uploads.policy.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ exports.invokeRolesPolicies = () => {
allows: [{
resources: '/api/uploads/:uploadName',
permissions: ['get', 'delete'],
}, {
resources: '/api/uploads/images/:imageName',
permissions: ['get'],
}],
}]);
};
12 changes: 6 additions & 6 deletions modules/uploads/repositories/uploads.repository.js
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
/**
* Module dependencies
*/
const path = require('path');
const mongoose = require('mongoose');
const { createModel } = require('mongoose-gridfs');
const path = require('path');

const AppError = require(path.resolve('./lib/helpers/AppError'));
const Attachment = createModel({ bucketName: 'uploads', model: 'Uploads' });
const Uploads = mongoose.model('Uploads');


/**
* @desc Function to get all upload in db with filter or not
* @param {Object} Filter
* @return {Array} uploads
*/
exports.list = (filter) => Uploads.find(filter).select('filename uploadDate contentType').sort('-createdAt').exec();

/**
* @desc Function to get an upload from db
* @param {String} id
* @param {String} uploadName
* @return {Stream} upload
*/
exports.get = (uploadName) => Uploads.findOne({ filename: uploadName }).exec();

/**
* @desc Function to get an upload from db
* @param {String} id
* @desc Function to get an upload stream from db
* @param {Object} Upload
* @return {Stream} upload
*/
exports.getStream = (upload) => Attachment.read(upload);
Expand All @@ -40,7 +40,7 @@ exports.update = (id, update) => Uploads.findOneAndUpdate({ _id: id }, update, {

/**
* @desc Function to delete an upload from db
* @param {String} id
* @param {Object} upload
* @return {Object} confirmation of delete
*/
exports.delete = async (upload) => {
Expand Down
5 changes: 5 additions & 0 deletions modules/uploads/routes/uploads.routes.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ module.exports = (app) => {
.get(uploads.get)
.delete(policy.isOwner, uploads.delete); // delete

// classic crud
app.route('/api/uploads/images/:imageName').all(passport.authenticate('jwt'), policy.isAllowed)
.get(uploads.getSharp);

// Finish by binding the task middleware
app.param('uploadName', uploads.uploadByName);
app.param('imageName', uploads.uploadByImageName);
};
19 changes: 11 additions & 8 deletions modules/uploads/services/uploads.service.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ const UploadRepository = require('../repositories/uploads.repository');

/**
* @desc Function to ask repository to get an upload
* @param {String} id
* @return {Stream} upload
* @param {String} uploadName
* @return {Promise} Upload
*/
exports.get = async (uploadName) => {
const result = await UploadRepository.get(uploadName);
Expand All @@ -18,8 +18,8 @@ exports.get = async (uploadName) => {

/**
* @desc Function to ask repository to get stream of chunks data
* @param {String} id
* @return {Stream} upload
* @param {Object} Upload
* @return {Promise} result stream
*/
exports.getStream = async (upload) => {
const result = await UploadRepository.getStream(upload);
Expand All @@ -28,14 +28,17 @@ exports.getStream = async (upload) => {

/**
* @desc Function to ask repository to get an upload
* @param {String} id
* @return {Stream} upload
* @param {Object} req.file
* @param {Object} User
* @param {String} kind, upload configuration path (important for futur transformations)
* @return {Promise} Upload
*/
exports.update = async (file, user) => {
exports.update = async (file, user, kind) => {
const update = {
filename: await multer.generateFileName(file.filename || file.originalname),
metadata: {
user: user.id,
kind: kind || null,
},
};
const result = await UploadRepository.update(file._id, update);
Expand All @@ -44,7 +47,7 @@ exports.update = async (file, user) => {

/**
* @desc Function to ask repository to delete chunks data
* @param {String} id
* @param {Object} Upload
* @return {Promise} confirmation of delete
*/
exports.delete = async (upload) => {
Expand Down
Loading

0 comments on commit ad6461a

Please sign in to comment.