Multiple file upload ParseFilePipeBuilder validation

[paramsinghvc]

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe it

When using FilesInterceptor for doing a bulk upload and trying to follow the docs for performing the files validation for max size and mime types, there's no definitive guide available to do that,

@UseInterceptors(
  FilesInterceptor('files[]', 5, {
    storage: diskStorage({
      destination: './dist/assets/uploads'
    }),
  }),
)
@Post('/bulk')
uploadFiles(
  @Body() _body: any,
  @UploadedFiles(
    new ParseFilePipeBuilder()
      .addFileTypeValidator({
        fileType: /(jpg|jpeg|png|gif)$/,
      })
      .addMaxSizeValidator({ maxSize: 5242880 })
      .build({
        errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY,
      }),
  )
  files: Express.Multer.File[],
) {
  const photo = new Photo();
  photo.name = file.filename;
  return this.photoService.create(photo);
}

Describe the solution you'd like

There should be defined way of doing these basic file validations when files are uploaded as an array.

Teachability, documentation, adoption, migration strategy

@UseInterceptors(
  FilesInterceptor('files[]', 5, {
    storage: diskStorage({
      destination: './dist/assets/uploads'
    }),
  }),
)
@Post('/bulk')
uploadFiles(
  @Body() _body: any,
  @UploadedFiles(
    new ParseFilePipeBuilder()
      .addFileTypeValidator({
        fileType: /(jpg|jpeg|png|gif)$/,
      })
      .addMaxSizeValidator({ maxSize: 5242880 })
      .build({
        errorHttpStatusCode: HttpStatus.UNPROCESSABLE_ENTITY,
      }),
  )
  files: Express.Multer.File[],
) {
  const photo = new Photo();
  photo.name = file.filename;
  return this.photoService.create(photo);
}

What is the motivation / use case for changing the behavior?

Right now, not able to perform or reuse the logic for multiple files upload.

[feeedback]

Need to add an example of using

.addFileTypeValidator({
       fileType: 'jpeg',
}) 

with a regular expression. Current examples sometimes seem like to validate the file extension rather than the mimetype . And people check /\.txt/

[gutovskii]

Any updates? 😕

[gutovskii]

How I solve this problem with multer:

////////////////// image-multer-options.ts
const imageFilter = (req: Request, file: Express.Multer.File, callback: (error: Error, acceptFile: boolean) => void) => {
    if (!Boolean(file.mimetype.match(/(jpg|jpeg|png|gif)/))) callback(null, false);
    callback(null, true);
}

export const imageOptions: MulterOptions = {
    limits: {fileSize: 5242880},
    fileFilter: imageFilter
}

////////////////// people.controller.ts
@Post() 
@ApiConsumes('multipart/form-data')
@UseInterceptors(FilesInterceptor('images', 100, imageOptions)) 
async createPerson(@Body(ValidationPipe) dto: CreatePersonDto, @UploadedFiles() images: Express.Multer.File[]) { 
    const personToCreate = this.mapper.map(dto, CreatePersonDto, Person); 
    const newPerson = await this.peopleService.create(personToCreate, images, dto.relations); 
    return newPerson;
}

But here we don't have a response notification that some of the files didn't pass the validation, instead incorrect file just won't pass to images array

[DanielMaranhao]

Is the @UploadedFiles decorator currently broken? As I'm sending a single image of 35KB and it accuses of exceeding 2MB. It cannot be used with ParseFilePipe, even though the tooltip states otherwise?

Sorry, I just noticed through my custom validator that it must treat as an array too. Otherwise any validation will fail. In this case, default validators won't work and I'll need to use custom implementations?

An example of a custom implementation of the MaxFileSizeValidator. It has the exact same functionality as the default one (as it inherits it), except it also accepts arrays of files.

import { MaxFileSizeValidator as DefaultMaxFileSizeValidator } from '@nestjs/common';

export class MaxFileSizeValidator extends DefaultMaxFileSizeValidator {  
  isValid(fileOrFiles: Express.Multer.File | Express.Multer.File[]): boolean {
    if (Array.isArray(fileOrFiles)) {
      const files = fileOrFiles;
      return files.every((file) => super.isValid(file));
    }

    const file = fileOrFiles;
    return super.isValid(file);
  }
}

[TonyDo99]

I'm not using @uploadedfiles decorator. Don't know why I can't catch errors in this. So in Multer, it had fileFilter method and I try using that to catch files right from the request, causing catch file from the request so u can check file size, name, path,..v..v.. is valid or not and threw an error if the file is not validated. This's an example hope it can help. If _.isMatch make u confuse, it's from lodash package.

MulterModule.register({
      fileFilter: (req, file, cb) => {
        console.log(file);
        if (_.isMatch(file, { fieldname: 'img' }) === false)
          cb(
            new UnsupportedMediaTypeException(
              'Missing field file to upload !. Please try again !',
            ),
            false,
          );
        else if (_.isMatch(file, { fieldname: 'images' }) === false)
          cb(
            new UnsupportedMediaTypeException(
              'Missing field file to upload !. Please try again !',
            ),
            false,
          );
        else cb(null, true);
      },

More detail in: https://www.npmjs.com/package/multer -> Find in fileFilter method

[brunnerh]

I wrapped one pipe in another to apply it to all files individually along the lines of:

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
  constructor(private readonly pipe: ParseFilePipe) { }

  async transform(files: Express.Multer.File[]) {
    for (const file of files)
      await this.pipe.transform(file);

    return files;
  }
}

The transform throws if the file is invalid. One could also catch and collect the errors to generate per-file messages.

[emircanok]

I wrapped one pipe in another to apply it to all files individually along the lines of:

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
  constructor(private readonly pipe: ParseFilePipe) { }

  async transform(files: Express.Multer.File[]) {
    for (const file of files)
      await this.pipe.transform(file);

    return files;
  }
}

The transform throws if the file is invalid. One could also catch and collect the errors to generate per-file messages.

@brunnerh, thanks for your reply. When you use FileFieldsInterceptor, files parameter type comes be object. For this issue, I updated the code as follows.

import { ParseFilePipe, PipeTransform } from '@nestjs/common';

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
  constructor(private readonly pipe: ParseFilePipe) {}

  async transform(
    files: Express.Multer.File[] | { [key: string]: Express.Multer.File },
  ) {
    if (typeof files === 'object') {
      files = Object.values(files);
    }

    for (const file of files) await this.pipe.transform(file);

    return files;
  }
}

[mohasalahh]

How I solve this problem with multer:

////////////////// image-multer-options.ts
const imageFilter = (req: Request, file: Express.Multer.File, callback: (error: Error, acceptFile: boolean) => void) => {
    if (!Boolean(file.mimetype.match(/(jpg|jpeg|png|gif)/))) callback(null, false);
    callback(null, true);
}

export const imageOptions: MulterOptions = {
    limits: {fileSize: 5242880},
    fileFilter: imageFilter
}

////////////////// people.controller.ts
@Post() 
@ApiConsumes('multipart/form-data')
@UseInterceptors(FilesInterceptor('images', 100, imageOptions)) 
async createPerson(@Body(ValidationPipe) dto: CreatePersonDto, @UploadedFiles() images: Express.Multer.File[]) { 
    const personToCreate = this.mapper.map(dto, CreatePersonDto, Person); 
    const newPerson = await this.peopleService.create(personToCreate, images, dto.relations); 
    return newPerson;
}

But here we don't have a response notification that some of the files didn't pass the validation, instead incorrect file just won't pass to images array

One thing also mentioned in multer doc is that you can throw errors in the filefilter function, which will be delegated to the express layer (which will then be delegated to nestjs), by passing an error to the callback function directly.

const imageFilter = (req: Request, file: Express.Multer.File, callback: (error: Error, acceptFile: boolean) => void) => {
    if (!Boolean(file.mimetype.match(/(jpg|jpeg|png|gif)/))) { 
        callback(new HttpException("Invalid files", HttpStatus.BAD_REQUEST), false);
    }else {
        callback(null, true);
    }
}

[joaopcm]

@emircanok , can you give us an example of how you're using your ParseFilesPipe?

[Mohanbarman]

I have created a custom file size and type validator to validate multiple files.

1. Create a new file for example file-validator.ts with the following contents

import { FileValidator } from '@nestjs/common';

type FileType = Express.Multer.File | Express.Multer.File[] | Record<string, Express.Multer.File[]>;
type Result = { errorFileName?: string; isValid: boolean };

export const runFileValidation = async (args: {
  multiple: boolean;
  file: FileType;
  validator: (file: Express.Multer.File) => Promise<boolean> | boolean;
}): Promise<Result> => {
  if (args.multiple) {
    const fileFields = Object.keys(args.file);
    for (const field of fileFields) {
      const fieldFile = args.file[field];
      if (Array.isArray(fieldFile)) {
        for (const f of fieldFile) {
          if (!args.validator(f)) {
            return { errorFileName: f.originalname, isValid: false };
          }
        }
      } else {
        if (!args.validator(fieldFile)) {
          return { errorFileName: fieldFile.originalname, isValid: false };
        }
      }
    }
    return { isValid: true };
  }

  if (Array.isArray(args.file)) {
    for (const f of args.file) {
      if (!args.validator(f)) {
        return { errorFileName: f.originalname, isValid: false };
      }
    }
    return { isValid: true };
  }

  if (args.validator(args.file as any)) {
    return { errorFileName: args.file.originalname as string, isValid: false };
  }

  return { isValid: true };
};

export class FileSizeValidator extends FileValidator {
  private maxSizeBytes: number;
  private multiple: boolean;
  private errorFileName: string;

  constructor(args: { maxSizeBytes: number; multiple: boolean }) {
    super({});
    this.maxSizeBytes = args.maxSizeBytes;
    this.multiple = args.multiple;
  }

  async isValid(
    file?: Express.Multer.File | Express.Multer.File[] | Record<string, Express.Multer.File[]>,
  ): Promise<boolean> {
    const result = await runFileValidation({
      file,
      multiple: this.multiple,
      validator: (f) => f.size < this.maxSizeBytes,
    });
    this.errorFileName = result.errorFileName;
    return result.isValid;
  }

  buildErrorMessage(file: any): string {
    return (
      `file ${this.errorFileName || ''} exceeded the size limit ` +
      parseFloat((this.maxSizeBytes / 1024 / 1024).toFixed(2)) +
      'MB'
    );
  }
}

export class FileTypeValidator extends FileValidator {
  private multiple: boolean;
  private errorFileName: string;
  private filetype: RegExp | string;

  constructor(args: { multiple: boolean; filetype: RegExp | string }) {
    super({});
    this.multiple = args.multiple;
    this.filetype = args.filetype;
  }

  isMimeTypeValid(file: Express.Multer.File) {
    return file.mimetype.search(this.filetype) === 0;
  }

  async isValid(
    file?: Express.Multer.File | Express.Multer.File[] | Record<string, Express.Multer.File[]>,
  ): Promise<boolean> {
    const result = await runFileValidation({
      multiple: this.multiple,
      file: file,
      validator: (f) => this.isMimeTypeValid(f),
    });
    this.errorFileName = result.errorFileName;
    return result.isValid;
  }

  buildErrorMessage(file: any): string {
    return `file ${this.errorFileName || ''} must be of type ${this.filetype}`;
  }
}

2. Now this can be used in a controller like

import { ParseFilePipe, Patch, UploadedFiles, UseInterceptors } from '@nestjs/common';
import { FileFieldsInterceptor } from '@nestjs/platform-express';
import { FileSizeValidator, FileTypeValidator } from './file-validator';

export class ImageController {
  @UseInterceptors(
    FileFieldsInterceptor([
      { name: 'logo', maxCount: 1 },
      { name: 'background', maxCount: 1 },
    ]),
  )
  @Patch('/upload-image')
  async updateLandingPage(
    @UploadedFiles(
      new ParseFilePipe({
        validators: [
          new FileSizeValidator({
            multiple: true,
            maxSizeBytes: 5 * 1024 * 1024, // 5MB
          }),
            new FileTypeValidator({
            multiple: true,
            filetype: /^image\/(jpeg|png|gif|bmp|webp|tiff)$/i,
          }),
        ],
      }),
    )
    files: {
      logo: Express.Multer.File[];
      background: Express.Multer.File[];
    },
  ) {
    console.log(files);
    return 'ok';
  }
}

[ekaram]

@emircanok , can you give us an example of how you're using your ParseFilesPipe?

  @Patch(':id/attempt/:attemptId/media')
  @UseInterceptors(AnyFilesInterceptor())
  async patchMedia(
    @UploadedFiles(
      new ParseFilesPipe(
        new ParseFilePipe({
          validators: [
            new MaxFileSizeValidator({ maxSize: 10000000 }),
            new FileTypeValidator({ fileType: /(jpg|jpeg|png|webp|gif|mp4|mov)$/ }),
          ]
        })
      )
    )
    files: Array<Express.Multer.File>
  ) {
    await this.uploadFiles(files);
  }

[joaopcm]

This is how I'm handling multi-file uploads using NestJS:

controller:

@Public()
  @ApiCreatedResponse()
  @ApiBadRequestResponse()
  @ApiNotFoundResponse()
  @ApiTags('Clients', 'Client Booking Process')
  @UseInterceptors(AnyFilesInterceptor(), new FilesSizeInterceptor())
  @Post(':formId/elements/:formElementId/upload')
  uploadImages(
    @Req() req: any,
    @Body() body: UploadImagesDto,
    @UploadedFiles()
    files: Express.Multer.File[],
  ) {
    new ImageFileValidationPipe().transform(files);
    return this.formsService.uploadImages(
      req.params.formId,
      req.params.formElementId,
      files,
      body,
    );
  }

FilesSizeInterceptor:

import {
  CallHandler,
  ExecutionContext,
  HttpException,
  Injectable,
  NestInterceptor,
} from '@nestjs/common';
import { Observable } from 'rxjs';

@Injectable()
export class FilesSizeInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const request = context.switchToHttp().getRequest();
    const files = request.files as Express.Multer.File[];

    for (const file of files) {
      if (file.size > 5 * 1024 * 1024) {
        throw new HttpException('File size too large', 400);
      }
    }

    return next.handle();
  }
}

ImageFileVlidationPipe:

import { BadRequestException, PipeTransform } from '@nestjs/common';

export class ImageFileValidationPipe implements PipeTransform {
  transform(files: Express.Multer.File[]): Express.Multer.File[] {
    const allowedMimeTypes = ['image/jpeg', 'image/png', 'image/webp'];
    for (const file of files) {
      if (!allowedMimeTypes.includes(file.mimetype)) {
        throw new BadRequestException('Invalid file type.');
      }
    }
    return files;
  }
}

[TonyDo99]

This is how I'm handling multi-file uploads using NestJS:

controller:

@Public()
  @ApiCreatedResponse()
  @ApiBadRequestResponse()
  @ApiNotFoundResponse()
  @ApiTags('Clients', 'Client Booking Process')
  @UseInterceptors(AnyFilesInterceptor(), new FilesSizeInterceptor())
  @Post(':formId/elements/:formElementId/upload')
  uploadImages(
    @Req() req: any,
    @Body() body: UploadImagesDto,
    @UploadedFiles()
    files: Express.Multer.File[],
  ) {
    new ImageFileValidationPipe().transform(files);
    return this.formsService.uploadImages(
      req.params.formId,
      req.params.formElementId,
      files,
      body,
    );
  }

FilesSizeInterceptor:

import {
  CallHandler,
  ExecutionContext,
  HttpException,
  Injectable,
  NestInterceptor,
} from '@nestjs/common';
import { Observable } from 'rxjs';

@Injectable()
export class FilesSizeInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const request = context.switchToHttp().getRequest();
    const files = request.files as Express.Multer.File[];

    for (const file of files) {
      if (file.size > 5 * 1024 * 1024) {
        throw new HttpException('File size too large', 400);
      }
    }

    return next.handle();
  }
}

ImageFileVlidationPipe:

import { BadRequestException, PipeTransform } from '@nestjs/common';

export class ImageFileValidationPipe implements PipeTransform {
  transform(files: Express.Multer.File[]): Express.Multer.File[] {
    const allowedMimeTypes = ['image/jpeg', 'image/png', 'image/webp'];
    for (const file of files) {
      if (!allowedMimeTypes.includes(file.mimetype)) {
        throw new BadRequestException('Invalid file type.');
      }
    }
    return files;
  }
}

I think you can handle the validate file type into Interceptor too instead of using another pipe

[joaopcm]

This is how I'm handling multi-file uploads using NestJS:
controller:

@Public()
  @ApiCreatedResponse()
  @ApiBadRequestResponse()
  @ApiNotFoundResponse()
  @ApiTags('Clients', 'Client Booking Process')
  @UseInterceptors(AnyFilesInterceptor(), new FilesSizeInterceptor())
  @Post(':formId/elements/:formElementId/upload')
  uploadImages(
    @Req() req: any,
    @Body() body: UploadImagesDto,
    @UploadedFiles()
    files: Express.Multer.File[],
  ) {
    new ImageFileValidationPipe().transform(files);
    return this.formsService.uploadImages(
      req.params.formId,
      req.params.formElementId,
      files,
      body,
    );
  }

FilesSizeInterceptor:

import {
  CallHandler,
  ExecutionContext,
  HttpException,
  Injectable,
  NestInterceptor,
} from '@nestjs/common';
import { Observable } from 'rxjs';

@Injectable()
export class FilesSizeInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const request = context.switchToHttp().getRequest();
    const files = request.files as Express.Multer.File[];

    for (const file of files) {
      if (file.size > 5 * 1024 * 1024) {
        throw new HttpException('File size too large', 400);
      }
    }

    return next.handle();
  }
}

ImageFileVlidationPipe:

import { BadRequestException, PipeTransform } from '@nestjs/common';

export class ImageFileValidationPipe implements PipeTransform {
  transform(files: Express.Multer.File[]): Express.Multer.File[] {
    const allowedMimeTypes = ['image/jpeg', 'image/png', 'image/webp'];
    for (const file of files) {
      if (!allowedMimeTypes.includes(file.mimetype)) {
        throw new BadRequestException('Invalid file type.');
      }
    }
    return files;
  }
}

I think you can handle the validate type file type into Interceptor too instead of using another pipe

You're totally right, but I use these different rules in different places. That's the main reason why I have these two validations in two different files.

[RobinKemna]

I wrapped one pipe in another to apply it to all files individually along the lines of:

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
  constructor(private readonly pipe: ParseFilePipe) { }

  async transform(files: Express.Multer.File[]) {
    for (const file of files)
      await this.pipe.transform(file);

    return files;
  }
}

The transform throws if the file is invalid. One could also catch and collect the errors to generate per-file messages.

@brunnerh, thanks for your reply. When you use FileFieldsInterceptor, files parameter type comes be object. For this issue, I updated the code as follows.

import { ParseFilePipe, PipeTransform } from '@nestjs/common';

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
  constructor(private readonly pipe: ParseFilePipe) {}

  async transform(
    files: Express.Multer.File[] | { [key: string]: Express.Multer.File },
  ) {
    if (typeof files === 'object') {
      files = Object.values(files);
    }

    for (const file of files) await this.pipe.transform(file);

    return files;
  }
}

That was very helpful and nearly did the trick for me only the reassignment of files changed the output so that the anticipated format was not matching in the controller.
Thus I would suggest the following approach to not change the input data structure:

import { ParseFilePipe, PipeTransform } from '@nestjs/common';

export class ParseFilesPipe implements PipeTransform<Express.Multer.File[]> {
    constructor(private readonly pipe: ParseFilePipe) {}

    async transform(
        files: Express.Multer.File[] | { [key: string]: Express.Multer.File[] },
    ) {
        for (const file of Object.values(files).flat())
            await this.pipe.transform(file);

        return files;
    }
}