Have a filesystem that supports both local and Cloud storages (such as AWS S3)

[LoicPoullain]

Issue

Global issue: Downloading and uploading files should be an easy task in FoalTS, whether we use the local file system or a Cloud storage (such as Amazon S3).

This particular issue: For this, we need an agnostic file storage where it will be easy to switch drivers from one environment to another. For example, with the same code but with different configuration files, we would use the local file system when developing the application locally and use AWS S3 when deploying the application to staging or production.

Solution and Examples

@dependency
disk: Disk;

Write files (examples)

const { path } = await this.disk.write('avatars/', myBuffer);
const { path } = await this.disk.write('avatars/', myStream);
const { path } = await this.disk.write('avatars/', myBuffer, {
  name: 'my-avater.jpg'
});

Read files (examples)

const { file } = await this.disk.read('avatars/xxx.jpg', 'buffer');
const { file } = await this.disk.read('avatars/xxx.jpg', 'stream');

Delete files (examples)

await this.disk.delete('avatars/xxx.jpg');

Download files in streaming (HTTP response)

@Get('/download')
download() {
  return this.disk.createHttpResponse('avatars/xxx.jpg');
}

@Get('/download')
download() {
  return this.disk.createHttpResponse('avatars/xxx.jpg', {
    forceDownload: true,
    filename: 'avatar.jpg'
  });
}

Configuration file for the local file system

settings:
  disk:
    driver: 'local'
    directory: 'uploaded'

Configuration file for AWS S3

settings:
  disk:
    driver: 's3'
    bucket: 'my-bucket'

Steps

TODO: Think how errors should be handled (especially errors in streams).

1. Create a new package named @foal/storage.

2. Add the abstract class AbstractDisk. All disks should inherit this class. Take a look at the function createHttpResponseFile when implementing the createHttpResponse method.

   type Type<C extends 'buffer'|'stream'> =
     C extends 'buffer' ? Buffer :
     C extends 'stream' ? ReadableStream :
     never;
   
   export abstract class AbstractDisk {
     abstract write(
       dirname: string,
       content: Buffer|ReadableStream,
       options?: { name?: string },
     ): Promise<{ path: string }>;
   
     abstract read<C extends 'buffer'|'stream'>(path: string, content: C): Promise<{ file: Type<C> }>;
   
     abstract delete(path: string): Promise<void>;
   
     createHttpResponse(path: string, options?: {
       forceDownload?: boolean, filename?: string
     }): Promise<HttpResponse>;
   }

3. Add the class LocalDisk. This class will use Node fs package and use the configuration key settings.disk.directory. Take also a look at the function createHttpResponseFile.

4. Add the class S3Disk. This class will use AWS SDK and use the configuration key settings.disk.bucket. In an external package called @foal/aws-s3?

5. Add the class Disk. This class will use the configuration settings.disk.drive to choose which disk to use.

Additional Notes

• In future major versions, remove createHttpResponseFile as it can be replaced with this.disk.createHttpReponse and remove mime from @foal/core dependencies.

Resources

• https://docs.aws.amazon.com/fr_fr/AWSJavaScriptSDK/latest/AWS/S3.html
• https://laravel.com/docs/5.8/filesystem
• https://nodejs.org/docs/latest-v8.x/api/fs.html

Related issues

#421

#560

[LoicPoullain]

PR merged. It will be released with version 1.6.