Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement File Upload Handling and Update Documentation #8

Merged
merged 2 commits into from
Aug 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,13 @@ ___
- **Automatic Asset Management:** Automatically include stylesheets, scripts, and meta tags from components and global settings in the rendered pages.
- **Preserved Asset Order:** Maintain the order of assets as defined, ensuring predictable rendering.
- **Secure Server** with **FREE** [SSL certificates](https://github.com/nasriyasoftware/HyperCloud/blob/main/examples/proxies.md#generate-ssl-certificates).
- Works well with proxies.
- Built-In & Custom [Error Pages](#error-handling--pages).
- Built-in (In-Memory) [Rate Limiter](#rate-limiter).
- Built-in [Helmet](#helmet-protection) Protection.

### Additional Features
- **[File Upload Handling](#file-upload-handling)**: Manage file uploads with built-in support for configuring file size limits, handling different file types, and automatically cleaning up temporary files.
- Works well with proxies.
___
## Status [![Status](https://img.shields.io/badge/Status-Stable-green.svg)](link-to-your-status-page)
If you encounter an issue or a bug, please [open an issue](https://github.com/nasriyasoftware/HyperCloud/issues).
Expand Down Expand Up @@ -213,6 +215,33 @@ import logify from '@nasriya/logify';

server.handlers.logger(logify.middlewares.hypercloud);
```
#### File Upload Handling
The framework includes a robust file upload handling module that supports various file types and sizes. It allows for flexible configuration of file size limits, dynamic directory management, and efficient memory usage through file streaming. This feature is designed to handle multipart form data, automatically manage temporary files, and integrate seamlessly with other server functionalities.

Here's how:
```ts
router.post('/api/v1/uploads', async (request, response, next) => {
try {
// Process the form data and handle the files
await request.processFormData(response);

// Extract fields, files, and the cleanup function from the request body
const { fields, files, cleanup } = request.body as FormDataBody;

// Process the files and fields (e.g., store files, update database)
// ............................

// Clean up temporary files after processing
await cleanup();

// Return a response or proceed to the next middleware/handler
next();
} catch(error) {
response.status(500).json(error);
}
});
```
Learn more about **File Upload Handling** [here](https://github.com/nasriyasoftware/HyperCloud/blob/main/examples/uploads.md).

#### Generating eTags
[ETags](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) can significantly improve server performance. To generate `eTags` for your resources, use the following syntax:
Expand Down
132 changes: 132 additions & 0 deletions examples/uploads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Server Uploads
This documentation covers the essential details of the `Uploads` module and its components, providing clarity on configuration, usage, and handling of form data.

## Overview

The `Uploads` module in the `HyperCloudServer` class provides a robust mechanism for handling file uploads. It allows developers to manage and configure various aspects of file uploads, such as setting file size limits for different types of files (images, videos, etc.), specifying upload directories, and handling MIME type-specific limits.

This document provides an overview of the module, detailing its features, usage examples, and configuration options.

Access the uploads module:
```js
server.uploads
```

## Features

- **Flexible File Size Limits:** Configure maximum file size limits for different types of files (images, videos, etc.) and specific MIME types.
- **Dynamic Directory Management:** Automatically create the upload directory if it doesn't exist.
- **Streamlined Upload Handling:** Use a file stream limit to manage memory usage efficiently by determining when uploads are stored in a stream rather than in memory.
- **MIME Type-Specific Limits:** Set and retrieve file size limits based on specific MIME types.

## Configuration Options

### `maxFileSize`

- **Type:** `number | StorageSize`
- **Default:** `100MB`
- **Description:** Sets the maximum allowed file size for uploads. It can be specified either as a number representing bytes or as an object with `value` and `unit` (e.g., `{ value: 10, unit: 'MB' }`).

### `directory`

- **Type:** `string`
- **Default:** `temp/uploads`
- **Description:** Specifies the directory where uploads are stored. If the directory does not exist, it is automatically created.

### `limits`

- **Type:** `UploadLimits`
- **Description:** Provides access to methods for setting and retrieving file size limits for different types of files and specific MIME types.

___
## Usage

### Setting File Size Limits

You can set the maximum allowed file size for images, videos, and specific MIME types using the `limits` property of the `Uploads` instance.

```js
// Set the maximum file size for images to 10 MB
server.uploads.limits.images = { value: 10, unit: 'MB' };

// Set the maximum file size for videos to 10485760 bytes (10 MB)
server.uploads.limits.videos = 10485760;

// Set a specific limit for a MIME type
server.uploads.limits.mime.set('application/pdf', { value: 5, unit: 'MB' });
```

### Retrieving File Size Limits
You can retrieve the current maximum file size for images, videos, or specific MIME types:

```js
const maxImageSize = server.uploads.limits.images;
const maxVideoSize = server.uploads.limits.videos;
const pdfLimit = server.uploads.limits.mime.get('application/pdf');
```

### Directory Management
The upload directory can be configured using the directory property. The directory is created automatically if it does not exist.

```js
server.uploads.directory = '/path/to/uploads';
```
___
## Example
### Processing Form Data
Handles incoming multipart form data for file uploads. This method processes the form data in an HTTP request, manages the upload process, and includes a `cleanup` function in the request `body` to handle temporary files after processing.

```ts
router.post('/api/v1/uploads', async (request, response, next) => {
try {
// Process the form data and handle the files
await request.processFormData(response);

// Extract fields, files, and the cleanup function from the request body
const { fields, files, cleanup } = request.body as FormDataBody;

// Process the files and fields (e.g., store files, update database)
// ............................

// Clean up temporary files after processing
await cleanup();

// Return a response or proceed to the next middleware/handler
next();
} catch(error) {
response.status(500).json(error);
}
});
```

#### Parameters

- **`response`**:
- **Type**: `HyperCloudResponse`
- **Description**: The response object used to send responses back to the client.

#### Returns

- **Type**: `Promise<void>`
- **Description**: A promise that resolves when the form data has been processed. The cleanup function is included in the request body and should be called after processing the files.

#### Throws

- **`Error`**: If an error occurs during form data processing, the response will send a 500 status with an error message, and the error will be re-thrown.

#### `FormDataBody` Interface
Represents the body of a request after processing form data. It includes fields and files from the form data, as well as a cleanup function.

#### Properties

- **`fields`**:
- **Type**: `Record<string, string>`
- **Description**: Contains form fields and their values extracted from the form data.

- **`files`**:
- **Type**: `(FormDataMemoryFile | FormDataStorageFile)[]`
- **Description**: An array of files uploaded through the form. Includes both memory files and storage files.

- **`cleanup`**:
- **Type**: `UploadCleanUpFunction`
- **Description**: A function to clean up temporary files after processing. Should be called after files have been processed, such as copying them to a permanent location or storing their metadata.
182 changes: 177 additions & 5 deletions src/docs/docs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,16 +74,188 @@ export type MimeType =

export type OnRenderHandler = (locals: Record<string, any> | any, include: (name: string, locals: Record<string, any>) => Promise<string>, lang: string) => string | Promise<string>;

export type StorageUnitName =
| 'Bit'
| 'Byte'
| 'Kilobyte'
| 'Kibibyte'
| 'Megabyte'
| 'Mebibyte'
| 'Gigabyte'
| 'Gibibyte'
| 'Terabyte'
| 'Tebibyte'
| 'Petabyte'
| 'Pebibyte'
| 'Exabyte'
| 'Exbibyte'
| 'Zettabyte'
| 'Zebibyte'
| 'Yottabyte'
| 'Yobibyte'
| 'Brontobyte'
| 'Geopbyte'
| 'Nibble'
| 'Word';

export type StorageUnitAbbreviation =
| 'b' // Bit
| 'B' // Byte
| 'KB' // Kilobyte
| 'KiB'// Kibibyte
| 'MB' // Megabyte
| 'MiB'// Mebibyte
| 'GB' // Gigabyte
| 'GiB'// Gibibyte
| 'TB' // Terabyte
| 'TiB'// Tebibyte
| 'PB' // Petabyte
| 'PiB'// Pebibyte
| 'EB' // Exabyte
| 'EiB'// Exbibyte
| 'ZB' // Zettabyte
| 'ZiB'// Zebibyte
| 'YB' // Yottabyte
| 'YiB'// Yobibyte
| 'BB' // Brontobyte
| 'GPB'; // Geopbyte

export type StorageUnit = StorageUnitName | StorageUnitAbbreviation;
export type UploadCleanUpFunction = () => Promise<void>;

export interface UploadLimitsController {
fileStream: {
get: () => number;
set: (limit: number) => void;
};
images: {
get: () => number;
set: (limit: number) => void;
};
videos: {
get: () => number;
set: (limit: number) => void;
};
mime: {
get: (mime: MimeType) => number | undefined;
set: (mime: MimeType, limit: number) => void;
};
}

export interface StorageSize {
/**A positive value */
value: number;
/**The storage unit */
unit: StorageUnit
}

/**
* Represents the body of a form data request after parsing.
* This interface includes metadata about the fields and files from the form data,
* as well as a cleanup function for managing temporary files.
*/
export interface FormDataBody {
/**
* A record of form fields, where the keys are field names and the values are the field data.
* This includes all non-file fields submitted in the form.
*
* @type {Record<string, any>}
* @example
* {
* "username": "john_doe",
* "age": 30
* }
*/
fields: Record<string, any>;
files: FormDataFile[];

/**
* An array of file objects representing the uploaded files.
* Each file object can either be a `FormDataMemoryFile` or `FormDataStorageFile`,
* depending on whether the file was stored in memory or on disk.
*
* @type {(FormDataMemoryFile | FormDataStorageFile)[]}
* @example
* [
* {
* fieldName: "profile_picture",
* fileName: "john.jpg",
* mime: "image/jpeg",
* size: 123456,
* content: <Buffer 89 50 4e ...> // For FormDataMemoryFile
* },
* {
* fieldName: "large_file",
* fileName: "document.pdf",
* mime: "application/pdf",
* size: 98765432,
* path: "/temp/uploads/document.pdf" // For FormDataStorageFile
* }
* ]
*/
files: (FormDataMemoryFile | FormDataStorageFile)[];

/**
* A function that cleans up temporary files created during the file upload process.
* This function should be called to remove any temporary files after processing is complete,
* such as after copying files to their final location or storing their metadata in a database.
*
* @type {UploadCleanUpFunction}
* @example
* // Call this function to clean up temporary files
* formDataBody.cleanup();
*/
cleanup: UploadCleanUpFunction;
}


/**Represents a file in form data. */
export interface FormDataFile {
name: string;
filename: string;
contentType: MimeType,
content: Buffer
/**
* The name of the form field associated with the file.
* @example 'profilePicture'
*/
fieldName: string;
/**
* The name of the uploaded file.
* @example 'profile.jpg'
*/
fileName: string;
/**
* The MIME type of the file.
* @example 'image/jpeg'
*/
mime: MimeType;
/**
* The size of the file in bytes.
* @example 204800
*/
size: number;
}

/**
* Represents a small file stored in memory.
* Inherits from {@link FormDataFile}.
*/
export interface FormDataMemoryFile extends FormDataFile {
/**
* The binary data of the file.
* This property is used for small-sized files that are stored entirely in memory.
* @example <Buffer 89 50 4e ...>
*/
content: Buffer;
}

/**
* Represents a large file stored in temporary storage.
* Inherits from {@link FormDataFile}.
*/
export interface FormDataStorageFile extends FormDataFile {
/**
* The path to the temporary file on disk.
* This property is used for large files that are stored in temporary storage.
* @example '/tmp/uploads/largefile.tmp'
*/
path?: string;
}

export interface HTMLScriptTag {
Expand Down
Loading