Skip to content

Commit

Permalink
feat(blob): multipart upload (nuxt-hub#71)
Browse files Browse the repository at this point in the history
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Sébastien Chopin <seb@nuxtlabs.com>
Co-authored-by: Sébastien Chopin <seb@nuxt.com>
Co-authored-by: Farnabaz <farnabaz@gmail.com>
  • Loading branch information
5 people committed May 31, 2024
1 parent 8242385 commit bb685d1
Show file tree
Hide file tree
Showing 13 changed files with 1,066 additions and 28 deletions.
324 changes: 323 additions & 1 deletion docs/content/docs/2.storage/3.blob.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,32 @@ export default eventHandler(async (event) => {
})
```

See an example on the Vue side:

```vue [pages/upload.vue]
<script setup lang="ts">
async function uploadImage (e: Event) {
const form = e.target as HTMLFormElement
await $fetch('/api/files', {
method: 'POST',
body: new FormData(form)
}).catch((err) => alert('Failed to upload image:\n'+ err.data?.message))
form.reset()
}
</script>
<template>
<form @submit.prevent="uploadImage">
<label>Upload an image: <input type="file" name="image"></label>
<button type="submit">
Upload
</button>
</form>
</template>
```

#### Params

::field-group
Expand Down Expand Up @@ -248,6 +274,212 @@ Returns a [`BlobObject`](#blobobject) or an array of [`BlobObject`](#blobobject)

Throws an error if `file` doesn't meet the requirements.

### `handleMultipartUpload()`

Handle the request to support multipart upload.

```ts [server/api/files/multipart/[action\\]/[...pathname\\].ts]
export default eventHandler(async (event) => {
return await hubBlob().handleMultipartUpload(event)
})
```

::important
Make sure your route includes `[action]` and `[...pathname]` params.
::

On the client side, you can use the `useMultipartUpload()` composable to upload a file in parts.

```vue
<script setup lang="ts">
async function uploadFile(file: File) {
const upload = useMultipartUpload('/api/files/multipart')
const { progress, completed, abort } = upload(file)
}
</script>
```

::note{to="#usemultipartupload"}
See [`useMultipartUpload()`](#usemultipartupload) on usage details.
::

#### Params

::field-group
::field{name="contentType" type="string"}
The content type of the blob.
::
::field{name="contentLength" type="string"}
The content length of the blob.
::
::field{name="addRandomSuffix" type="boolean"}
If `true`, a random suffix will be added to the blob's name. Defaults to `false`.
::
::

### `createMultipartUpload()`

::note
We suggest to use [`handleMultipartUpload()`](#handlemultipartupload) method to handle the multipart upload request.
::

Start a new multipart upload.

```ts [server/api/files/multipart/[...pathname\\].post.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)

const mpu = await hubBlob().createMultipartUpload(pathname)

return {
uploadId: mpu.uploadId,
pathname: mpu.pathname,
}
})
```

#### Params

::field-group
::field{name="pathname" type="String"}
The name of the blob to serve.
::
::field{name="options" type="Object"}
The put options. Any other provided field will be stored in the blob's metadata.
::field{name="contentType" type="String"}
The content type of the blob. If not given, it will be inferred from the Blob or the file extension.
::
::field{name="contentLength" type="String"}
The content length of the blob.
::
::field{name="addRandomSuffix" type="Boolean"}
If `true`, a random suffix will be added to the blob's name. Defaults to `true`.
::
::
::

#### Return

Returns a `BlobMultipartUpload`

### `resumeMultipartUpload()`

::note
We suggest to use [`handleMultipartUpload()`](#handlemultipartupload) method to handle the multipart upload request.
::

Continue processing of unfinished multipart upload.

To upload a part of the multipart upload, you can use the `uploadPart()` method:

```ts [server/api/files/multipart/[...pathname\\].put.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)
const { uploadId, partNumber } = getQuery(event)

const stream = getRequestWebStream(event)!
const body = await streamToArrayBuffer(stream, contentLength)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
return await mpu.uploadPart(partNumber, body)
})
```

Complete the upload by calling `complete()` method:

```ts [server/api/files/multipart/complete.post.ts]
export default eventHandler(async (event) => {
const { pathname, uploadId } = getQuery(event)
const parts = await readBody(event)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
return await mpu.complete(parts)
})
```

If you want to cancel the upload, you need to call `abort()` method:

```ts [server/api/files/multipart/[...pathname\\].delete.ts]
export default eventHandler(async (event) => {
const { pathname } = getRouterParams(event)
const { uploadId } = getQuery(event)

const mpu = hubBlob().resumeMultipartUpload(pathname, uploadId)
await mpu.abort()

return sendNoContent(event)
})
```

A simple example of multipart upload in client with above routes:

```ts [utils/multipart-upload.ts]
async function uploadLargeFile(file: File) {
const chunkSize = 10 * 1024 * 1024 // 10MB

const count = Math.ceil(file.size / chunkSize)
const { pathname, uploadId } = await $fetch(
`/api/files/multipart/${file.name}`,
{ method: 'POST' },
)

const uploaded = []

for (let i = 0; i < count; i++) {
const start = i * chunkSize
const end = Math.min(start + chunkSize, file.size)
const partNumber = i + 1
const chunk = file.slice(start, end)

const part = await $fetch(
`/api/files/multipart/${pathname}`,
{
method: 'PUT',
query: { uploadId, partNumber },
body: chunk,
},
)

uploaded.push(part)
}

return await $fetch(
'/api/files/multipart/complete',
{
method: 'POST',
query: { pathname, uploadId },
body: { parts: uploaded },
},
)
}
```

#### Params

::field-group
::field{name="pathname" type="String"}
The name of the blob to serve.
::
::field{name="uploadId" type="String"}
The upload ID of the multipart upload.
::
::

#### Return

Returns a `BlobMultipartUpload`


#### Params

::field-group
::field{name="event" type="H3Event" required}
The event to handle.
::
::


## `ensureBlob()`

`ensureBlob()` is a handy util to validate a `Blob` by checking its size and type:
Expand Down Expand Up @@ -284,6 +516,10 @@ Throws an error if `file` doesn't meet the requirements.

## Composables

::note
The following composables are meant to be used in the Vue side of your application (not the `server/` directory).
::

### `useUpload()`

`useUpload` is to handle file uploads in your Nuxt application.
Expand Down Expand Up @@ -327,6 +563,59 @@ async function onFileSelect({ target }: Event) {
::
::

#### Return

Return a `MultipartUpload` function that can be used to upload a file in parts.

```ts
const { completed, progress, abort } = upload(file)
const data = await completed
```

### `useMultipartUpload()`

Application composable that creates a multipart upload helper.

```ts [utils/multipart-upload.ts]
export const mpu = useMultipartUpload('/api/files/multipart')
```

#### Params

::field-group
::field{name="baseURL" type="string"}
The base URL of the multipart upload API handled by [`handleMultipartUpload()`](#handlemultipartupload).
::
::field{name="options"}
The options for the multipart upload helper.
::field{name="partSize" type="number"}
The size of each part of the file to be uploaded. Defaults to `10MB`.
::
::field{name="concurrent" type="number"}
The maximum number of concurrent uploads. Defaults to `1`.
::
::field{name="maxRetry" type="number"}
The maximum number of retry attempts for the whole upload. Defaults to `3`.
::
::field{name="prefix" type="string"}
The prefix to use for the blob pathname.
::
::field{name="fetchOptions" type="Omit<FetchOptions, 'method' | 'baseURL' | 'body' | 'parseResponse' | 'responseType'>"}
Override the ofetch options.
The `query` and `headers` will be merged with the options provided by the uploader.
::
::
::

#### Return

Return a `MultipartUpload` function that can be used to upload a file in parts.

```ts
const { completed, progress, abort } = mpu(file)
const data = await completed
```

## Types

### `BlobObject`
Expand All @@ -340,6 +629,40 @@ interface BlobObject {
}
```

### `BlobMultipartUpload`

```ts
export interface BlobMultipartUpload {
pathname: string
uploadId: string
uploadPart(
partNumber: number,
value: string | ReadableStream<any> | ArrayBuffer | ArrayBufferView | Blob
): Promise<BlobUploadedPart>
abort(): Promise<void>
complete(uploadedParts: BlobUploadedPart[]): Promise<BlobObject>
}
```

### `BlobUploadedPart`

```ts
export interface BlobUploadedPart {
partNumber: number;
etag: string;
}
```

### `MultipartUploader`

```ts
export type MultipartUploader = (file: File) => {
completed: Promise<SerializeObject<BlobObject> | undefined>
progress: Readonly<Ref<number>>
abort: () => Promise<void>
}
```
### `BlobListResult`
```ts
Expand All @@ -351,7 +674,6 @@ interface BlobListResult {
}
```


## Examples

### List blobs with pagination
Expand Down
Loading

0 comments on commit bb685d1

Please sign in to comment.