proposal: mime/multipart: add WriteFile method to mime/multipart.Writer for streamlined file uploads from io.Reader

[Azhovan]

Proposal Details

Abstract

This proposal suggests adding a new method, WriteFile, to the mime/multipart.Writer type in the Go standard library. The WriteFile method will simplify the process of adding file content to multipart forms directly from an io.Reader, enhancing the package's capability to handle file uploads efficiently, especially for large files or files sourced from streams.

Background

The current mime/multipart.Writer API provides methods like CreateFormFile and CreateFormField for adding files and fields to multipart forms, which are widely used in HTTP client and server implementations for file uploads. However, after calling CreateFormFile, developers must manually copy the file content into the returned writer, which can be cumbersome.

Proposal

I propose adding a new method to the mime/multipart.Writer type:

func (w *Writer) WriteFile(fieldname, filename string, src io.Reader) error {
    p, err := w.CreateFormFile(fieldname, filename)
    if err != nil {
        return err
    }
    _, err = io.Copy(p, src)
    return err
}

This method will perform the following tasks:

• Create a form file part with the provided field name and filename.
• Stream data from the provided io.Reader to the form file part.
• Simplify adding file content to multipart forms, especially when the content is already available as an io.Reader (e.g., file streams, network data).

Rationale

• Ease of Use: The WriteFile method abstracts away the boilerplate code associated with setting up form file parts and copying data, leading to cleaner and more concise client code.

• Efficiency: It's particularly beneficial for streaming large files or data from network sources, as it doesn't require loading the entire file content into memory.

• Consistency with Existing APIs: While the mime/multipart package already provides WriteField for simple text fields, there is no equivalent convenience method for files. WriteFile fills this gap and aligns with the existing API design.

Compatibility

This change is fully backward-compatible. It introduces a new method to the existing mime/multipart.Writer type without modifying any existing functionality. It will not affect existing codebases using the mime/multipart package.

Conclusion

Adding WriteFile to mime/multipart.Writer will enhance the Go standard library's ability to handle file uploads in a more efficient and user-friendly manner. This addition is a natural extension of the existing multipart API and will be beneficial for a wide range of applications that rely on file uploads, particularly those dealing with large or streamed files.

[Azhovan]

@bradfitz @ianlancetaylor
what do you think?

[jimwei]

[like] WeiYaPeng(Jim) reacted to your message:

…

________________________________ From: Jabar Asadi ***@***.***> Sent: Saturday, January 13, 2024 4:07:48 PM To: golang/go ***@***.***> Cc: Subscribed ***@***.***> Subject: [golang/go] proposal: mime/multipart: add WriteFile method to mime/multipart.Writer for streamlined file uploads from io.Reader (Issue #65096) Proposal Details Abstract This proposal suggests adding a new method, WriteFile, to the mime/multipart.Writer type in the Go standard library. The WriteFile method will simplify the process of adding file content to multipart forms directly from an io.Reader, enhancing the package's capability to handle file uploads efficiently, especially for large files or files sourced from streams. Background The current mime/multipart.Writer API provides methods like CreateFormFile and CreateFormField for adding files and fields to multipart forms, which are widely used in HTTP client and server implementations for file uploads. However, after calling CreateFormFile, developers must manually copy the file content into the returned writer, which can be cumbersome. Proposal I propose adding a new method to the mime/multipart.Writer type: func (w *Writer) WriteFile(fieldname, filename string, src io.Reader) error { p, err := w.CreateFormFile(fieldname, filename) if err != nil { return err } _, err = io.Copy(p, src) return err } This method will perform the following tasks: * Create a form file part with the provided field name and filename. * Stream data from the provided io.Reader to the form file part. * Simplify adding file content to multipart forms, especially when the content is already available as an io.Reader (e.g., file streams, network data). Rationale * Ease of Use: The WriteFile method abstracts away the boilerplate code associated with setting up form file parts and copying data, leading to cleaner and more concise client code. * Efficiency: It's particularly beneficial for streaming large files or data from network sources, as it doesn't require loading the entire file content into memory. * Consistency with Existing APIs: While the mime/multipart package already provides WriteField for simple text fields, there is no equivalent convenience method for files. WriteFile fills this gap and aligns with the existing API design. Compatibility This change is fully backward-compatible. It introduces a new method to the existing mime/multipart.Writer type without modifying any existing functionality. It will not affect existing codebases using the mime/multipart package. Conclusion Adding WriteFile to mime/multipart.Writer will enhance the Go standard library's ability to handle file uploads in a more efficient and user-friendly manner. This addition is a natural extension of the existing multipart API and will be beneficial for a wide range of applications that rely on file uploads, particularly those dealing with large or streamed files. — Reply to this email directly, view it on GitHub<#65096>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAKTSGKQFU2UDZIIMHHUFIDYOKWNJAVCNFSM6AAAAABBZND7XSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4DAMZWGMYTGOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ianlancetaylor]

Thanks. It would help to point to some existing code that would benefit from this method.

[seankhliao]

Is this just to save 4 lines of code?

[Azhovan]

@ianlancetaylor
Absolutely, following are some examples with or without using this feature.

Example 1: HTTP Handler
This example is a Go HTTP handler than processes a user-uploaded image:
Current approach without WriteFile method, the code might look like this:

http.HandleFunc("/upload-profile-picture", func(w http.ResponseWriter, r *http.Request) {
    r.ParseMultipartForm(10 << 20) // 10 MB limit
    file, header, err := r.FormFile("profilePicture")
    if err != nil {
        fmt.Fprintf(w, "File upload error: %v", err)
        return
    }
    defer file.Close()

    var buf bytes.Buffer
    writer := multipart.NewWriter(&buf)
    part, err := writer.CreateFormFile("profilePicture", header.Filename)
    if err != nil {
        fmt.Fprintf(w, "Error creating form file: %v", err)
        return
    }
    _, err = io.Copy(part, file)
    if err != nil {
        fmt.Fprintf(w, "Error copying file: %v", err)
        return
    }
    // Further processing...
})

Improved approach with WriteFile, the process simplifies to:

http.HandleFunc("/upload-profile-picture", func(w http.ResponseWriter, r *http.Request) {
    r.ParseMultipartForm(10 << 20) // 10 MB limit
    file, header, err := r.FormFile("profilePicture")
    if err != nil {
        fmt.Fprintf(w, "File upload error: %v", err)
        return
    }
    defer file.Close()

    var buf bytes.Buffer
    writer := multipart.NewWriter(&buf)
    if err := writer.WriteFile("profilePicture", header.Filename, file); err != nil {
        fmt.Fprintf(w, "Error writing file: %v", err)
        return
    }
    // Further processing...
})

Example 2: Batch Processing of Uploads
This example is a scenario where your application needs to upload several files

Current approach without WriteFile, looping through files:

var buffer bytes.Buffer
writer := multipart.NewWriter(&buffer)

for _, file := range files { // Assuming 'files' is a slice of file streams
    part, err := writer.CreateFormFile("file", file.Name)
    if err != nil {
        log.Fatal(err)
    }
    _, err = io.Copy(part, file.Stream)
    if err != nil {
        log.Fatal(err)
    }
}
writer.Close()

// Buffer now contains all the files in multipart form, ready for batch upload.

Improved approach with WriteFile, looping through files:

var buffer bytes.Buffer
writer := multipart.NewWriter(&buffer)

for _, file := range files { // Assuming 'files' is a slice of file streams with Name and Stream properties
    err := writer.WriteFile("file", file.Name, file.Stream)
    if err != nil {
        log.Fatal(err)
    }
}
writer.Close()

// The process is simplified, focusing on what matters: preparing the files for upload.

As you see the WriteFile method provides a clear, concise way to handle file uploads, reducing the potential for errors and focusing developer efforts on the core functionality of their app.

[Azhovan]

Is this just to save 4 lines of code?

@seankhliao

Thank you for your comment. On the surface, the addition of the WriteFile method may seem to primarily offer a reduction in the amount of code developers need to write. While simplifying code by a few lines is a benefit, the proposal's value extends beyond just brevity:

Error Handling: Centralizing the file-writing process into a single method allows for more consistent and potentially sophisticated error handling(just like in my examples abve). It reduces the risk of common mistakes made during manual copying and error checks.

Consistency and Readability: The proposal also hinges on creating an API that is consistent and easy-to-understand. Having a WriteFile method aligns well with the API's current structure which already includes methods like WriteField.

[seankhliao]

Given how trivial this is, it can be written as a helper function where it is used.
https://go.dev/doc/faq#x_in_std