feat(Body): Added support for BodyMixin.formData() and constructing bodies with FormData

[jimmywarting]

Purpose

• Add support for both response.formData() and request.formData()
• Support constructing not only new Request(url, {body: formData}) but also new Response(formData) as it's supposed to be (only supported Request before...)
• It's also able to convert a URLSearchParam to a formData as it dose in browser/spec

This would be a great addition so you can avoid having to depend on some external multiparty, formidable, busboy or any other body parser packages. that all have some different (non-speced) apis that you would need to learn.
Instead you could just do:

import { Response } from 'node-fetch'

http.createServer(async function (req, res) {
  const formData = await new Response(req, { headers: req.headers }).formData()
  const file = formData.get('avatar')
  
  var arrayBuffer = await file.arrayBuffer()
  var text = await file.text()
  var whatwgReadableStream = file.stream()
}) 

Changes

• I changed the FormData test file to test more end to end instead.
• Added a fast algoritm to parse formData

Additional information

Maybe thinking it would be nice to expose FormData, Blob and File from index.js so they are accessible, thoughts?
Think we have had this discussion before with exposing Blob before... #392

---

• I updated ./docs/CHANGELOG.md
• I updated readme
• I added unit test(s)

---

closes #199
closes #1118
closes #1293
closes #876

[jimmywarting]

Ready to be reviewed...

[LinusU]

Sorry for the dragged out review, I've been swamped lately...

Personally, there are a few things that I would like to clean up more, but I could send those in a follow up PR if you would prefer that? As I understood it this code was copied from another package, do you have a link to that package? (e.g. I think that the mark/clear functions are quite hard to follow, and could be done in a better way)

The only thing that has me a bit weary right now is the return statements inside the for loop. I really think that this parsing will return garbage data back if the response is malformed. Personally, I think it's better to throw in those cases, so that no one starts relying on improper parsing which we might break in a future update. That being said, since you did say that you wanted to leave it as is I won't push it further 😅

Would you be open to me addressing that in a follow up PR, after testing what the browser does, or do you want that behaviour?

Apart from that, great PR! Nice to get this in 👍

---

Also, I'm also maintaining Multer which parses multipart data, and I've been interested in switching out the parser to a bit more modern/simpler one. What do you think about keeping this code in a separate package that both node-fetch and multer could depend on?

[gr2m]

What do you think about keeping this code in a separate package that both node-fetch and multer could depend on?

I think that'd be great, as long as it doesn't conflict with the standard compatibility of fetch

[jimmywarting]

Sorry for the dragged out review, I've been swamped lately...

Personally, there are a few things that I would like to clean up more, but I could send those in a follow up PR if you would prefer that? As I understood it this code was copied from another package, do you have a link to that package? (e.g. I think that the mark/clear functions are quite hard to follow, and could be done in a better way)

The code is from an older version of formidable that i ported to https://github.com/github/fetch a while back (to be more browser friendlier (without the need of having node:streams but got rejected due to the fact that there was no streaming method in xhr)

Here is theirs new and updated code: https://github.com/node-formidable/formidable/blob/master/src/parsers/Multipart.js

Formidable have been tested with a variety of different multipart uploads and it seems pretty speedy at parsing the payload.
If we keep changing it then i think we should be bringing in theirs test cases as well... I do not know how spec compatible it is.

This is the official spec: https://andreubotella.com/multipart-form-data

The only thing that has me a bit weary right now is the return statements inside the for loop. I really think that this parsing will return garbage data back if the response is malformed. Personally, I think it's better to throw in those cases, so that no one starts relying on improper parsing which we might break in a future update. That being said, since you did say that you wanted to leave it as is I won't push it further 😅

Would you be open to me addressing that in a follow up PR, after testing what the browser does, or do you want that behaviour?

Sure, As long as the body can be splittet up in chunks (anywhere) and parse different amount of bytes (even if it's splited in the middle of a boundary) and be streamable, then i'm fine with it.

It would also be nice if we could write very large files to the disk instead of keeping it in the memory also. and then use our blobs blobFrom(path, [mimetype]) later on.

---

Also, I'm also maintaining Multer which parses multipart data, and I've been interested in switching out the parser to a bit more modern/simpler one. What do you think about keeping this code in a separate package that both node-fetch and multer could depend on?

I think this would be a grate fit for the stream-consumers it's mostly a straight out copy from undici BodyMixin that made it into node.js core. it already uses fetch-blob and is also written in ESM + that it's new so not so many breaking changes for some ppl
It's something i have been wanting to bring into node-fetch anyway: #1252

It dose not have formData doe since FormData isn't implemented in NodeJs but it could be one day? there is issues about bringing in FormData into node
Also saw that @octet-stream linked undici to this PR: nodejs/undici#974

Personally i would like to remove the .buffer() method in my stream consumer package also since i think it should be discouraged to use it anywhere outside of NodeJS. you can just as easily write arrayBuffer(iterable).then(Buffer.from) if you really need a node:buffer

[LinusU]

Formidable have been tested with a variety of different multipart uploads and it seems pretty speedy at parsing the payload. If we keep changing it then i think we should be bringing in theirs test cases as well... I do not know how spec compatible it is.

I think that we are already working in a different way in the specific return statements that I talked about, because their write methods returns the number of bytes consumed, whereas our version does. So that's why I'm wary about that specific case.

Porting over the tests would be great, also adding some test case that we run both against our parser and the browser equivalent...

Also, I'm also maintaining Multer which parses multipart data, and I've been interested in switching out the parser to a bit more modern/simpler one. What do you think about keeping this code in a separate package that both node-fetch and multer could depend on?

I think this would be a grate fit for the stream-consumers it's mostly a straight out copy from undici BodyMixin that made it into node.js core. it already uses fetch-blob and is also written in ESM + that it's new so not so many breaking changes for some ppl It's something i have been wanting to bring into node-fetch anyway: #1252

It dose not have formData doe since FormData isn't implemented in NodeJs but it could be one day? there is issues about bringing in FormData into node Also saw that @octet-stream linked undici to this PR: nodejs/undici#974

For Multer I would probably need something a bit more low level, since we are doing field and size validation on the fly, and also stream the files to temporary files on disk 🤔

Depends on how the API would look of course!

Personally i would like to remove the .buffer() method in my stream consumer package also since i think it should be discouraged to use it anywhere outside of NodeJS. you can just as easily write arrayBuffer(iterable).then(Buffer.from) if you really need a node:buffer

👍

[octet-stream]

I was kind of thinking to make a low-level multipart parser as part of my form-data-encoder that could've basically get an AsyncIterable<Uint8Array> as the input and give API to attach callback to get parts once they parsed. I didn't come up with the look of that API yet, and not yet sure if I could make thing like that (never done any complex parser before), but it still may be implemented there at some point in a future.

[jimmywarting]

Personally i would like to remove the .buffer() method in my stream consumer package also since i think it should be discouraged to use it anywhere outside of NodeJS. you can just as easily write arrayBuffer(iterable).then(Buffer.from) if you really need a node:buffer

👍

fyi, I already went ahead and removed it... (put a warning sign on it)

all fixed

[LinusU]

@jimmywarting seems to be a conflict in package.json, could you fix that?

@Richienb would you be able to review this? ☺️

[jacob-ebey]

Oh this is exciting. Can't wait to see this merged.