Feedback on Multipart

[dylemma]

Hi folks, and thanks for this great library.

Recently, I implemented a small webservice using Http4s and its Multipart support. I noticed the "we'd love the feedback" comment on the package object, so I figured I'd give some.

• As far as I can tell, the current decoder implementation for Multipart doesn't let you make any on-the-fly decisions that could cause parsing to abort. For example, if a part looks like a file upload but your endpoint isn't expecting any uploads; or if you're only expecting a file upload in a specific part. In these situations where your business logic will end up rejecting the request (responding with a a 40x), you end up wasting time if there are more parts with lots of bytes to receive before the decoder finishes.

• I also noticed the way Parts are represented is with a Stream[F, Byte], which gives rise to a double-buffering problem when your business logic actually wants to dump a Part's content to a temp file. If the size of the file is large enough, the underlying decoder logic will buffer the content to a temp file of its own, and then the business logic would copy that content to its own temp file. Ideally, when receiving the headers for a Part, the business logic could decide to buffer that Part's body to a temp file, and expose a Path to that temp file.

• There also doesn't seem to be a way to impose size limits on individual part bodies. I'd like to be able to "hang up" if for example someone tried to upload an overly-large file or an infinite stream of bytes.

• I also noticed that parseStreamedFile and parseToPartsStreamedFile are marked as deprecated, andthey recommend using parseSupervisedFile instead... but parseSupervisedFile is package-private.

• In my efforts to work around these issues, I found the MultipartParser.Event type and the MultipartParser.parseEvents pipe, which I could theoretically use to implement a new multipart decoder that addresses my above complaints, but those two things are currently private in http4s 0.23.25.

I've got a prototype implementation that uses Part[IO] => DecodeResult[Resource[IO, *], A] to represent on-the-fly decision making for Parts, and resource-allocation concerns for e.g. the buffer-to-temp-file case. It currently works in terms of an Event class that mirrors the one in MultipartParser, but for my testing sandbox, I'm just hard-coding a sequence of Events. I've also created a MultipartReceiver[F, A] trait which has an Applicative and a builder DSL.

It's not quite PR-ready, but I figured I'd show some highlights and see if there's interest in incorporating it into http4s-core, or prompt the the Events/parseEvents code to become public so I could reuse them in a library I publish myself.

// low-level implementation:
// Each part is consumed on-the-fly as a resource which is closed by the supervisor.
// If a part's decoder raises an error, the Event pull is aborted. There's no buffering
// in this Pipe unless the `partDecoder` decides to buffer.
def decodeMultipartEvents[A](
    supervisor: Supervisor[IO],
    partDecoder: Part[IO] => DecodeResult[Resource[IO, *], A],
): Pipe[IO, Event, Either[DecodeFailure, A]]

// eventually to be wrapped as e.g. a `Resource[F, EntityDecoder[A]]`
def decodeMultipart[A](
    partEvents: Stream[IO, Event],
    supervisor: Supervisor[IO],
    receiver: MultipartReceiver[IO, A],
): DecodeResult[IO, A]

// Convenience layer for the `partDecoder`, allowing you to make
// per-part decisions for parts based on their headers, e.g. whether
// to consume as a String or dump to a File, or to just ignore a part,
// or to immediately raise an error when an unexpected part is encountered
val receiver: MultipartReceiver[IO, (String, Path, Map[String, MultipartReceiver.FieldValue])] = (
    MultipartReceiver[IO].textPart("foo")(_.readString),
    MultipartReceiver[IO].filePart("file")(_.toTempFile.withSizeLimit(19)),
    MultipartReceiver[IO].auto,
).tupled

[dylemma]

Prototype here: https://github.com/dylemma/http4s-multipart