Issue with Azure Functions v4 FormData Handling for Large field JSON Data

[Dev-InSoft]

Issue with Azure Functions v4 FormData Handling for Large field JSON Data

I hope this message finds you well. I am writing to report an issue encountered after upgrading from Azure Functions version 3.1.0 to version 4.x, specifically related to handling FormData in TypeScript.

In my previous implementation using Azure Functions version 3.1.0, I utilized the npm package @anzp/azure-function-multipart to process FormData, with a specific focus on handling a large JSON payload along with other file types. I set a substantial field size limit using the 'fieldSize' property, ensuring proper processing of large JSON data.

let ObjParse: ParsedMultipartFormData = await parseMultipartFormData(request, { limits: { fieldSize: 999999999 } });

However, upon upgrading to Azure Functions version 4.x, I discovered that this npm package is no longer compatible, as it specifically supports version 3.1.x. In the new version, when utilizing

let ObjParse = await request.formData()

the JSON data is received incomplete, containing only 1048570 characters, even though the actual size of the JSON is 1536897 characters.

This issue severely impacts the functionality of my Azure Function, as it is crucial to process complete JSON data within the FormData.

I kindly request your assistance in addressing this matter. If there are alternative approaches or recommended updates to handle FormData with large JSON payloads in Azure Functions v4.x, I would greatly appreciate your guidance.

Thank you for your attention to this issue, and I look forward to a prompt resolution.

[ejizba]

Hi @Dev-InSoft I'm not familiar with @anzp/azure-function-multipart, but yes request.formData() is the recommended way to parse form data. Could you perhaps give more detailed repro steps with sample data? Any idea what characters are missing? For example is it from the end of the data or is it maybe a difference in whitespace?

The biggest problem we have with form data right now is we don't support streaming http requests. I don't know if this is your problem, but if you are sending the request with "transfer-encoding" set to "chunked", that could explain it. We hope to support streams soon (tracked by #97).

[Dev-InSoft]

In the following example, it sent a JSON that has a size of 1,197,284 bytes, but when the information arrives at the server, it indicates that the size of the “jsondatafield” field is only 1,048,576 bytes, evidencing that it trims it.

import { app, HttpRequest, HttpResponseInit, InvocationContext } from "@azure/functions";
import type { FormData, FormDataEntryValue } from "undici"

export async function Test(request: HttpRequest, context: InvocationContext): Promise<HttpResponseInit> {
	try {
		//let text = await request.text();
		let formData: FormData = await request.formData();
		let data: FormDataEntryValue | null = formData.get("jsondatafield");
		if (data && typeof data == "string")
			return { jsonBody: JSON.parse(data) }
		return { status: 400, body: "bad requet" }
	} catch (error) {
		return { status: 500, body: error?.message }
	}
};

app.post('Test', {
	handler: Test
});

If I read the Body information with await request.text() I notice that the JSON is complete, and that it is truncated by await request.formData()

In the same example, if I sent a JSON with a size less than 1,048,576, as in the following case, I am sending one of 1040640 bytes, the reading is normal and there are no problems in the execution of the function

Below I attach an example in postman and a mini project in which the case can be reproduced
Azure Function V4.postman.json
Example.zip

[ejizba]

So it turns out this default limit of "1048576" is set by the busboy npm package. Your original package "@anzp/azure-function-multipart" and "undici" are both using "busboy" as a dependency, but it seems like only the former supports passing the config on to busboy. My current recommendation would be to use "busboy" directly if you need to configure the limits. The other alternative is to file an issue on undici so that we can configure the limits, but the fetch spec for formData doesn't allow any parameters so I'm skeptical we'd get traction.

Here's some rough code that worked for me. It's just a combination of your code and the busboy sample in their readme:

import { app, HttpRequest, HttpResponseInit, InvocationContext } from "@azure/functions";
import * as busboy from 'busboy';

export async function Test(request: HttpRequest, context: InvocationContext): Promise<HttpResponseInit> {
    try {
        console.log('POST request');

        const formData = {};
        const bb = busboy({ headers: Object.fromEntries(request.headers), limits: { fieldSize: 999999999 } });
        bb.on('field', (name, val, info) => {
            formData[name] = val;
        });
        bb.on('close', () => {
            console.log('Done parsing form!');
        });

        const body = await request.text();
        bb.write(body);
        bb.end();

        let data = formData["jsondatafield"];
        if (data && typeof data == "string") {
            console.log(`Data length: ${data.length}`);
            return { jsonBody: JSON.parse(data) }
        }
        return { status: 400, body: "bad request" }
    } catch (error) {
        return { status: 500, body: error?.message }
    }
};

app.post('Test', {
    handler: Test
});

[Dev-InSoft]

Ok, thank you, I understand then that the problem is due to the use found in "undici", which is why we have decided not to work with the request function:

await request.formData()

But rather with busBoy but working the same with “undici” data types such as "FormData", since in the same function we receive files and fields

import { app, HttpRequest, HttpResponseInit, InvocationContext } from "@azure/functions";
import * as busboy from 'busboy';
import { FormData, FormDataEntryValue } from "undici";
import type { Readable } from "stream";
import type { IncomingHttpHeaders } from "http";

function parseFormData(body: Uint8Array, headers: IncomingHttpHeaders): Promise<FormData> {
	return new Promise<FormData>((resolve, reject) => {
		try {
			let result: FormData = new FormData();
			let files: { [key: string]: Buffer } = {}
			let bb: busboy.Busboy = busboy({ headers, limits: { fieldSize: Number.MAX_SAFE_INTEGER } });
			bb.on('field', (name: string, value: string) => result.append(name, value));
			bb.on('file', (name: string, file: Readable, info: busboy.FileInfo) => {
				if (!files[name]) files[name] = Buffer.from([]);
				file.on('data', (data) => files[name] = Buffer.concat([files[name], data]))
				file.on('end', () => result.append(name, new File([files[name]], info.filename, { type: info.mimeType })))
			})
			bb.on('close', () => resolve(result));
			bb.on('error', (error: unknown) => reject(error))
			bb.write(body);
			bb.end();
		} catch (error) {
			reject(error)
		}
	})
}

export async function Test(request: HttpRequest, context: InvocationContext): Promise<HttpResponseInit> {
	try {
		const formData = await parseFormData(new Uint8Array(await request.arrayBuffer()), Object.fromEntries(request.headers))
		let data: FormDataEntryValue | null = formData.get("jsondatafield")// formData.get("jsondatafield");
		if (data && typeof data == "string")
			return { jsonBody: JSON.parse(data) }
		return { status: 400, body: "bad requet" }
	} catch (error) {
		return { status: 500, body: error?.message }
	}
};

app.post('Test', {
	handler: Test
});