Fix handling of multipart/form-data

[Dreamsorcerer]

https://datatracker.ietf.org/doc/html/rfc7578

[bdraco]

I started digging into this one

Unrelated but it looks there is blocking I/O in the event loop here:

aiohttp/aiohttp/web_request.py

Line 727 in 742a7d9

tmp = tempfile.TemporaryFile()

aiohttp/aiohttp/web_request.py

Line 731 in 742a7d9

tmp.write(chunk)

aiohttp/aiohttp/web_request.py

Line 739 in 742a7d9

tmp.seek(0)

#8283

[bdraco]

Its a bit unexpected that any formdata that is bytes is treated as a file

aiohttp/aiohttp/formdata.py

Line 56 in 742a7d9

elif isinstance(value, (bytes, bytearray, memoryview)):

[Dreamsorcerer]

Its a bit unexpected that any formdata that is bytes is treated as a file

That was my initial thought too. It might be a convenience if you avoid bytes for typical use, but a little confusing otherwise. Should we switch v4 to only create files when filename is passed or similar? We could use a sentinel so that filename=None would still create a file?

[codecov]

Codecov Report

Attention: Patch coverage is 93.05556% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 97.54%. Comparing base (cbc0c86) to head (c02cab6).

Files	Patch %	Lines
aiohttp/multipart.py	91.11%	2 Missing and 2 partials ⚠️
aiohttp/formdata.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8280      +/-   ##
==========================================
+ Coverage   97.50%   97.54%   +0.04%     
==========================================
  Files         107      107              
  Lines       33026    33024       -2     
  Branches     3862     3872      +10     
==========================================
+ Hits        32201    32213      +12     
+ Misses        605      591      -14     
  Partials      220      220              

Flag	Coverage Δ
CI-GHA	97.45% <93.05%> (+0.03%)	⬆️
OS-Linux	97.12% <93.05%> (+0.03%)	⬆️
OS-Windows	95.61% <93.05%> (-0.03%)	⬇️
OS-macOS	96.83% <93.05%> (-0.03%)	⬇️
Py-3.10.11	95.46% <93.05%> (-0.03%)	⬇️
Py-3.10.14	96.92% <93.05%> (-0.03%)	⬇️
Py-3.11.8	97.12% <93.05%> (-0.03%)	⬇️
Py-3.12.2	97.24% <93.05%> (-0.03%)	⬇️
Py-3.8.10	95.39% <93.05%> (-0.03%)	⬇️
Py-3.8.18	96.77% <93.05%> (-0.03%)	⬇️
Py-3.9.13	95.43% <93.05%> (-0.03%)	⬇️
Py-3.9.19	96.90% <93.05%> (-0.02%)	⬇️
Py-pypy7.3.15	96.43% <93.05%> (?)
VM-macos	96.83% <93.05%> (-0.03%)	⬇️
VM-ubuntu	97.12% <93.05%> (+0.03%)	⬆️
VM-windows	95.61% <93.05%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Dreamsorcerer]

OK, I think this should be fine now. Will need a followup PR to do the actual removal in master.

[bdraco]

I'll be back home in an hour and can throw this on production to make sure there's no unexpected side effects

[bdraco]

The backport to 3.9 is messy because #3492 isn't in 3.9

[bdraco]

I squashed this, cherry-picked fcedc66, than cherry-picked this on to my 3.9 integration branch.

Testing that now

[bdraco]

Started up OK, and file uploads work. Watching the logs now.

[bdraco]

Nothing obvious showing up in the logs. Everything I could think to test still works

[Dreamsorcerer]

The backport to 3.9 is messy because #3492 isn't in 3.9

Guess we just backport that first then? Clearly got missed from any backports.

[bdraco]

The backport to 3.9 is messy because #3492 isn't in 3.9

Guess we just backport that first then? Clearly got missed from any backports.

While I'm not a big fan of the complexity added by #3492, given that PR is already accepted and merged, and this fix is developed on top of it, that seems like the most reasonable path forward.

[Dreamsorcerer]

The backport to 3.9 is messy because #3492 isn't in 3.9

Guess we just backport that first then? Clearly got missed from any backports.

While I'm not a big fan of the complexity added by #3492, given that PR is already accepted and merged, and this fix is developed on top of it, that seems like the most reasonable path forward.

Wait, is this multipart reader server-side or client-side? If it's server-side then the change will likely introduce another HTTP request smuggling vulnerability. If it's only server-side, then we'd probably be best reverting the commit. If it's both, then we probably want to backport it and add a followup PR to disable it when the parser is not in lax mode...

[Dreamsorcerer]

The backport to 3.9 is messy because #3492 isn't in 3.9

Guess we just backport that first then? Clearly got missed from any backports.

While I'm not a big fan of the complexity added by #3492, given that PR is already accepted and merged, and this fix is developed on top of it, that seems like the most reasonable path forward.

Wait, is this multipart reader server-side or client-side? If it's server-side then the change will likely introduce another HTTP request smuggling vulnerability. If it's only server-side, then we'd probably be best reverting the commit. If it's both, then we probably want to backport it and add a followup PR to disable it when the parser is not in lax mode...

I think this is server-side only, I think the best option right now is to revert that change as nobody seems to have missed the feature, given that it failed to make it into a release for 5 years.

[Dreamsorcerer]

There are a couple of lines of coverage dropped in multipart.py:
https://app.codecov.io/gh/aio-libs/aiohttp/pull/8280/indirect-changes

I'm unclear how to get that covered again, or whether the code should maybe be deleted now. If anyone figures it out by tomorrow, feel free to just edit the PR directly.

[patchback]

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 7d0be3f on top of patchback/backports/3.9/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280

Backporting merged PR #8280 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.9/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280 upstream/3.9

4. Now, cherry-pick PR Fix handling of multipart/form-data #8280 contents into that branch:

   $ git cherry-pick -x 7d0be3fee540a3d4161ac7dc76422f1f5ea60104

   If it'll yell at you with something like fatal: Commit 7d0be3fee540a3d4161ac7dc76422f1f5ea60104 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 7d0be3fee540a3d4161ac7dc76422f1f5ea60104

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Fix handling of multipart/form-data #8280 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.9/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[patchback]

Backport to 3.10: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 7d0be3f on top of patchback/backports/3.10/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280

Backporting merged PR #8280 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.10/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280 upstream/3.10

4. Now, cherry-pick PR Fix handling of multipart/form-data #8280 contents into that branch:

   $ git cherry-pick -x 7d0be3fee540a3d4161ac7dc76422f1f5ea60104

   If it'll yell at you with something like fatal: Commit 7d0be3fee540a3d4161ac7dc76422f1f5ea60104 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 7d0be3fee540a3d4161ac7dc76422f1f5ea60104

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Fix handling of multipart/form-data #8280 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.10/7d0be3fee540a3d4161ac7dc76422f1f5ea60104/pr-8280

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.