Skip to content

Conversation

m4drat
Copy link
Contributor

@m4drat m4drat commented Dec 27, 2022

Hi!

I've been fuzzing different pytorch modules, and found a few crashes.

Inside unpickler.cpp/irparser.cpp there are a few places, where .at() and .pop_back() are called before checking target container size. Lack of these checks results in an attempt to access elements oob (in case of .at()), and an actual out-of-bounds access while calling .pop_back()/.pop() on a stack_ variable.

Crash-files:

  1. Crash location: unpickler.cpp:439 (Call to .at(idx) with idx that exceeds memo_table_ size).

  2. Crash location: irparser.cpp:504 (Call to .at(idx) with idx that exceeds schema->returns() size).

  3. Crash location: unpickler.cpp:451 (Call to .pop_back() with empty stack_).

  4. Crash location: unpickler.cpp:469 (Call to .pop() with empty stack_).

The provided patch adds missing size checks.

How to reproduce

  1. To reproduce the crashes, use provided docker: Dockerfile

  2. Build the container: docker build -t oss-sydr-fuzz-pytorch-reproduce .

  3. Copy crash file to the current directory

  4. Run the container: docker run --privileged --network host -v `pwd`:/homedir --rm -it oss-sydr-fuzz-pytorch-reproduce /bin/bash

  5. And execute fuzz-targets with the given arguments

After execution completes you will see ASAN reports.

@pytorch-bot
Copy link

pytorch-bot bot commented Dec 27, 2022

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91401

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ba70472:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: jit release notes category label Dec 27, 2022
@bdhirsh
Copy link
Contributor

bdhirsh commented Dec 27, 2022

Thanks for the fix! This looks good to me, I'll let CI run

@davidberard98
Copy link
Contributor

@pytorchbot rebase -s

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased check-size-before-using-containers onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout check-size-before-using-containers && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the check-size-before-using-containers branch from 7497bd8 to d0cf2ee Compare December 28, 2022 05:15
@@ -501,6 +501,13 @@ void IRParser::parseOperator(Block* b) {
for (const VarWithType& v : outs) {
vmap[v.name] = n->outputs()[idx];
if (schema && !schema->is_varret()) {
TORCH_CHECK(
Copy link
Contributor

@davidberard98 davidberard98 Dec 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious, why do we need this check?

I understand the pop() issue... but doesn't .at() already check for out of bounds? (i.e. this is why you would use .at() instead of [])?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's true.
However, I thought about this issue from the user's point of view. I'd say it's mostly a quality-of-life improvement that helps to clearly show where and why the error occurs. There are many places, where using .at() is enough, however, I think in this context it's beneficial.

@davidberard98
Copy link
Contributor

@pytorchbot rebase -s

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased check-size-before-using-containers onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout check-size-before-using-containers && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the check-size-before-using-containers branch from d0cf2ee to b733091 Compare December 28, 2022 21:49
Copy link
Contributor

@davidberard98 davidberard98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@davidberard98
Copy link
Contributor

@pytorchbot rebase -s

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a rebase job. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased check-size-before-using-containers onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout check-size-before-using-containers && git pull --rebase)

@pytorchmergebot pytorchmergebot force-pushed the check-size-before-using-containers branch from b733091 to ba70472 Compare December 29, 2022 16:19
@davidberard98
Copy link
Contributor

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 29, 2022
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged open source release notes: jit release notes category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants