Fix for MPS regression in #122016 and #123178 #123234

jhavukainen · 2024-04-03T01:34:33Z

Fixes #122016 and #123178. This regression is related to an OS side change that requires a slight adjustment from us on PyTorch side to restore the previous behavior. Additionally we cleared out pre-MacOS13 related workarounds.

Before the fix on MacOS 14.4:

python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)"
tensor([0., 3., 3.], device='mps:0')

After the fix:

python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)"
tensor([0., 1., 3.], device='mps:0')

This also fixes complex number initialization and as such makes nn.functional.rms_norm pass on MacOS-14+

pytorch-bot · 2024-04-03T01:34:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/123234

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a3a2acf with merge base 74b3a79 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jhavukainen · 2024-04-03T01:35:10Z

@kulinseth @DenisVieriu97

malfet

If it passes CI, then looks good to me

malfet · 2024-04-03T17:10:27Z

@pytorchbot rebase

malfet · 2024-04-03T17:10:46Z

(Lint failures are likely due to a broken base, but let's rebase to be on the safe ground)

pytorchmergebot · 2024-04-03T17:12:06Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-04-03T17:12:11Z

Successfully rebased dev/joona/Issue122016 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout dev/joona/Issue122016 && git pull --rebase)

aten/src/ATen/native/mps/operations/ConstantOps.mm

malfet · 2024-04-03T19:57:53Z

@pytorchbot r -f "Lint and MPS tests are green"

pytorch-bot · 2024-04-03T19:57:56Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: argument command: invalid choice: 'r' (choose from 'merge', 'revert', 'rebase', 'label', 'drci', 'cherry-pick', 'close')

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try @pytorchbot --help for more info.

malfet · 2024-04-03T20:13:04Z

@pytorchbot merge -f "Lint and MPS tests are green"

malfet · 2024-04-03T22:59:15Z

@pytorchbot merge -f "Lint and MPS tests are green"

pytorchmergebot · 2024-04-03T23:00:51Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2024-04-04T14:34:40Z

@jhavukainen have you run any benchmarks to measure perf implication of this change? I.e. it used to be a purely async operation but now it will pause as scalar needs to be copied to MPS before it can continue, wouldn't it?

malfet · 2024-04-04T21:25:18Z

@pytorchbot cherry-pick --onto release/2.3 --fixes foo -c criticalfix

pytorch-bot · 2024-04-04T21:25:20Z

❌ 🤖 pytorchbot command failed:

@pytorchbot cherry-pick: error: argument -c/--classification: invalid choice: 'criticalfix' (choose from 'regression', 'critical', 'fixnewfeature', 'docs', 'release')

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Try @pytorchbot --help for more info.

malfet · 2024-04-04T21:25:50Z

@pytorchbot cherry-pick --onto release/2.3 --fixes foo -c critical

Fixes #122016 and #123178. This regression is related to an OS side change that requires a slight adjustment from us on PyTorch side to restore the previous behavior. Additionally we cleared out pre-MacOS13 related workarounds. Before the fix on MacOS 14.4: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 3., 3.], device='mps:0') ``` After the fix: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 1., 3.], device='mps:0') ``` This also fixes complex number initialization and as such makes `nn.functional.rms_norm` pass on MacOS-14+ Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Pull Request resolved: #123234 Approved by: https://github.com/malfet, https://github.com/kulinseth (cherry picked from commit 05289a2)

pytorchbot · 2024-04-04T21:29:48Z

Cherry picking #123234

The cherry pick PR is at #123385 and it is linked with issue foo

Details for Dev Infra team

Raised by workflow job

Fixes #122016 and #123178. This regression is related to an OS side change that requires a slight adjustment from us on PyTorch side to restore the previous behavior. Additionally we cleared out pre-MacOS13 related workarounds. Before the fix on MacOS 14.4: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 3., 3.], device='mps:0') ``` After the fix: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 1., 3.], device='mps:0') ``` This also fixes complex number initialization and as such makes `nn.functional.rms_norm` pass on MacOS-14+ Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Pull Request resolved: #123234 Approved by: https://github.com/malfet, https://github.com/kulinseth (cherry picked from commit 05289a2)

Fixes #122016 and #123178. This regression is related to an OS side change that requires a slight adjustment from us on PyTorch side to restore the previous behavior. Additionally we cleared out pre-MacOS13 related workarounds. Before the fix on MacOS 14.4: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 3., 3.], device='mps:0') ``` After the fix: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 1., 3.], device='mps:0') ``` This also fixes complex number initialization and as such makes `nn.functional.rms_norm` pass on MacOS-14+ Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Pull Request resolved: #123234 Approved by: https://github.com/malfet, https://github.com/kulinseth (cherry picked from commit 05289a2) Co-authored-by: Joona Havukainen <jhavukainen@apple.com>

I.e. if issue are mention in fixed PR as `#\d+` script should still look for the cherry pick in the branch Before that change pytorch/pytorch#123234 were missed in release/2.3 branch

…123234) Fixes pytorch#122016 and pytorch#123178. This regression is related to an OS side change that requires a slight adjustment from us on PyTorch side to restore the previous behavior. Additionally we cleared out pre-MacOS13 related workarounds. Before the fix on MacOS 14.4: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 3., 3.], device='mps:0') ``` After the fix: ``` python -c "import torch;x=torch.zeros(3, device='mps');x[1] = 1; x[2] = 3; print(x)" tensor([0., 1., 3.], device='mps:0') ``` This also fixes complex number initialization and as such makes `nn.functional.rms_norm` pass on MacOS-14+ Co-authored-by: Nikita Shulga <2453524+malfet@users.noreply.github.com> Pull Request resolved: pytorch#123234 Approved by: https://github.com/malfet, https://github.com/kulinseth

jhavukainen requested review from kulinseth and malfet as code owners April 3, 2024 01:34

pytorch-bot bot added the release notes: mps Release notes category label Apr 3, 2024

pytorchbot added the open source label Apr 3, 2024

malfet added the ciflow/mps Run MPS tests (subset of trunk) label Apr 3, 2024

malfet approved these changes Apr 3, 2024

View reviewed changes

kulinseth approved these changes Apr 3, 2024

View reviewed changes

Fix for MPS regression in pytorch#122016

16eb447

pytorchmergebot force-pushed the dev/joona/Issue122016 branch from c4529ba to 16eb447 Compare April 3, 2024 17:12

malfet reviewed Apr 3, 2024

View reviewed changes

aten/src/ATen/native/mps/operations/ConstantOps.mm Outdated Show resolved Hide resolved

Fix lint

100336c

malfet reviewed Apr 3, 2024

View reviewed changes

aten/src/ATen/native/mps/operations/ConstantOps.mm Outdated Show resolved Hide resolved

Fix lint take 2

fb6cf89

malfet added 2 commits April 3, 2024 13:52

Update test_mps.py

ff82133

Sorry, it passes only on MacOS-14

a3a2acf

pytorchmergebot added the merging label Apr 3, 2024

pytorchmergebot added the Merged label Apr 3, 2024

pytorchmergebot closed this in 05289a2 Apr 3, 2024

pytorchmergebot removed the merging label Apr 3, 2024

malfet mentioned this pull request Apr 4, 2024

[MPS] Gradient error with LeakyRELU/Mish/GELU + .sum() on 1D and scalar tensors #123178

Closed

malfet mentioned this pull request Apr 5, 2024

[v.2.3.0] Release Tracker #121760

Closed

This was referenced Apr 8, 2024

linear model cannot calculate grads correctly on device MPS #123259

Open

llama-fast produces gibberish with --device mps, works with --device cpu #123151

Closed

riklopfer mentioned this pull request Apr 16, 2024

Nondeterministic behavior from GPT with MPS backend huggingface/transformers#30229

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for MPS regression in #122016 and #123178 #123234

Fix for MPS regression in #122016 and #123178 #123234

jhavukainen commented Apr 3, 2024 •

edited by malfet

pytorch-bot bot commented Apr 3, 2024 •

edited

jhavukainen commented Apr 3, 2024

malfet left a comment

malfet commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorch-bot bot commented Apr 3, 2024

malfet commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

malfet commented Apr 4, 2024

malfet commented Apr 4, 2024

pytorch-bot bot commented Apr 4, 2024

malfet commented Apr 4, 2024

pytorchbot commented Apr 4, 2024

Fix for MPS regression in #122016 and #123178 #123234

Fix for MPS regression in #122016 and #123178 #123234

Conversation

jhavukainen commented Apr 3, 2024 • edited by malfet

pytorch-bot bot commented Apr 3, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/123234

✅ No Failures

jhavukainen commented Apr 3, 2024

malfet left a comment

Choose a reason for hiding this comment

malfet commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorch-bot bot commented Apr 3, 2024

malfet commented Apr 3, 2024

malfet commented Apr 3, 2024

pytorchmergebot commented Apr 3, 2024

Merge started

malfet commented Apr 4, 2024

malfet commented Apr 4, 2024

pytorch-bot bot commented Apr 4, 2024

malfet commented Apr 4, 2024

pytorchbot commented Apr 4, 2024

Cherry picking #123234

jhavukainen commented Apr 3, 2024 •

edited by malfet

pytorch-bot bot commented Apr 3, 2024 •

edited