ND hang of SD unit tests on N300 device #7560

mtatsumiTT · 2024-04-17T21:02:24Z

Running SD unit tests with WH_ARCH_YAML on N300 devices non-deterministically hangs.

To repro the issue, switch to main branch and run the following on N300 device:

WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest tests/ttnn/integration_tests/stable_diffusion

EDIT:
Running the same test with enabling watcher in the fast-dispatch CI raises the std::runtime_error below on tests/ttnn/integration_tests/stable_diffusion/test_cross_attn_up_block_2d.py (full log):

terminate called after throwing an instance of 'std::runtime_error'
  what():  Read 0xffffffff from ARC scratch[6]: auto-reset succeeded.
Fatal Python error: Aborted
Thread 0x00007f3744ff9700 (most recent call first):
  File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 306 in wait
  File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 558 in wait
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007f38db2c1740 (most recent call first):
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 410 in call_wrapper
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 616 in call_wrapper
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/decorators.py", line 693 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 306 in time_sharded_attention
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 471 in get_attention_scores_opt
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attention.py", line 706 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_basic_transformer_block.py", line 90 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_transformer_2d.py", line 298 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/models/experimental/functional_stable_diffusion/tt2/ttnn_functional_cross_attn_upblock.py", line 153 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/tests/ttnn/integration_tests/stable_diffusion/test_cross_attn_up_block_2d.py", line 321 in test_cross_attn_up_block_2d_512x512
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/python.py", line 1789 in runtest
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/python_env/bin/pytest", line 8 in <module>

fyi @AleksKnezevic @vtangTT @TT-billteng

The text was updated successfully, but these errors were encountered:

#7560

TT-billteng · 2024-04-18T15:00:14Z

hanging on N150 in main post-commit

https://github.com/tenstorrent/tt-metal/actions/runs/8728371259/job/23948244160
https://github.com/tenstorrent/tt-metal/actions/runs/8739120961/job/23980015228

seems to be specifically this test

tests/ttnn/unit_tests/test_sd_e2e.py::test_unet_2d_condition_model_512x512[batch_size=2-in_channels=4-input_height=64-input_width=64]

#7560

jliangTT · 2024-04-25T21:38:10Z

some discussions are happening over here - https://tenstorrent.slack.com/archives/C055REZR6Q3/p1713992657339109

mtatsumiTT · 2024-05-01T14:56:15Z

quick update: some one the unit tests were raising OOM and allocator errors, but e2e test was passing. I'll skip the tests with OOM errors, and launch a pipeline after rebasing to the latest main to double-check all SD unit tests pass

jliangTT · 2024-05-01T15:15:51Z

Next step: Please try to repro it on the lastest FD2/main branch.

jliangTT · 2024-05-01T20:52:28Z

Next step:

debugging/repro with watcher

AleksKnezevic · 2024-05-02T01:59:41Z

I have been able to successfully reproduce the hang running watcher without NOC sanitization three times. Twice on the same op, once on a different one. I also ran without hangs ~5 times, so still ND. The different one is in the same submodule (the one @mtatsumiTT identified as problematic), so probably related.

AleksKnezevic · 2024-05-02T02:00:24Z

To repro, run WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest --count=100 -svv tests/ttnn/integration_tests/stable_diffusion/test_unet_2d_condition_model.py -k 512 on aknezevic/hang_debug

AleksKnezevic · 2024-05-02T02:01:36Z

hang_debug.txt

jvasilje · 2024-05-02T02:01:54Z

Sounds like we should try to repro on the submodule as next step. Then we will have a smaller test to debug in detail. Less ops.

… hang observed on BM machine where perf tests are run.

… ND hang. No hang observed on BM machine where perf tests are run." This reverts commit e517e56.

jliangTT · 2024-05-06T17:50:43Z

the FD2 merged over this weekend - can we rebase and re-test against the main to get the latest baseline result?

AleksKnezevic · 2024-05-06T18:24:49Z

Hang is still present on both tests. I rebased and pushed aknezevic/repro_MM_hang

tt-aho · 2024-05-06T21:49:18Z

@AleksKnezevic have you even seen a seg fault on WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest -svv tests/ttnn/integration_tests/stable_diffusion/test_geglu.py::test_geglu_512x512[N=1-C=2-H=256-W=1280-index=1-model_name=CompVis/stable-diffusion-v1-4-device_l1_small_size=32768]?

Seem to be getting it around ~150 iterations (note this was on a t3k system, will try to repro on a vm next)

AleksKnezevic · 2024-05-07T01:08:46Z

I have on occasion seen a seg fault on VM but not on BM. I've been using an N300. @tt-aho, are you clearing the built directory before running the test?

… hang observed on BM machine where perf tests are run.

jliangTT · 2024-05-07T17:55:47Z

High bw debugging happening in slack thread (internal) - https://tenstorrent.slack.com/archives/C055REZR6Q3/p1715102452304479

aliuTT · 2024-05-08T00:40:10Z

catching up on debug log, @jliangTT can you add me to the slack thread you linked?

ttmtrajkovic · 2024-05-13T17:14:39Z

hey @AleksKnezevic,

Could you please summarize the tests used to reproduce the problem: both a small one and the complex one?
Also, is it reproducible on main?

AleksKnezevic · 2024-05-14T00:46:58Z

The small test is two matmuls and an elementwise multiply, the larger test is self attention (matmul, softmax, matmul). The tests are currently on branch aknezevic/repro_MM_hang

pavlepopovic · 2024-05-30T08:04:45Z

Fyi: #8644
In falcon7b prefill, we've also had a problem with matmul->softmax->matmul.
We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.

TT-BrianLiu · 2024-05-30T17:50:09Z

Fyi: #8644 In falcon7b prefill, we've also had a problem with matmul->softmax->matmul. We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.

Subblok specs shouldn't affect how many cores are used for matmul. It increases the number of loops in compute which essentially slows it down.

pavlepopovic · 2024-05-30T18:16:44Z

Fyi: #8644 In falcon7b prefill, we've also had a problem with matmul->softmax->matmul. We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.

Subblok specs shouldn't affect how many cores are used for matmul. It increases the number of loops in compute which essentially slows it down.

Yup, those are 2 changes we needed to apply to get rid of di/dt (subblocks and grid size reduction)
(Just realised that ‘furthermore’ is a bad choice of words for what I was trying to say :D)

vtangTT added a commit that referenced this issue Apr 17, 2024

#7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue

e037114

#7560

tt-rkim pushed a commit that referenced this issue Apr 18, 2024

#7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue

e97c49b

#7560

tt-rkim pushed a commit that referenced this issue Apr 18, 2024

#7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue

b6eea46

#7560

TT-billteng assigned mtatsumiTT Apr 18, 2024

TT-billteng added ci-bug bugs found in CI bug Something isn't working labels Apr 18, 2024

dimitri-tenstorrent pushed a commit that referenced this issue Apr 19, 2024

#7502: skipping 'integration_tests/stable_diffusion' bc of hangs: issue

6cd3bae

#7560

smehtaTT added the P0_Showstopper label Apr 24, 2024

mtatsumiTT added a commit that referenced this issue Apr 26, 2024

#7560: Add SD tests to FD nightly with legacy pass

e23bedf

mtatsumiTT added a commit that referenced this issue Apr 26, 2024

#7560: Add SD tests to FD nightly with legacy pass

d957a4e

mtatsumiTT added a commit that referenced this issue Apr 26, 2024

#7560: Add SD tests to FD nightly with legacy pass

d6739cc

mtatsumiTT added a commit that referenced this issue Apr 26, 2024

#7560: Add SD tests to FD nightly with legacy pass

5e007b4

mtatsumiTT added a commit that referenced this issue Apr 29, 2024

#7560: Add SD tests to FD nightly with legacy pass

df5d260

mtatsumiTT added a commit that referenced this issue Apr 30, 2024

#7560: Add SD tests to FD nightly with legacy pass

984b420

mtatsumiTT added a commit that referenced this issue Apr 30, 2024

#7560: Add SD tests to FD nightly with legacy pass

a566bdd

mtatsumiTT added a commit that referenced this issue Apr 30, 2024

#7560: Add SD tests to FD nightly with legacy pass

a2d9139

mtatsumiTT added a commit that referenced this issue May 1, 2024

#7560: Add SD tests to FD nightly with legacy pass

8f63e6d

tt-rkim mentioned this issue May 2, 2024

Stable Diffusion hang on N300 #8008

Closed

mtatsumiTT added a commit that referenced this issue May 2, 2024

#7560: Add SD tests to FD nightly with legacy pass

f58a0cb

mtatsumiTT added a commit that referenced this issue May 2, 2024

#7560: Add SD tests to FD nightly with legacy pass

32baaa4

mtatsumiTT added a commit that referenced this issue May 2, 2024

#7560: Add SD tests to FD nightly with legacy pass

dcbb868

AleksKnezevic added a commit that referenced this issue May 4, 2024

#7560: Slow down MMs when running non-perf tests to avoid ND hang. No…

169a213

… hang observed on BM machine where perf tests are run.

AleksKnezevic added a commit that referenced this issue May 4, 2024

#7560: Slow down MMs when running non-perf tests to avoid ND hang. No…

65c745e

… hang observed on BM machine where perf tests are run.

AleksKnezevic added a commit that referenced this issue May 4, 2024

#7560: Slow down MMs when running non-perf tests to avoid ND hang. No…

e517e56

… hang observed on BM machine where perf tests are run.

tt-asaigal added a commit that referenced this issue May 4, 2024

#0: Revert "#7560: Slow down MMs when running non-perf tests to avoid…

48602a3

… ND hang. No hang observed on BM machine where perf tests are run." This reverts commit e517e56.

ankitmcw pushed a commit that referenced this issue May 7, 2024

#7560: Slow down MMs when running non-perf tests to avoid ND hang. No…

eef54da

… hang observed on BM machine where perf tests are run.

ankitmcw pushed a commit that referenced this issue May 7, 2024

#7560: Slow down MMs when running non-perf tests to avoid ND hang. No…

7283c48

… hang observed on BM machine where perf tests are run.

jliangTT assigned aliuTT, ttmtrajkovic and rtawfik01 and unassigned jliangTT May 7, 2024

tt-rkim mentioned this issue May 8, 2024

Disable tensor cache in Mamba performance tests #8248

Merged

jvasilje added the didt_confirmed label May 9, 2024

ttmtrajkovic added didt_suspected didt_confirmed and removed didt_confirmed didt_suspected labels May 13, 2024

smehtaTT added P2_should_have and removed P0_Showstopper labels May 29, 2024

pavlepopovic mentioned this issue May 30, 2024

Llama2-70B Prefill on 2K sequence Length Non-deterministic Hang on Decoder Unit Test #8959

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ND hang of SD unit tests on N300 device #7560

ND hang of SD unit tests on N300 device #7560

mtatsumiTT commented Apr 17, 2024 •

edited

Loading

TT-billteng commented Apr 18, 2024 •

edited

Loading

jliangTT commented Apr 25, 2024

mtatsumiTT commented May 1, 2024

jliangTT commented May 1, 2024

jliangTT commented May 1, 2024

AleksKnezevic commented May 2, 2024

AleksKnezevic commented May 2, 2024 •

edited

Loading

AleksKnezevic commented May 2, 2024

jvasilje commented May 2, 2024

jliangTT commented May 6, 2024

AleksKnezevic commented May 6, 2024

tt-aho commented May 6, 2024 •

edited

Loading

AleksKnezevic commented May 7, 2024

jliangTT commented May 7, 2024

aliuTT commented May 8, 2024

ttmtrajkovic commented May 13, 2024

AleksKnezevic commented May 14, 2024

pavlepopovic commented May 30, 2024

TT-BrianLiu commented May 30, 2024

pavlepopovic commented May 30, 2024 •

edited

Loading

ND hang of SD unit tests on N300 device #7560

ND hang of SD unit tests on N300 device #7560

Comments

mtatsumiTT commented Apr 17, 2024 • edited Loading

TT-billteng commented Apr 18, 2024 • edited Loading

jliangTT commented Apr 25, 2024

mtatsumiTT commented May 1, 2024

jliangTT commented May 1, 2024

jliangTT commented May 1, 2024

AleksKnezevic commented May 2, 2024

AleksKnezevic commented May 2, 2024 • edited Loading

AleksKnezevic commented May 2, 2024

jvasilje commented May 2, 2024

jliangTT commented May 6, 2024

AleksKnezevic commented May 6, 2024

tt-aho commented May 6, 2024 • edited Loading

AleksKnezevic commented May 7, 2024

jliangTT commented May 7, 2024

aliuTT commented May 8, 2024

ttmtrajkovic commented May 13, 2024

AleksKnezevic commented May 14, 2024

pavlepopovic commented May 30, 2024

TT-BrianLiu commented May 30, 2024

pavlepopovic commented May 30, 2024 • edited Loading

mtatsumiTT commented Apr 17, 2024 •

edited

Loading

TT-billteng commented Apr 18, 2024 •

edited

Loading

AleksKnezevic commented May 2, 2024 •

edited

Loading

tt-aho commented May 6, 2024 •

edited

Loading

pavlepopovic commented May 30, 2024 •

edited

Loading