-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed
Labels
high priorityoncall: cpu inductorCPU Inductor issues for Intel team to triageCPU Inductor issues for Intel team to triageoncall: pt2triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate moduleThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
Description
π Describe the bug
fp32 inductor_max_autotune dynamic shape multiple thread failures
suite | name | thread | accuracy | perf | reason(reference only) |
---|---|---|---|---|---|
huggingface | AllenaiLongformerBase | multiple | β | X | AllenaiLongformerBase, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
torchbench | BERT_pytorch | multiple | X | X | |
huggingface | BlenderbotForCausalLM | multiple | pass_due_to_skip | X | BlenderbotForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | BlenderbotSmallForCausalLM | multiple | β | X | BlenderbotSmallForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | BlenderbotSmallForConditionalGeneration | multiple | β | X | BlenderbotSmallForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | DebertaForMaskedLM | multiple | β | X | DebertaForMaskedLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | DebertaForQuestionAnswering | multiple | β | X | DebertaForQuestionAnswering, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | DebertaV2ForMaskedLM | multiple | pass_due_to_skip | X | DebertaV2ForMaskedLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | DistilBertForMaskedLM | multiple | β | X | DistilBertForMaskedLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | DistilBertForQuestionAnswering | multiple | β | X | DistilBertForQuestionAnswering, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | M2M100ForConditionalGeneration | multiple | β | X | M2M100ForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | MBartForCausalLM | multiple | β | X | MBartForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | MBartForConditionalGeneration | multiple | β | X | MBartForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | MT5ForConditionalGeneration | multiple | β | X | MT5ForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | OPTForCausalLM | multiple | β | X | OPTForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | PLBartForCausalLM | multiple | β | X | PLBartForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | PLBartForConditionalGeneration | multiple | β | X | PLBartForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | PegasusForCausalLM | multiple | β | X | PegasusForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | PegasusForConditionalGeneration | multiple | β | X | PegasusForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | Speech2Text2ForCausalLM | multiple | β | X | Speech2Text2ForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | T5ForConditionalGeneration | multiple | β | X | T5ForConditionalGeneration, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | T5Small | multiple | β | X | T5Small, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | TrOCRForCausalLM | multiple | β | X | TrOCRForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | XGLMForCausalLM | multiple | β | X | XGLMForCausalLM, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
huggingface | XLNetLMHeadModel | multiple | β | X | XLNetLMHeadModel, LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints but found element of type Mul at pos 0 |
timm_models | botnet26t_256 | multiple | X | X | |
timm_models | cait_m36_384 | multiple | X | X | |
timm_models | coat_lite_mini | multiple | X | X | |
timm_models | convit_base | multiple | X | X | |
timm_models | eca_botnext26ts_256 | multiple | X | X | |
timm_models | eca_halonext26ts | multiple | X | X | |
torchbench | hf_DistilBert | multiple | X | β | |
torchbench | hf_T5 | multiple | X | β | |
torchbench | hf_T5_base | multiple | X | β | hf_T5_base, TIMEOUT |
timm_models | jx_nest_base | multiple | X | X | |
timm_models | levit_128 | multiple | X | X | |
torchbench | moondream | multiple | X | β | |
timm_models | sebotnet33ts_256 | multiple | X | X | |
timm_models | swin_base_patch4_window7_224 | multiple | X | X | |
timm_models | tnt_s_patch16_224 | multiple | X | X | |
timm_models | visformer_small | multiple | X | X | |
timm_models | volo_d1_224 | multiple | X | X | |
timm_models | xcit_large_24_p8_224 | multiple | X | X |
File "/workspace/pytorch/torch/_inductor/select_algorithm.py", line 1701, in benchmark_in_current_process
raise e from None
File "/workspace/pytorch/torch/_inductor/select_algorithm.py", line 1665, in benchmark_in_current_process
timing = benchmark_choice_in_current_process(choice, inputs)
File "/workspace/pytorch/torch/_inductor/select_algorithm.py", line 1651, in benchmark_choice_in_current_process
result = choice.benchmark(*inpts, out=output)
File "/workspace/pytorch/torch/_inductor/select_algorithm.py", line 1225, in benchmark
new_args, new_out = self._preprocessor(args, out)
File "/workspace/pytorch/torch/_inductor/codegen/cpp_gemm_template.py", line 900, in preprocessor
new_inputs, new_layout = normalize_shapes(
File "/workspace/pytorch/torch/_inductor/codegen/cpp_gemm_template.py", line 861, in normalize_shapes
new_inputs[1] = new_inputs[1].as_strided(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
LoweringException: TypeError: as_strided(): argument 'size' (position 1) must be tuple of ints, but found element of type Mul at pos 0
Versions
SW info
name | target_branch | target_commit | refer_branch | refer_commit |
---|---|---|---|---|
torchbench | main | 766a5e3a | main | 766a5e3a |
torch | main | 2682e5e | main | e29dabb |
torchvision | main | 0.19.0a0+d23a6e1 | main | 0.19.0a0+d23a6e1 |
torchtext | main | 0.16.0a0+b0ebddc | main | 0.16.0a0+b0ebddc |
torchaudio | main | 2.5.0a0+332760d | main | 2.5.0a0+332760d |
torchdata | main | 0.7.0a0+11bb5b8 | main | 0.7.0a0+11bb5b8 |
dynamo_benchmarks | main | nightly | main | nightly |
Repro:
inductor_single_run.sh
bash inductor_single_run.sh multiple inference accuracy timm_models cait_m36_384 float32 first dynamic default 0 inductor_max_autotune
Suspected guilty commit: 20f24e3
timm_models-eca_botnext26ts_256-inference-float32-dynamic-default-multiple-accuracy-crash_guilty_commit.log
cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @chauhang @penguinwu @chuanqi129
shaoyuyoung
Metadata
Metadata
Assignees
Labels
high priorityoncall: cpu inductorCPU Inductor issues for Intel team to triageCPU Inductor issues for Intel team to triageoncall: pt2triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate moduleThis issue has been looked at a team member, and triaged and prioritized into an appropriate module