hf_LongFormer failing eval with inductor and dynamic shapes

[ezyang]

🐛 Describe the bug

Here is the error:

2023-05-06T11:09:15.3987736Z cuda eval  hf_Longformer                       WARNING:common:fp64 golden ref were not generated for hf_Longformer. Setting accuracy check to cosine
2023-05-06T11:09:23.9126068Z [2023-05-06 11:09:23,911] torch._dynamo.variables.torch: [WARNING] Calling <built-in method div of type object at 0x7f6ee83e7ec0> on only torch.SymInt arguments is not yet supported.
2023-05-06T11:09:23.9127806Z To support this behavior, we need to allow const-propping tensors that store symint data.
2023-05-06T11:09:23.9130519Z For now, dynamo will explicitly graph break when it encounters user code with this behavior.
2023-05-06T11:09:23.9130774Z 
2023-05-06T11:09:29.3972191Z ERROR:common:backend='inductor' raised:
2023-05-06T11:09:29.3972744Z LoweringException: AttributeError: 'View' object has no attribute 'get_stride'
2023-05-06T11:09:29.3973066Z   target: aten.sym_stride
2023-05-06T11:09:29.3973300Z   args[0]: TensorBox(
2023-05-06T11:09:29.3973508Z     View(
2023-05-06T11:09:29.3973680Z       View(
2023-05-06T11:09:29.3980329Z         PermuteView(data=PermuteView(data=View(
2023-05-06T11:09:29.3980772Z           StorageBox(
2023-05-06T11:09:29.3981578Z             Pointwise(
2023-05-06T11:09:29.3982239Z               'cuda',
2023-05-06T11:09:29.3982597Z               torch.float16,
2023-05-06T11:09:29.3982979Z               def inner_fn(index):
2023-05-06T11:09:29.3983372Z                   i0, i1, i2 = index
2023-05-06T11:09:29.3983722Z                   tmp0 = ops.load(buf1, i2 + 768 * i1 + 768 * i0 * s0)
2023-05-06T11:09:29.3984197Z                   tmp1 = ops.load(arg1_1, i2)
2023-05-06T11:09:29.3984537Z                   tmp2 = tmp0 + tmp1
2023-05-06T11:09:29.3984997Z                   tmp3 = ops.constant(8.0, torch.float16)
2023-05-06T11:09:29.3985422Z                   tmp4 = tmp2 / tmp3
2023-05-06T11:09:29.3985786Z                   return tmp4
2023-05-06T11:09:29.3986166Z               ,
2023-05-06T11:09:29.3986453Z               ranges=[4096, s0, 768],
2023-05-06T11:09:29.3986872Z               origin_node=div,
2023-05-06T11:09:29.3987238Z               origins={add, div}
2023-05-06T11:09:29.3987584Z             )
2023-05-06T11:09:29.3987943Z           ),
2023-05-06T11:09:29.3988239Z           size=(4096, s0, 12, 64),
2023-05-06T11:09:29.3988695Z           reindex=lambda i0, i1, i2, i3: [i0, i1, 64*i2 + i3],
2023-05-06T11:09:29.3989110Z           origins={add, div, view_6}
2023-05-06T11:09:29.3989558Z         ), dims=[1, 0, 2, 3]), dims=[0, 2, 1, 3]),
2023-05-06T11:09:29.3989890Z         size=(12*s0, 4096, 64),
2023-05-06T11:09:29.3990546Z         reindex=lambda i0, i1, i2: [ModularIndexing(i0, 12, s0), ModularIndexing(i0, 1, 12), i1, i2],
2023-05-06T11:09:29.3991025Z         origins={view_8}
2023-05-06T11:09:29.3991405Z       ),
2023-05-06T11:09:29.3991743Z       size=(12*s0, 8, 512, 64),
2023-05-06T11:09:29.3992205Z       reindex=lambda i0, i1, i2, i3: [i0, 512*i1 + i2, i3],
2023-05-06T11:09:29.3992593Z       origins={view_10}
2023-05-06T11:09:29.3992932Z     )
2023-05-06T11:09:29.3993256Z   )
2023-05-06T11:09:29.3993505Z   args[1]: 1
2023-05-06T11:09:29.3993712Z 
2023-05-06T11:09:29.3993723Z 
2023-05-06T11:09:29.3994041Z You can suppress this exception and fall back to eager by setting:
2023-05-06T11:09:29.3994494Z     import torch._dynamo
2023-05-06T11:09:29.3994998Z     torch._dynamo.config.suppress_errors = True
2023-05-06T11:09:29.3995420Z Traceback (most recent call last):
2023-05-06T11:09:29.3996020Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1448, in check_accuracy
2023-05-06T11:09:29.3996946Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-05-06T11:09:29.3998022Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 282, in _fn
2023-05-06T11:09:29.3998548Z     return fn(*args, **kwargs)
2023-05-06T11:09:29.3999141Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1291, in run_n_iterations
2023-05-06T11:09:29.3999774Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-05-06T11:09:29.4000427Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 392, in forward_pass
2023-05-06T11:09:29.4000965Z     return mod(*inputs)
2023-05-06T11:09:29.4001847Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
2023-05-06T11:09:29.4002802Z     return self._call_impl(*args, **kwargs)
2023-05-06T11:09:29.4003672Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
2023-05-06T11:09:29.4004335Z     return forward_call(*args, **kwargs)
2023-05-06T11:09:29.4005286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1848, in forward
2023-05-06T11:09:29.4005907Z     outputs = self.longformer(
2023-05-06T11:09:29.4006783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
2023-05-06T11:09:29.4007391Z     return self._call_impl(*args, **kwargs)
2023-05-06T11:09:29.4008313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
2023-05-06T11:09:29.4008981Z     return forward_call(*args, **kwargs)
2023-05-06T11:09:29.4009571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1750, in forward
2023-05-06T11:09:29.4010216Z     encoder_outputs = self.encoder(
2023-05-06T11:09:29.4010872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
2023-05-06T11:09:29.4011242Z     return self._call_impl(*args, **kwargs)
2023-05-06T11:09:29.4011745Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
2023-05-06T11:09:29.4012152Z     return forward_call(*args, **kwargs)
2023-05-06T11:09:29.4012789Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1294, in forward
2023-05-06T11:09:29.4013218Z     is_global_attn = is_index_global_attn.flatten().any().item()
2023-05-06T11:09:29.4013834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1326, in <resume in forward>
2023-05-06T11:09:29.4014220Z     layer_outputs = layer_module(
2023-05-06T11:09:29.4014726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
2023-05-06T11:09:29.4015096Z     return self._call_impl(*args, **kwargs)
2023-05-06T11:09:29.4015595Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
2023-05-06T11:09:29.4015951Z     return forward_call(*args, **kwargs)
2023-05-06T11:09:29.4016589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 435, in catch_errors
2023-05-06T11:09:29.4016975Z     return callback(frame, cache_size, hooks, frame_state)
2023-05-06T11:09:29.4017495Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 519, in _convert_frame
2023-05-06T11:09:29.4017889Z     result = inner_convert(frame, cache_size, hooks, frame_state)
2023-05-06T11:09:29.4018410Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 122, in _fn
2023-05-06T11:09:29.4018744Z     return fn(*args, **kwargs)
2023-05-06T11:09:29.4019250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 355, in _convert_frame_assert
2023-05-06T11:09:29.4019604Z     return _compile(
2023-05-06T11:09:29.4020068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 177, in time_wrapper
2023-05-06T11:09:29.4020426Z     r = func(*args, **kwargs)
2023-05-06T11:09:29.4020908Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 425, in _compile
2023-05-06T11:09:29.4021290Z     out_code = transform_code_object(code, transform)
2023-05-06T11:09:29.4021861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1000, in transform_code_object
2023-05-06T11:09:29.4022449Z     transformations(instructions, code_options)
2023-05-06T11:09:29.4022980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 410, in transform
2023-05-06T11:09:29.4023467Z     tracer.run()
2023-05-06T11:09:29.4023933Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2010, in run
2023-05-06T11:09:29.4024259Z     super().run()
2023-05-06T11:09:29.4024725Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 703, in run
2023-05-06T11:09:29.4025036Z     and self.step()
2023-05-06T11:09:29.4025509Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 663, in step
2023-05-06T11:09:29.4025861Z     getattr(self, inst.opname)(inst)
2023-05-06T11:09:29.4026514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2098, in RETURN_VALUE
2023-05-06T11:09:29.4026880Z     self.output.compile_subgraph(
2023-05-06T11:09:29.4027408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 736, in compile_subgraph
2023-05-06T11:09:29.4027833Z     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2023-05-06T11:09:29.4028191Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-05-06T11:09:29.4028490Z     return func(*args, **kwds)
2023-05-06T11:09:29.4029009Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 813, in compile_and_call_fx_graph
2023-05-06T11:09:29.4029406Z     compiled_fn = self.call_user_compiler(gm)
2023-05-06T11:09:29.4029896Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 177, in time_wrapper
2023-05-06T11:09:29.4030301Z     r = func(*args, **kwargs)
2023-05-06T11:09:29.4030823Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 872, in call_user_compiler
2023-05-06T11:09:29.4031273Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-05-06T11:09:29.4031845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 868, in call_user_compiler
2023-05-06T11:09:29.4032243Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-05-06T11:09:29.4032790Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 108, in debug_wrapper
2023-05-06T11:09:29.4033158Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-05-06T11:09:29.4033672Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-05-06T11:09:29.4034027Z     return compile_fx(*args, **kwargs)
2023-05-06T11:09:29.4034517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 728, in compile_fx
2023-05-06T11:09:29.4034859Z     return aot_autograd(
2023-05-06T11:09:29.4035357Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 56, in compiler_fn
2023-05-06T11:09:29.4035754Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-05-06T11:09:29.4036304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3334, in aot_module_simplified
2023-05-06T11:09:29.4036979Z     compiled_fn = create_aot_dispatcher_function(
2023-05-06T11:09:29.4037541Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 177, in time_wrapper
2023-05-06T11:09:29.4037864Z     r = func(*args, **kwargs)
2023-05-06T11:09:29.4038408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2975, in create_aot_dispatcher_function
2023-05-06T11:09:29.4038882Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-05-06T11:09:29.4039476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1911, in aot_wrapper_dedupe
2023-05-06T11:09:29.4040078Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-05-06T11:09:29.4040747Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2082, in aot_wrapper_synthetic_base
2023-05-06T11:09:29.4041194Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-05-06T11:09:29.4041775Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1348, in aot_dispatch_base
2023-05-06T11:09:29.4042192Z     compiled_fw = compiler(fw_module, adjusted_flat_args)
2023-05-06T11:09:29.4042845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 177, in time_wrapper
2023-05-06T11:09:29.4043183Z     r = func(*args, **kwargs)
2023-05-06T11:09:29.4043674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 684, in fw_compiler_base
2023-05-06T11:09:29.4044102Z     return inner_compile(
2023-05-06T11:09:29.4044610Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
2023-05-06T11:09:29.4045003Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-05-06T11:09:29.4045515Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 220, in inner
2023-05-06T11:09:29.4045829Z     return fn(*args, **kwargs)
2023-05-06T11:09:29.4046141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-05-06T11:09:29.4046442Z     return func(*args, **kwds)
2023-05-06T11:09:29.4046928Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 210, in compile_fx_inner
2023-05-06T11:09:29.4047283Z     graph.run(*example_inputs)
2023-05-06T11:09:29.4047765Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 177, in time_wrapper
2023-05-06T11:09:29.4048099Z     r = func(*args, **kwargs)
2023-05-06T11:09:29.4048547Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 249, in run
2023-05-06T11:09:29.4048878Z     return super().run(*args)
2023-05-06T11:09:29.4049342Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 138, in run
2023-05-06T11:09:29.4049673Z     self.env[node] = self.run_node(node)
2023-05-06T11:09:29.4050196Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 488, in run_node
2023-05-06T11:09:29.4050539Z     result = super().run_node(n)
2023-05-06T11:09:29.4051006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 195, in run_node
2023-05-06T11:09:29.4051392Z     return getattr(self, n.op)(n.target, args, kwargs)
2023-05-06T11:09:29.4051908Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 392, in call_function
2023-05-06T11:09:29.4052321Z     raise LoweringException(e, target, args, kwargs).with_traceback(
2023-05-06T11:09:29.4052851Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 389, in call_function
2023-05-06T11:09:29.4053212Z     out = lowerings[target](*args, **kwargs)
2023-05-06T11:09:29.4053705Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 228, in wrapped
2023-05-06T11:09:29.4054050Z     out = decomp_fn(*args, **kwargs)
2023-05-06T11:09:29.4054550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/lowering.py", line 4036, in sym_stride
2023-05-06T11:09:29.4054888Z     return a.get_stride()[dim]
2023-05-06T11:09:29.4055413Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/ir.py", line 3823, in __getattr__
2023-05-06T11:09:29.4055739Z     fn = getattr(self.data, name)
2023-05-06T11:09:29.4056167Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-05-06T11:09:29.4056799Z LoweringException: AttributeError: 'View' object has no attribute 'get_stride'
2023-05-06T11:09:29.4057098Z   target: aten.sym_stride
2023-05-06T11:09:29.4057329Z   args[0]: TensorBox(
2023-05-06T11:09:29.4057536Z     View(
2023-05-06T11:09:29.4057712Z       View(
2023-05-06T11:09:29.4057964Z         PermuteView(data=PermuteView(data=View(
2023-05-06T11:09:29.4058222Z           StorageBox(
2023-05-06T11:09:29.4058418Z             Pointwise(
2023-05-06T11:09:29.4058673Z               'cuda',
2023-05-06T11:09:29.4058887Z               torch.float16,
2023-05-06T11:09:29.4059105Z               def inner_fn(index):
2023-05-06T11:09:29.4059335Z                   i0, i1, i2 = index
2023-05-06T11:09:29.4059693Z                   tmp0 = ops.load(buf1, i2 + 768 * i1 + 768 * i0 * s0)
2023-05-06T11:09:29.4059987Z                   tmp1 = ops.load(arg1_1, i2)
2023-05-06T11:09:29.4060260Z                   tmp2 = tmp0 + tmp1
2023-05-06T11:09:29.4060562Z                   tmp3 = ops.constant(8.0, torch.float16)
2023-05-06T11:09:29.4060973Z                   tmp4 = tmp2 / tmp3
2023-05-06T11:09:29.4061212Z                   return tmp4
2023-05-06T11:09:29.4061441Z               ,
2023-05-06T11:09:29.4061658Z               ranges=[4096, s0, 768],
2023-05-06T11:09:29.4061877Z               origin_node=div,
2023-05-06T11:09:29.4062197Z               origins={add, div}
2023-05-06T11:09:29.4062544Z             )
2023-05-06T11:09:29.4062834Z           ),
2023-05-06T11:09:29.4063103Z           size=(4096, s0, 12, 64),
2023-05-06T11:09:29.4063560Z           reindex=lambda i0, i1, i2, i3: [i0, i1, 64*i2 + i3],
2023-05-06T11:09:29.4063818Z           origins={add, div, view_6}
2023-05-06T11:09:29.4064068Z         ), dims=[1, 0, 2, 3]), dims=[0, 2, 1, 3]),
2023-05-06T11:09:29.4064317Z         size=(12*s0, 4096, 64),
2023-05-06T11:09:29.4064631Z         reindex=lambda i0, i1, i2: [ModularIndexing(i0, 12, s0), ModularIndexing(i0, 1, 12), i1, i2],
2023-05-06T11:09:29.4064941Z         origins={view_8}
2023-05-06T11:09:29.4065145Z       ),
2023-05-06T11:09:29.4065336Z       size=(12*s0, 8, 512, 64),
2023-05-06T11:09:29.4065597Z       reindex=lambda i0, i1, i2, i3: [i0, 512*i1 + i2, i3],
2023-05-06T11:09:29.4065847Z       origins={view_10}
2023-05-06T11:09:29.4066048Z     )
2023-05-06T11:09:29.4066214Z   )
2023-05-06T11:09:29.4066398Z   args[1]: 1
2023-05-06T11:09:29.4066524Z 
2023-05-06T11:09:29.4066531Z 
2023-05-06T11:09:29.4066697Z You can suppress this exception and fall back to eager by setting:
2023-05-06T11:09:29.4066975Z     import torch._dynamo
2023-05-06T11:09:29.4067255Z     torch._dynamo.config.suppress_errors = True
2023-05-06T11:09:29.4067433Z 
2023-05-06T11:09:29.4067600Z TorchDynamo optimized model failed to run because of following error
2023-05-06T11:09:29.4067877Z fail_to_run

full logs https://ossci-raw-job-status.s3.amazonaws.com/log/13280812757

Possibly related to #100115

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @soumith

Versions

master