Value range refinement using multi-variate expressions.

[ysiraichi]

Stack from ghstack (oldest at bottom):

• -> Value range refinement using multi-variate expressions. #97964
• Value range refinement using uni-variate expressions. #97963
• Turn translation validation on for tests and accuracy runs by default. #103611
• Translation validator for dynamo guards. #102563

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/97964

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d98aa5e:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ysiraichi]

As is, the only difference (benchmark-wise) is that pytorch_unet goes from 9/73/73 (univariate only) to 9/70/70. Still have to run the whole benchmark set, though.

[lezcano]

In this case, when refining the bounds, it could be the case that rhs_vr.lower/upper are variables. In that case, the refinement could then be lower = max(lower, rhs_vr.lower), right?

[lezcano]

At any rate, this needs some testing.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[ysiraichi]

@ezyang @huydhn
Best guess I have is: multi-variate range refinement is not sound. Could you try running it while setting torch._dynamo.config.translation_validation = True? Or, is there an example I can try to reproduce the error?

[lezcano]

This implementations looks good to me, so I don't think it's "not sound" by itself.

My gut tells me that the issue is lurking in the value range analysis. I already found a number of bugs in it, so I wouldn't be entirely surprised that there were more issues lurking there.

[ysiraichi]

@lezcano I think the translation validator should be able to catch that.

[ysiraichi]

@ezyang @huydhn
This is a friendly reminder. If you have some time, could you try and run the failing test with TV on?

[ezyang]

going to attempt this, might be tricky because i'm not sure we have z3 on our internal

[ezyang]

Patching this

diff --git a/fbcode/caffe2/torch/_dynamo/config.py b/fbcode/caffe2/torch/_dynamo/config.py
--- a/fbcode/caffe2/torch/_dynamo/config.py
+++ b/fbcode/caffe2/torch/_dynamo/config.py
@@ -237,7 +237,7 @@
 numpy_ndarray_as_tensor = False
 
 # Uses z3 for validating the guard optimizations transformations.
-translation_validation = False
+translation_validation = True
 
 
 def is_fbcode():

and rerunning the test, I still get the same error as before:

  File "/data/users/ezyang/fbsource/buck-out/v2/gen/fbcode/8c7462494077df89/pye/model_inventory/inside_out_tracking_model/__inside_out_tracking_model_test__/
inside_out_tracking_model_test#link-tree/torch/_dynamo/eval_frame.py", line 1012, in graph_with_interpreter                                                      return torch.fx.Interpreter(graph).run(*args)                                                                                                            
  File "/data/users/ezyang/fbsource/buck-out/v2/gen/fbcode/8c7462494077df89/pye/model_inventory/inside_out_tracking_model/__inside_out_tracking_model_test__/
inside_out_tracking_model_test#link-tree/torch/fx/interpreter.py", line 138, in run                                                                              self.env[node] = self.run_node(node)                                                                                                                     
  File "/data/users/ezyang/fbsource/buck-out/v2/gen/fbcode/8c7462494077df89/pye/model_inventory/inside_out_tracking_model/__inside_out_tracking_model_test__/
inside_out_tracking_model_test#link-tree/torch/fx/interpreter.py", line 195, in run_node                                                                         return getattr(self, n.op)(n.target, args, kwargs)                                                                                                       
  File "/data/users/ezyang/fbsource/buck-out/v2/gen/fbcode/8c7462494077df89/pye/model_inventory/inside_out_tracking_model/__inside_out_tracking_model_test__/
inside_out_tracking_model_test#link-tree/torch/fx/interpreter.py", line 267, in call_function                                                                
    return target(*args, **kwargs)                                                                                                                           
IndexError: tuple index out of range                                                                                                                         
                                                                                                                                                             
While executing %getitem_6 : [num_users=1] = call_function[target=operator.getitem](args = (%chunk, 2), kwargs = {})   

which suggests that we don't get to TV (which makes sense, TV only runs all the way at the end of execution, but we're erroring midway through.

I have a live session failing, lmk what kind of debugging instrumentation you want.

[ysiraichi]

@ezyang

This is really odd. AFAIU, this PR modifies var_to_range which affects what guards are issued. So, if there was a problem with it, I would expect it to be executing a wrong compiled function (i.e. we should not have eliminated some guards).

Just to be completely sure, could you try commenting these lines at the end of refine_ranges and check that it passes the test?

# Updates the range and the guards corresponding to each bound of the symbol.
self.var_to_range[symbol] = ValueRanges(lower, upper)
self.var_to_guards[symbol] = (lower_guard, upper_guard)

[ezyang]

Applying this patch

diff --git a/fbcode/caffe2/torch/fx/experimental/symbolic_shapes.py b/fbcode/caffe2/torch/fx/experimental/symbolic_shapes.py
--- a/fbcode/caffe2/torch/fx/experimental/symbolic_shapes.py
+++ b/fbcode/caffe2/torch/fx/experimental/symbolic_shapes.py
@@ -3467,8 +3467,8 @@
                 continue
 
             # Updates the range and the guards corresponding to each bound of the symbol.
-            self.var_to_range[symbol] = ValueRanges(lower, upper)
-            self.var_to_guards[symbol] = (lower_guard, upper_guard)
+            #self.var_to_range[symbol] = ValueRanges(lower, upper)
+            #self.var_to_guards[symbol] = (lower_guard, upper_guard)
             # Clears the cache, since this update can change the result.
             self._maybe_evaluate_static.cache_clear()
 

and confirmed the test passes afterwards.