[Bug]: aot_compile does not preserve dynamic shapes state on cache hit.

[laithsakka]

Your current environment

latest pytorch commit and latest vLLM commit.

🐛 Describe the bug

ok on a cache hit more things are marked dynamic than needed, this seems very similar to what happen in
#27899 (read my last comment)
this is risky because marking more things dynamic with duck shapes can cause silent specialization that are not wanted.

to repo run any model +dynamic logs
For example for "Qwen/Qwen2-7B-Instruct" on a cold run we will see only three things dynamic.

on a warm run we will see so many things marked dynamic.

This is. BAD!

We need to

1. disable automatic dynamic.
2. on a warm run know what things to be marked dynamic and mark them explicitly.
   OR
   serialize fake tensor for the warm run!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[laithsakka]

cc @zou3519 @zhxchen17
I wonder if we shall come up with a plan to do comprehensive testing for aot_compile before the next release of pytorch to avoid sprints on issues like last time

[laithsakka]

problem starts here
where we pass real examples as input, but what do we really do to determine what shall be dynamic and what shall be static when example inputs are converted to fake tensors, they are not fake tensors here. and this is problematic since we will depend on default automatic dynamic to decide.

   def optimized_call(*example_inputs):
            """
            On the first run of the optimized call, we rerun the compiler
            backend which should result in a cache hit. After the backend
            call returns, we just do a one-time replacement of the optimized
            call with the compiled function, so that subsequent calls are on
            the AOT compiled path.
            """
            compile_inputs = []
            for i, inp in enumerate(fn.example_inputs):
                if inp is not None:
                    compile_inputs.append(inp)
                else:
                    compile_inputs.append(example_inputs[i])

            with tracing(TracingContext(fake_mode)):
                fn.optimized_call = vllm_backend(
                    state["graph_module"], compile_inputs
                ).optimized_call
            return fn.optimized_call(*example_inputs)

[zou3519]

We need a rollout plan for aot_compile

[zhxchen17]

@laithsakka Not sure I understand the issue well here. Would like to discuss this offline so that I can understand correctly.

[zou3519]

I don't really understand what the problem is. Does dynamic=False (this disables automatic dynamic, right?) fix the issue?

[laithsakka]

I don't really understand what the problem is. Does dynamic=False (this disables automatic dynamic, right?) fix the issue?

This PR fix the issue
#29104