fix(tdt): preserve disable_cuda_graphs state across change_decoding_strategy() calls

[CodersAcademy006]

Problem

When disable_cuda_graphs() is called on the decoding computer, the disabled
state is lost whenever change_decoding_strategy() is invoked — because it
creates a brand new decoding_computer object with CUDA graphs re-enabled
by default (cuda_graphs_mode='full_graph').

This silently re-enables CUDA graphs even when the user explicitly disabled
them, re-introducing the torch.load() corruption bug reported in #15423.

Concretely, calling model.transcribe(timestamps=True) triggers
change_decoding_strategy() internally, which replaces the entire
decoding_computer object — resetting CUDA graph state in the process.

Root Cause

• disable_cuda_graphs() state was only stored inside _decoding_computer
• change_decoding_strategy() creates a completely new _decoding_computer
• The new object always has CUDA graphs enabled by default
• No state was preserved or restored across this replacement

Fix

Two minimal changes across two files:

1. tdt_decoding.py — BeamBatchedTDTInfer.disable_cuda_graphs():
   Track the disabled state via a _cuda_graphs_disabled = True flag on the
   outer object so it survives _decoding_computer replacement.

2. rnnt_models.py — EncDecRNNTModel.change_decoding_strategy():

   • Read _cuda_graphs_disabled from the current decoding_computer BEFORE
     replacing self.decoding
   • After the new self.decoding is created, restore the disabled state on
     the new decoding_computer
   • Both read and write are wrapped in try/except AttributeError to remain
     safe across all model types that call this method

Testing

The reproducer from #15423 no longer corrupts torch.load() after
transcribe(timestamps=True) with this fix applied:

model = nemo_asr.models.ASRModel.from_pretrained('nvidia/parakeet-tdt-0.6b-v3')
model = model.cuda().eval()
model.decoding.decoding.decoding_computer.disable_cuda_graphs()
model.transcribe(['/tmp/test.wav'], timestamps=True, return_hypotheses=True)

# torch.load now works correctly after this — previously failed with:
# TypeError: 'str' object is not callable
sd = OrderedDict((f'layer.{i}', torch.randn(32, 32, device='cuda')) for i in range(10))
buf = io.BytesIO()
torch.save(sd, buf)
buf.seek(0)
torch.load(buf, weights_only=False)  # ✅ Works

Fixes #15423

Notes

• Zero breaking changes — all existing behaviour is preserved
• The try/except AttributeError guards make this safe for all RNNT model
  variants, not just TDT
• The deeper PyTorch-side corruption of torch.load() dispatch table entries
  during CUDA graph capture is a separate upstream issue and should be tracked
  in pytorch/pytorch

[CodersAcademy006]

@rsclafani Please look into this as well, Thank You. And please provide me what else i can change in this?

[artbataev]

Thanks for the contribution.
Unfortunately, I need to close this PR: the correct way to disable CUDA graphs is to change the values in the config.

from omegaconf import open_dict

# greedy decoding
with open_dict(model.cfg.decoding.greedy):
    model.cfg.decoding.greedy.use_cuda_graph_decoder = False

# beam decoding
with open_dict(model.cfg.decoding.beam):
    model.cfg.decoding.beam.allow_cuda_graphs = False

model.change_decoding_strategy(model.cfg.decoding)

We should not introduce such stateful changes - change_decoding_strategy expected to fully reinstantiate the decoder using the provided decoding parameters.

[CodersAcademy006]

@artbataev I sincerely apologize if I moved forward without formal confirmation. Please guide me on the necessary steps to rectify this and ensure the issue is completely resolved.