Speed up parameter resolution for cirq.Duration

[tanujkhattar]

Fixes #6269
cc #6097

[andbe91]

LGTM from my side, thanks for fixing!

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.27% 🎉

Comparison is base (83609eb) 97.60% compared to head (b171bd4) 97.87%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6270      +/-   ##
==========================================
+ Coverage   97.60%   97.87%   +0.27%     
==========================================
  Files        1116     1106      -10     
  Lines       95912    95610     -302     
==========================================
- Hits        93619    93583      -36     
+ Misses       2293     2027     -266     

Files Changed	Coverage Δ
cirq-core/cirq/circuits/circuit.py	98.43% <100.00%> (+<0.01%)	⬆️
cirq-core/cirq/circuits/frozen_circuit.py	99.26% <100.00%> (+0.25%)	⬆️
cirq-core/cirq/circuits/frozen_circuit_test.py	100.00% <100.00%> (ø)
cirq-core/cirq/value/duration.py	98.80% <100.00%> (+0.15%)	⬆️
cirq-core/cirq/value/duration_test.py	100.00% <100.00%> (ø)

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.