[Feature] MultiCollector: track policy version per fresh continue command

[vmoens]

Stack from ghstack (oldest at bottom):

• [Doc] Migrate shifted=True callers to legacy/compact + docstring polish #3776
• [Example] Expose compact GAE cat dimension #3775
• [Example] Add Isaac RNN PPO rollout mode flags #3769
• [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes #3763
• [BugFix] Fix GAE compact path bias on recurrent value nets at internal truncations #3771
• [Refactor] Simplify LSTM/GRUModule recurrent-mode shape normalization #3768
• [Refactor] Keep [B, T] dim in value estimators #3767
• [Feature] Collector.fake_tensordict() / MultiCollector.fake_tensordict() #3764
• -> [Feature] MultiCollector: track policy version per fresh continue command #3759
• [BugFix] GAE shifted=True: tolerate missing next obs, non-canonical strides, docs #3757
• [Feature] timeit.mark_start/mark_end for non-context-manager timing #3762
• [BugFix] PolicyVersion: int64 dtype + preserve tensordict device #3755
• [Performance] LSTM/GRU scan: canonical strides + cuDNN flat-storage clones + thread-local recurrent mode #3754
• [BugFix] Recurrent policy auto-register with policy_factory #3753

When the parent calls update_policy_weights_() and immediately
sends a continue to a worker, the resulting batch was getting
tagged with the pre-update policy version: the worker's policy
version is only bumped when increment_version() is called, and
that previously only fired on direct user calls. Off-policy logging,
staleness gating, and prioritized sampling that key on the version
saw a 1-step lag.

Pass track_policy_version=(self.policy_version_tracker is not None)
to the worker, plumb a fresh_command flag through the recv loop
so the worker knows whether msg was just received vs replayed
from a timed-out iteration, and increment the version on each fresh
continue / continue_random (only when not in run_free,
where the worker drives its own pacing).

No new unit test: existing MultiCollector policy-version coverage
in test/test_collectors.py continues to assert the per-batch
version. End-to-end coverage comes through the example at the top of
the stack.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3759

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Run pull request jobs on OSDC runners in shadow mode

❌ 4 New Failures, 1 Cancelled Job

As of commit 4479a5d with merge base 0a01ee8 ():

NEW FAILURES - The following jobs have failed:

• Build Windows Wheels / pytorch/rl / build-wheel-py3_10-cpu (gh)
  Process completed with exit code 1.
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.10_cpu_x64
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Unit-tests on Linux / tests-optdeps (3.12, 13.0) / linux-job (gh)
  test/transforms/test_env_transforms.py::TestTargetReturn::test_parallel_trans_env_check[device0-constant]

CANCELLED JOB - The following job was cancelled. Please retry:

• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Unmergeable, this goes against the idea of policy version tracking. Continue != update policy weight