[RLlib] SampleBatch slicing crashes when using tf + SEQ_LENS + zero-padding.

[sven1977]

SampleBatch slicing crashes when using tf + SEQ_LENS + zero-padding.

In the following scenario, an error is thrown:

SampleBatch B w/ seq_lens=[16, 16, 16, 16, 16, 16, 2, 2, 14, 14] and max_seq_len=16 (zero-padded)

Then do as slice:

B[128:]

# and get error:
...
  File "/home/thwu/miniconda3/envs/py310/lib/python3.10/site-packages/ray/rllib/policy/sample_batch.py", line 911, in __getitem__
    return self._slice(key)
  File "/home/thwu/miniconda3/envs/py310/lib/python3.10/site-packages/ray/rllib/policy/sample_batch.py", line 1122, in _slice
    stop_seq_len, stop_unpadded = self._slice_map[stop]
IndexError: list index out of range

Other values in B:

len self: 160 (zero padded)
len slice map: 129
slice_.start: 128
slice_.stop: None
slice_map: [(0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (1, 16), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (2, 32), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (3, 48), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (4, 64), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (5, 80), (6, 96), (6, 96), (7, 98), (7, 98), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (8, 100), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (9, 114), (10, 128)], 
stop: 160

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[kouroshHakha]

Stamped.