[RLlib] Make actions sent by RLlib to the env immutable.

[simonsays1980]

Why are these changes needed?

As actions could be mutated by users in the environment step() function and produces hardly traceable errors a better solution is to warn users not to mutate the actions (directly). This PR gives a solution by setting the numpy flag WRITEABLE to False in the _env_runner() function whn calling the sampler.

The last PR I sent throwed many errors in tests due to the fact that actions can be of different type depending on the action space. This PR includes now immutability for any type of action using tree.map_structure() and the MappingProxyType from Python to make dicts immutable.

Related issue number

#23890

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simonsays1980]

@gjoliver

As said the errors before were due to the fact that actions from different action spaces could of non-numpy type. In this case in has to be accounted for the structure. This PR should now work for any action structure, as MappingProxyType makes dicts immutable, setflags(write=Fale) deal with the numpy.ndarrays and Tuple and other basic types like int, float, etc. are immutable by nature.

[gjoliver]

this looks awesome. what do you think we move this into rllib/utils/numpy.py?
I will try to reuse it for some connectors.

[simonsays1980]

this looks awesome. what do you think we move this into rllib/utils/numpy.py? I will try to reuse it for some connectors.

@gjoliver Thanks! Yes I can do this. Regarding the connectors the dataclass decorator could also become interesting as it makes classes immutable by the __hash__() function - I was thinking about using it here, but it was an overload

[gjoliver]

thanks, awesome tech :)

[sven1977]

Awesome PR @simonsays1980 ! Thanks for going the extra mile here after our concerns about the first "deepcopy" solution.
Really appreciate this. Looks super neat with the tests and everything.