Feature/param reset

[joshuaspear]

Closes #326

[joshuaspear]

@takuseno I am still working on the tests but please let me know if you think the implementation is a reasonable approach

[joshuaspear]

@takuseno assuming you're happy with the general approach of using the epoch_callback to inject the parameter reset functionality - I wondered if you could recommend a better approach for obtaining the encoder and fc layers which follows static typing?

[takuseno]

@joshuaspear Thanks for the proposal! For now, can we make this as an experimental feature? I imagine something like this:

file location

Let's make experimental directory:

d3rlpy/experimental/parameter_reset.py

usage

Just rough illustration:

import d3rlpy

# e.g. 50% reset, every 1000 gradient steps
parameter_reset = d3rlpy.experimental.ParameterReset(reset_ratio=0.5, reset_interval=1000)

cql = d3rlpy.algos.CQLConfig().create()

def callback(algo, epoch, total_step):
    parameter_reset(algo.q_function, total_step)
    
cql.fit(..., callback=callback)

In this way, we can use the existing callback to inject reset operation.

Reset is still under investigation in RL community. Once it gets more mature, we can lift this from experimental.

[joshuaspear]

Makes sense - will have a go next week :)