Custom model for Soft-Actor-Critic in [rllib]

[sahikagenc]

Is it possible to provide a custom model to SAC from a configuration file such as the case for model parameter as follows:

                    # Model options for the Q network(s).
                    "Q_model": {
                        "model": "SoftQ_V2_model_sac",
                        #"fcnet_activation": "relu",
                        #"fcnet_hiddens": [256, 256],
                    },
                    # Model options for the policy function.
                    "policy_model": {
                        "model": "Policy_V2_model_sac",
                        #"fcnet_activation": "relu",
                        #"fcnet_hiddens": [256, 256],
                    },

[sven1977]

Great question, @sahikagenc . Let me try to make this work with the existing model building APIs. ...

[sven1977]

Btw, did you try simply sub-classing SACTF|TorchModel and then implement your own get_q_value, get_policy_output, etc.. logics?

[sven1977]

There is also a bug in SAC, which makes it not learn the "state-preprocessor" (e.g. when you use a CNN in front of the policy- and Q-nets). The problem is in SAC's compute_and_clip_gradients (tf) and optimizer_fn (torch), where the optimizers are told to only optimize the policy and Q-nets, but never the pre-network.
Fixing this now ...

[sven1977]

This is now fixed (via this PR: #13522). You can provide custom models to SAC via the following options:

1. custom Q-model:

config:
    Q_model:
        custom_model: [your registered custom Q-model class]

2. custom policy-model:
   config:
   policy_model:
   custom_model: [your registered custom p-model class]

3. custom SAC model (as a whole):

• sub-class SACTF|TorchModel
• override the new build_policy_model and build_q_model methods in there to return whatever custom model(s) you want.

Closing this issue now.