Multi-objective reinforcement learning

[borisdjartov]

Hi guys, i really love the work you have done here and am currently using xcsf in my research. However, i was just wondering if you have any plans or maybe an idea how i can employ this for a multi-objective reinforcement learning problem. Is there a way to have the reward be a vector instead of a scaler and how could i implement something like the NSGA-II algorithm. Ideally this should be done in python since my knowledge of C is non existent. I would love to hear from you!

Keep up the good work!

Kind regards,
Boris

[rpreen]

I haven't really done much multi-objective learning and while I think there is some old work with a multi-objective XCS I'm not sure it was multi-objective in the current sense.

If you look at the cartpole example, it uses the predict() and fit() functions to update the reward during experience replay:

https://github.com/rpreen/xcsf/blob/9804b0d44cbde418dcb0b1a748e7924e4bd496bd/python/example_cartpole.py#L118-L129

In that example, XCSF was initialised with n_actions=1 which means there are effectively no action sets - it's just creating a match set to decompose the input space and then each classifier has a prediction y_dim=2 which is an output vector set to the number of possible actions (similar to deep-Q neural nets) - each value in the vector therefore represents the expected payoff for performing each action (and hence the best action is the argmax). And you can see that the delayed reward $\gamma$ is calculated in that Python function before calling fit() to update the prediction vectors. So I don't know if you can just adjust the reward calculation there to do multi-objective learning.

If it involves something more complicated then it probably means having to start hacking with the underlying C code.