Reinforcement learning?

[nfreesto]

hey, I'm pretty new to AI and Neural Networks as a whole, so excuse me if I'm incorrect, but it seems like there is no currently implemented way to train a model using reinforcement learning. Is there a way to achieve this with currently implemented methods? Barring that, is that something that's in the plans for the future of this library?

Thanks for your help

[zaleslaw]

Hi, @NahatanIII thanks for the raised question. You are not the first person who is interested in RL with Kotlin.
Unfortunately, the current models are related to the classic Computer Vision domain and implemented such CNN architectures like VGG, ResNet, and MobileNet (I mean upcoming release 0.2)

But we are not limit this library to mainstream Deep Learning only and the RL package is possible in the future if it will have high users demand.

Could I ask you, what the basic RL models and for what goals or in what fields are you going to apply these models? For playing video games, for solving optimization tasks, or something else? Please, share more about your needs.

[dosier]

Nice work :D! One other thing I just noticed is that KotlinDL doesn't have GradientTape implemented. So maybe that should be a priority, once implemented and combined with your gym code, the full Actor-Critic example from keras should be implementable in KotlinDL right?

[d-lowl]

Right, yeah, it looks like, again, tf java api lacks GradientTape indeed.

[zaleslaw]

So, it is a great idea to start from the CartPole examples, but it will be hard without GradientType and low-level control of gradients. In my humble opinion, classic Keras is not designed well for the RL tasks.

But I want to share some ideas what could be starting points here:

1. Simple RL tasks could be implemented from scratch in Kotlin without TF backend usage; the simplest backpropagation for Dense layers could be added
2. Implement something that could be added later to KotlinDL without TF backend usage at all: Hidden Markov chains for example.
3. Train RL models in Python and try to use them for prediction in KotlinDL (under InferenceModel), for example
4. Wait for GradientTape support in TF Java API 2.x and KotlinDL integration with this API (I think it's 2022)
5. Wait for (or even collaborate on) basic NN implementation in pure Kotlin in KotlinDL during 0.4-0.5 releases.
6. Wait for PyTorch integration during 0.5-0.6 releases and use an alternative backend with full control on the training cycle
7. Actively prepare examples in KotlinDL excluding backprop implementation
8. Learn this repository with Kotlin examples to Satton book or this

[d-lowl]

Good points.

1/ Is sensible, but probably isn't enough for my application. (Might be totally good enough for many other uses though)
4/ I think is the way to go, so not to reimplement the wheel (i.e. the GradientTape) -- but point 3/ is probably what I'll end up doing
8/ I like these resources! From my side, I can also recommend Stanford's course on RL, for theoretical background (if anyone here is seeking that), available on youtube: https://www.youtube.com/watch?v=FgzM3zpZ55o&list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u

[zaleslaw]

I hope about GradientTape too, now there are a few problems with low-level C/C++ API for that, but the TF core team works on it.

[lukasgppl]

I'm also interested in deep reinforcement learning with KotlinDL.

Is there a possiblity by now, that allows me to implement a standard PPO for example? I'm still learning about KotlinDL, but I don't really see a way how I could implement the PPO objective. Is there a way to feed the calculated advantages into a custom loss function that includes the clip in every episode? Any workarounds (like only using one epoch in the training with .fit)?

Also, it would be crucial to export and import trained models. Is there a good way to do this? saveModel() and loadModel() don't seem like they are made for this purpose.

I'd love to contribute to a simple drl example if I can get some assistance. I already wrote some code for it but I'd need some help with some details @zaleslaw.