Deep Q Learning

[tejank10]

Implemented DQN model on CartPole-v1

[iblislin]

There is a CartPole env implementation in pure Julia: Reinforce.jl
Consider changing to it?

[MikeInnes]

Yes, the Python dependency is a bit of an issue here, but otherwise it looks great. Will review in more detail soon.

[iblislin]

@MikeInnes Do you have the committer bit of JuliaML? I want to drop support of julia 0.5 of Reinforce.jl...

[tejank10]

@iblis17 , Reinforce.jl (and OpenAIGym.jl) was my first option but I guess that's not updated. There were some issues with old functions-new functions there.

[iblislin]

That was my first option but I guess that's not updated. There were some issues with old functions-new functions there.

Could you file an issue on that repo (and we can move discussion)? I want to improve it.

[tejank10]

Cool, I'll file an issue

[tejank10]

@iblis17 , #6 in OpenAIGym.jl, which is built on Reinforce.jl

[iblislin]

OpenAIGym still requires PyCall 🤦‍.
Is there any environment spec of CartPole-v1?
There is an implement of CartPole-v0 in Reinforce.jl already. I wondering what the main difference is.

[tejank10]

Ohh yes, that's where I got the idea to use PyCall! I was getting confused between OpenAIGym.jl and Reinforce.jl as former is based on the latter. Sorry about that.

I have re-implemented the model using Reinforce.jl on CartPole-v0.

[iblislin]

Ohh yes, that's where I got the idea to use PyCall! I was getting confused between OpenAIGym.jl and Reinforce.jl. Sorry about that.

Haha. I just tried to install OpenAIGym, and it surpised me.

And it will be nice if we port CartPole-v1. (but I still cannot found any detail from googling, maybe I should read the source code.

[tejank10]

CartPole-v0 defines "solving" as getting average reward of 195.0 over 100 consecutive trials.

Whereas

CartPole-v1 defines "solving" as getting average reward of 475.0 over 100 consecutive trials.

From CartPole-v0 and CartPole-v1

[iblislin]

@tejank10 😎

[iblislin]

mutable struct ?

[tejank10]

For the CartPolePolicy?
Also, would like to know the advantage that would offer. AFAIK (and I may be wrong, as I am new to Julia), mutable struct for that would not have any fields.

[iblislin]

mutable struct CartPolePolicy <: Reinforce.AbstractPolicy end

Keyword type is deprecated syntax since Julia 0.6.

[tejank10]

Oh, I see. Thanks for that. I've changed it now. :)

[MikeInnes]

Is this still relevant, or did you move the code to flux-baselines?

[I actually don't think it will be as bad to have the extra dependency on Reinforce.jl once we have a Pkg3 setup, but we can figure that out later]

[tejank10]

Yes, it has been moved to Flux-baselines.

[MikeInnes]

Great, let's leave it there for now then.