Eager Execution Support

[karanchahal]

Having the code base in some dynamic graph style library would be great. Pytorch does this well, but since google has released eager execution, having the codebase in eager execution mode will be really nice to have since the codebase is a learning tool. I was having a bit of trouble understanding the session based control of tensorflow (coming from a pytorch background) for the vanilla policy gradient, inspite of it being written really well :')

I would be happy to slowly convert some of the codebase to support eager execution. Do you think this would be useful ?

[jachiam]

Hi @karanchahal! I'm tentatively planning to spend a month sometime around April or May making extensive updates to Spinning Up, which will likely include writing eager execution versions of the current algorithm implementations. I think it would be a great exercise for you to write your own implementations in eager mode, and if you build it I'm happy to link to it somewhere! But just to appropriately calibrate expectations, merging rewritten implementations into the core codebase is unlikely.

[karanchahal]

Alright no problem :). I was thinking of converting the codebase to pytorch too and maybe that would be better as a standalone thing. I'll let you know when I complete it :)

[kashif]

@karanchahal can you also have a look at https://github.com/kashif/spinningup-pytorch where all the algorithms are implemented now ... would love some feedback as I continue to clean it up

[karanchahal]

Oh this is great, I didn't realise some one is already trying to do this. I'll look into it. Did you observe any difference in the performance of the pytorch algo compared to the tensorflow one ? (for simple gym envs like cartpole etc). I am quite curious to know if there would be any perf difference.

[karanchahal]

I cloned your repo and ran the vpg algo and compared the perf with the tensorflow version. I did an average of 5 runs to take care of the random seed and I saw some interesting results

1. Tensorflow: Avg Episode Return 81
2. Pytorch: Avg Episode Return 31

Why do you think this might be the case.
@jachiam what do you think about this ?

Disclaimer: I haven't read your code thoroughly @kashif so there might be some very small mistake. But is diff in performance of RL algos substantial in tf and pytorch ?

[kashif]

@karanchahal perhaps close this issue and make an issue on my repo and we can discuss it there. Thanks!

[karanchahal]

Yes, apologies for this @jachiam

[jachiam]

No worries @karanchahal! Thank you both for the insightful discussion and for sharing resources.

[EduardoGRocha]

@jachiam can you be more precise on "merging rewritten implementations into the core codebase is unlikely." it means that the algorithms using eager execution are going to be structured differently? I need tensorflow 2 support for my project and would be willing to put some effort to rewrite the current algorithms to tensorflow 2.

It would be great if my code has any chance of being merged in the end

Edit: I won't necessarily rewrite line by line just to get the eager execution working. My main question is if the code is open for contributions and how one can possibly get involved