Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Models not being saved #31

Closed
Ujwal2910 opened this issue Jul 1, 2019 · 6 comments
Closed

Models not being saved #31

Ujwal2910 opened this issue Jul 1, 2019 · 6 comments
Labels
bug Something isn't working

Comments

@Ujwal2910
Copy link

I have been trying to run an A3C chainer script on the environment and although the learning is taking place and is evident in the rendering but the model is not being saved.
When I replace the environment with Cartpole it works fine and the model gets saved.
Could you please update?

@AntonRaichuk
Copy link
Collaborator

Could you be a bit more specific about what script exactly do you run? Since we do not provide any runner script for A3C.

Given that this repo provides an environment and checkpointing of models usually happens in RL learning scripts, it seems like the problem is likely on your side.

@Ujwal2910
Copy link
Author

I ran the a3c script from chainerRL - https://github.com/chainer/chainerrl
The problem is that when any other different environment is chosen the checkpointing of models works perfectly fine.
In my log, I keep getting the message that the model is being saved but actually nothing is being saved or written.
We have been using this code- https://github.com/Ujwal2910/RL-in-Football
If you could help us figure why only with this environment such an issue is happening it would help us a lot in the research.

@AntonRaichuk
Copy link
Collaborator

Could you share how the log looks like when you run Cartpole vs GFootball?
Also please share "ls -l" output of the outdir for both cases.

@Ujwal2910
Copy link
Author

for the Cartpole models the log returns-
Screenshot from 2019-07-01 16-19-05
and for the Football env it returns-
Screenshot from 2019-07-01 16-20-42

For the result of ls -I the result is as follows-

(base) ujwal@ujwal-Precision-7920-Tower:~/PycharmProjects/Gfootball/football_models$ ls -l
total 4
(base) ujwal@ujwal-Precision-7920-Tower:~/PycharmProjects/Gfootball/football_models$ 

and for cartpole it gives-

(base) ujwal@ujwal-Precision-7920-Tower:~/PycharmProjects/Gfootball/cartpole_models$ ls -l
total 620
drwxr-xr-x 2 ujwal ujwal   4096 Jul  1 16:18 200_finish
drwxr-xr-x 2 ujwal ujwal   4096 Jul  1 16:18 4024_except
drwxr-xr-x 2 ujwal ujwal   4096 Jul  1 16:18 best
-rw-r--r-- 1 ujwal ujwal 310639 Jul  1 16:18 model.npz
-rw-r--r-- 1 ujwal ujwal 306909 Jul  1 16:18 optimizer.npz
-rw-r--r-- 1 ujwal ujwal    182 Jul  1 16:18 scores.txt
(base) ujwal@ujwal-Precision-7920-Tower:~/PycharmProjects/Gfootball/cartpole_models$ 

@AntonRaichuk
Copy link
Collaborator

Thanks for pointing out this issue. After debugging it is indeed a problem in our environment that it changes the current working directory. We are working on a fix.

For now I'd just recommend using absolute paths for your logging. It should work that way.

@qstanczyk
Copy link
Collaborator

We just pushed a new release, the fix is included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants