Reward thresholding not working

[alexchacon2341]

I was able to get curriculum to work using a test with the WallJump environment. For some reason, however, the training doesn't seem to register when the thresholds have been reached when using a nearly identical .json file in a GridWorld environment.

Has anyone else experienced a similar issue? Could it have something to do with using Visual Observations or On Demand Decisions?

[xiaomaogy]

Could you give more detail on how "the training doesn't seem to register" please?

[alexchacon2341]

Yes, of course. When I use curriculum training with the WallJump environment, I enter the following command in my terminal using this wall.json file: Everything works perfectly -- I am notified in the terminal once the first threshold of 3000 steps has been reached and the curriculum proceeds to Lesson 1. The wall begins to appear in the training, exactly as expected. When I attempt to do the same with GridWorld, however, the curriculum seems to be ignored entirely. Here are my commands and json file for that environment: Please note that while this particular json file uses rewards as its measure instead of progress, I have also attempted to train using progress as my measure with identical thresholds and max steps as I used in the WallJump environment. Even then, the curriculum did not register in the GridWorld environment. It seems that, regardless of how I set up my json file, the training skips over the thresholds as if no curriculum is being used. In the following training, you can see that, regardless of whether the json file is using 3000 steps or two appearances of a mean reward greater than 0.5 as a threshold, the criteria has been met. For some reason, however, the lesson is not increased: I am at a loss as to what could be causing this difference. As I mentioned in my post, I am using Visual Observations and On Demand Decisions in the GridWorld environment. I can't imagine these would affect the curriculum, but also can't think of any other factors that would disrupt the training. Thanks in advance for your help, I greatly appreciate it. Sincerely, Alex Chacon

…

On Wed, Jun 20, 2018 at 5:41 PM, Vincent(Yuan) Gao ***@***.*** > wrote: Could you give more detail on how "the training doesn't seem to register" please? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#895 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Adg71wwG0CV-cF4p1OQ5My6vIVd0gzGNks5t-sGRgaJpZM4Uuvu9> .

[xiaomaogy]

@alexchacon2341 It seems that I cannot see the pictures you attached.

[alexchacon2341]

Sorry, I responded via email. I'll try replying in GitHub itself with the images.

…

On Wed, Jun 20, 2018 at 6:39 PM, Vincent(Yuan) Gao ***@***.*** > wrote: @alexchacon2341 <https://github.com/alexchacon2341> It seems that I cannot see the pictures you attached. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#895 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Adg71z6hR7a7ZwFBrAhhTijNqHVAZTCaks5t-s86gaJpZM4Uuvu9> .

[alexchacon2341]

Yes, of course.

When I use curriculum training with the WallJump environment, I enter the
following command in my terminal using this wall.json file:

Everything works perfectly -- I am notified in the terminal once the first
threshold of 3000 steps has been reached and the curriculum proceeds to
Lesson 1. The wall begins to appear in the training, exactly as expected.

When I attempt to do the same with GridWorld, however, the curriculum seems
to be ignored entirely. Here are my commands and json file for that
environment:

Please note that while this particular json file uses rewards as its
measure instead of progress, I have also attempted to train using progress
as my measure with identical thresholds and max steps as I used in the
WallJump environment. Even then, the curriculum did not register in the
GridWorld environment.

It seems that, regardless of how I set up my json file, the training skips
over the thresholds as if no curriculum is being used. In the following
training, you can see that, regardless of whether the json file is using
3000 steps or two appearances of a mean reward greater than 0.5 as a
threshold, the criteria has been met. For some reason, however, the lesson
is not increased:

I am at a loss as to what could be causing this difference. As I mentioned
in my post, I am using Visual Observations and On Demand Decisions in the
GridWorld environment. I can't imagine these would affect the curriculum,
but also can't think of any other factors that would disrupt the training.

Thanks in advance for your help, I greatly appreciate it.

Sincerely,
Alex Chacon

[vincentpierre]

I think the problem is that the grid world academy does not reset (only the agent does). Try setting an academy max step in the grid world. Please let us know if this helped.

[alexchacon2341]

Are you sure the GridWorld academy doesn't reset? This is the agent reset function from GridAgent.cs: public override void AgentReset() { academy.AcademyReset(); } Is this not causing the academy to reset? Thanks, Alex

…

On Thu, Jun 21, 2018 at 4:13 AM, vincentpierre ***@***.***> wrote: I think the problem is that the grid world academy does not reset (only the agent does). Try setting an academy max step in the grid world. Please let us know if this helped. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#895 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Adg714HKUjhMsSHqd72fwrwykLXlQ5ZTks5t-1WZgaJpZM4Uuvu9> .

[vincentpierre]

Can you try with setting a max step. There is a possibility that the academy resets without being set to done. This means that python might not realize that the academy is done and has reset.

[alexchacon2341]

Sure. I'm running a training session at the moment, but I'll run another one with a max step after. Just to be sure we're on the same page, this is where I should be setting the max step, correct?

[vincentpierre]

Yes

[alexchacon2341]

OK, great. I'll run the training and get back to you ASAP. Thanks again!

[alexchacon2341]

No luck, I'm afraid. With a max step of 30,000 and the first threshold at 0.1, the lesson should have increased at 3,000. Here is my data:

[marcelwessely]

Same problem here. Curriculum does not jump to next step. It is possible to start every lesson by handy by using --lessong=[0,1,2...], but there is no next lesson by the curriculum.
Reward and progress mesure do not work.