Curriculum file with measure as "progress"

[menondj]

I was running a very long 24 hour test for a tic-tac-toe game with max_steps 18 million and a curriculum file with measure as "progress" and threshold as 0.7. I understood that the lesson would kick in at 12.6 million. What I want to be able to do is change the parameters after 70% progress.
But it kicked in only towards the end of a test, here is a snapshot of PPO output:
Step: 17980000. Mean Reward: -0.106296445069. Std of Reward: 0.672540501635.
Step: 17990000. Mean Reward: -0.100342190016. Std of Reward: 0.674290931663.
INFO:unityagents:
Lesson changed. Now in Lesson 1 : defence_penalty -> -0.75, defence_reward -> 0.75
Step: 18000000. Mean Reward: -0.0938710993269. Std of Reward: 0.674583112292.
Saved Model
Saved Model

Is there anything I'm doing wrong? The tensor board data showed no other discrepancy and it seemed to run well except for the curriculum file:.
{
"measure" : "progress",
"thresholds" : [0.7],
"min_lesson_length" : 2,
"signal_smoothing" : true,
"parameters" :
{
"defence_reward" : [0.25, 0.75],
"defence_penalty" : [-0.25, -0.75]
}
}

BTW, awesome job on ML agents! Has opened up so many possibilities!

[awjuliani]

Hi @menondj,

Thanks for trying out Curriculum Learning! This is a little counterintuitive, but we only check if the measure is met when the academy is reset. It may be that your academy only resets very infrequently. I would recommend setting a smaller max step for the academy. Please let me know if that works!

[menondj]

thanks @awjuliani for the clarification! I misunderstood "progress" as max_steps from the PPO file.
Its a 3x3 tic-tac-toe, so max_steps before an academy reset can only be 9 :--). And the academy resets very often, even before 9 is achieved.

My initial understanding of the whole curriculum file process was that once the learning has progressed sufficiently ( as can be viewed from the trends on the tensor board side), I should be able to change the behaviour using the curriculum file to make it converge faster e.g. teach it offence/defence with a higher penalty once it learnt how to win. Would this be a possibility in future?

The tic-tac-toe is learning, but takes forever because most of the time its blocking on sockets going back and forth Unity & Python. Is there some local caching possible in cases where it knows it has reached an optimal value for a state? Would greatly help board games.

[shihzy]

Thanks for reaching out to us. Hopefully you were able to resolve your issue. We are closing this due to inactivity, but if you need additional assistance, feel free to reopen the issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.