Round 2 #164
Round 2 #164
Comments
@kidzik What are the rules regarding advancing to Round 2? |
When are you planning to switch to branch v2.1? |
@mattiasljungstrom it'll be around 10 seconds (I've just updated the initial post) |
That's a rough formula for requested velocity vector (2d projection) in the Round 2: We've received the Google Cloud credits and we will send them out tomorrow. |
Fixed, thanks! |
Will the velocity be clipped to prevent it from going negative? |
We plan to leave the possibility of negative velocity (which happens with very low probability). |
Version 2.1 is now merged with the master branch and deployed on the grader. The reward for the second round might still change. We extended the first round until October 20th. |
https://github.com/stanfordnmbl/osim-rl/blob/master/osim/env/osim.py#L527 |
I think that a target velocity should be given in advance so that it can know a goal and take an action pro-actively. penalty += (state_desc[“body_vel”][“pelvis”][0] - state_desc[“target_vel”][0])**2 |
Why does the "target_vel" not include in observation state when using the summited server? Is the difficulty=0 with the round 2? |
@phunghx: If you set the difficulty to 0, a target velocity is a constant, so it may not need to specify. |
@beforethefirst Sure, from my computer, I set the difficulty to 1 and get the "target_vel" in the observation state. But when I submit to the server for grading, I did not get the "target_vel". So I guess there are some problems with the server code or they also use diificulty=0. @beforethefirst Have you tried to submit your model to the server? |
@phunghx right now the reward on the server is the same as it was before ( |
@kidzik Thank you very much |
@kidzik Currently, I am trying to train an agent to learn smooth transition, so it needs to know the target velocity when it takes an action. It might not be that critical, but could you update the code earlier? Or, could you make sure that it will be fixed when Round 2 begins? |
@kidzik WRT the google cloud credits, is there still time to get a positive score and be eligible for the credits? In order to qualify for them to we simply have to place before the end of round 1? |
@JohnnyRisk we will be able to award more credits (at least 50 x 250$). We will give details as soon as we know that everything went well in the first batch. |
now, the observation is 160 dimensional. as my placeholder is <None, 158>, if I still want to submit my current model, how to modify the observation returned by client? how about obs[:158]? Thanks |
@joneswong The two new dimension shall occurs at the index 156 and 159, I think you can remove them manually. |
@wangzhengfei0730 thanks for that, it was giving me a headache, I had the same problem as @joneswong. |
@kidzik The way the reward function is setup for round 2 the model doesn't need to turn in the movement direction, it only needs to move sideways. Is this what you intended? I had expected that the model should turn in the movement direction. |
@mattiasljungstrom good point! We hypothesize that moving sideways will be suboptimal due to energy constraints. Hopefully, at some point solutions to the challenge will serve as a tool for hypothesis testing (or at least screening for hypotheses). |
@kidzik Interesting, thanks for the reply! |
Some updates regarding qualifications to the second round:
|
Hi! I have a question about observation vector. Right now it is 160-dim independently of difficulty level. Or am I missing something obvious here? For me it seems that tasks in Round 2 and Round 1 are very different, so doesn't make much sense to concentrate on Round 1 task and instead invest more time in Round 2. |
Do submissions to the live Round 2 leaderboard count towards the submissions limit? Or will only the full docker submissions count for that? |
@spiglerg they will not count towards the limit. Current round 2 is preliminary -- the final results will be based on docker submissions. |
@kidzik Thank you very much. Just to be clear, all the people are qualified for the second round, and it is the score of the second round that will qualify people (in case of capacity limitation) for the final docker submission. |
@yobobobo Right, we extended the deadline a little bit since then. |
all the homepages have not been updated and I am very confused. before 21, oct, may I train models with round 2 env? If it is possible, no example in the homepage. just by for submission, no example also. There have been some submissions appearing in the leader board of round2. How to configure Could anyone help me? Thanks! |
@joneswong Yes, you can train models for Round 2, the evaluation won't change. |
@kidzik thanks for your reply!
I found that there are many ~9000 in the round2 leaderboard. How to submit to round2? (i.e., how to configure the crwdAI client? I cannot find any example.) |
@wangzhengfei0730 hi zhengfei, could you teach me how to submit to round2? I didn't find any example in the homepage. Besides, when I set difficulty = 1, the target_vel is constantly [1.25, 0.0, 0.0]. could you also tell me how to use round2 env? I am very confused with current situation...actually, I have been tuning my agent with round1 env up to now. |
@joneswong you need to submit to the right server as you can see here 3ceadcc |
@joneswong When set the difficuly as 1, the target_vel will change every 300 steps. I think your constantly target_vel caused by random seed. |
@kidzik @wangzhengfei0730 thanks for your help! I have completed training and submission by reading the code and following your explanation. Thanks again. @kidzik Frankly speaking, I am very disappointed at the organization of this competition. The following announcement: Round 2 (test run): 1.10.2018 - 28.10.2018 appeared just around three days ago. I keep checking https://www.crowdai.org/challenges/nips-2018-ai-for-prosthetics-challenge every 2-3 days and it always Round 2: 21.10.2018 - 28.10.2018 (tentative) Besides, the qualification condition was not declared officially (maybe just in this thread). I was afraid that there will be more than 50 teams above 1000+ and had kept tuning my agent for round 1 until it reaches high rankings. I just turned to round2 yesterday (T_T). We sincerely suggest to postpone the ddl of round2. Could you help us to communicate this with crowdAI? It would be more friendly to competitors like us who have not participated in the nips2017 competition. As you can see, most of the experienced competitors are more familiar with this channel (i.e., via a github issue). Anyway, thanks for your efforts! This simulator is awesome and brings much fun to us. We will keep our RL research upon osim. |
Thank you @joneswong for your comment. We moved the deadlines for 1 week (both round 1 and round 2) and that’s the only change. We added the current test version of the round 2 only for convenience sorry for the miscommunication. NIPS is already very soon and we need some time to arrange the logistics (visa invitation letters etc), so it might be difficult to push it further. |
@kidzik I see. For now, how to qualify for docker submission? topK in round2 leaderboard or 9700+? Besides, Is there any entry to register for docker submission? It seems that crowdAI has no means to communicate with me. Thanks. |
We qualify top 50 solutions from Round 1, we've updated the website to reflect that. |
@kidzik, what is the exact end time for round 1? 29 Oct 10:00 UTC as stated on crowdAI? |
Yes, the time on crowdAI is official. We were shooting for the midnight October 28 Hawaii time. |
Thanks a lot for confirming it:-) |
@kidzik hi, as the ddl is approaching, what's the condition of qualifying for docker submission? |
We accept top 50 participants from the first round. |
@kidzik Thansk. It seems that I misunderstood that top50 from round1 qualify for round2 interaction (client-server) and topK from round2 qualify for docker submission. So I was afraid that my ranking was no high enough to qualify for the docker submission. |
I admit we were not 100% clear about that. But yes, the client-server test round 2 is meant to be just a test. |
Is docker submission prepared? or should i use round1's submission method for round2? |
@whikwon check this repo for submission for round2: https://github.com/crowdAI/nips2018-ai-for-prosthetics-round2-starter-kit |
Hmm I followed the instructions for the final round2 submissions, but I get this error in the submission link: CrowdAI Submission Failed " Submission failed : The challenge is not open for submissions at this time. Please check the challenge page at www.crowdai.org " It must be something trivial, but any help would be appreciated. :) |
@spiglerg please try again now. |
Same problem. :/ |
Could you try 1 more time @spiglerg |
It seems to be working now, thanks a lot! :) |
Anybody get trouble with test submission? I was following the instruction and got error in ./run_local_grader.sh |
I changed the name to 'redis2' in the run_local_grader script and it worked. |
Please note that there is also a discussion and a 'support group' on the gitter channel: https://gitter.im/crowdAI/NIPS-Learning-To-Run-Challenge (some of quick small issues might have been solved there) |
The reward in the second round:
Other updates:
The text was updated successfully, but these errors were encountered: