New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Confuse about the success rate #202
Comments
Another question! In ml45/10, we set 45 train environments, what is the relationship between the class num and the task num? If the task num is also 45/10, which means every environment only has one task? |
Hi @jp18813100494, thanks for checking out Metaworld! Let me try to answer your questions in order:
Let me start by saying that From there to compute the |
I'm not quite sure if I understand your second question, so let me know if this answer doesn't satisfy you: In the benchmark ml45, which is made up of 45 different environments, there are more than 45 tasks: Depending on the benchmark that you are using, there is either 1 unique task per environment in that benchmark, or 50 unique tasks. For ML benchmarks such as ML1, ML10, and ML45, there are 50 unique tasks per environment in those benchmarks. For MT benchmarks such as MT1, MT10, and MT50, there is only 1 unique task per environment in those benchmarks. I'm going to close this issue for now, but feel free to @ me (avnishn) and ask any more questions! |
Thanks for your sincere reply! As you said, for ML benchmarks such as ML1, ML10, and ML45, there are 50 unique tasks per environment. What is the num of the training tasks in the ML45 setting? I am clear of your setting of metaworld, but I don't know the detail of setting in your algorithm, MAML, RL, PEARL. Can you tell me the total num of your train task and test task for 50 environments for ML45 in PEARL? 45,5 or 45 50,550? |
Hmm. So the number of train tasks and test tasks for each benchmark doesn't change based on the algorithm. Ml45 has 45 training environments and 5 testing environments. ML10 has 10 training environments and 5 test environments. Each one of these environments has 50 unique task configurations. This would mean that in ML45. There are 4550 train tasks and 550 test tasks. A meta learning algorithm that is being evaluated on ML10 or ml45 would train on the 10 and 45 environments, and each one of those environment's 50 unique tasks, and then would be evaluated on their meta adapt to the 5 test environments in ML10 and ML45. These test environments still have 50 unique tasks each, and each should be sampled during the many meta adapt test rollouts that will be completed throughout an algorithm's evaluation. In terms of the code, changing an environment's task is as easy as calling the function A question one might have is: how often should This may have been a bit more than you were looking for in an answer, but I hope that everything you are looking for is here! |
@avnishn
Therefore, it seems that in both MT10, MT50, we have 10 or 50 environments each with 50 tasks, which is contradicting to what you mention that only one unique task per environment. |
I am new to metaworld! Thank you for the configuration of such meaningful and complex projects. I have some questions about the metaworld. In the project setting, does the success rate represent the success in one step or one episode? Because in your project, the agent will reach the max path length in one episode, such as 150, so does the success rate represent the average success rate in 150 steps or the success rate in the last step? Can you give me some points about it.
The text was updated successfully, but these errors were encountered: