Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

League Evaluation Metric #18

Closed
5 of 11 tasks
zxzzz0 opened this issue Aug 4, 2021 · 2 comments
Closed
5 of 11 tasks

League Evaluation Metric #18

zxzzz0 opened this issue Aug 4, 2021 · 2 comments
Assignees
@PaParaZz1
Labels
enhancement New feature or request P1 Issue that should be fixed within a few weeks

Comments

@zxzzz0
Copy link

zxzzz0 commented Aug 4, 2021

Added this issue as suggested by @PaParaZz1.

  • I have marked all applicable categories:
    • exception-raising bug
    • RL algorithm bug
    • system worker bug
    • system utils bug
    • code design/refactor
    • documentation request
    • new feature request
  • I have visited the readme and doc
  • I have searched through the issue tracker and pr tracker
  • I have mentioned version numbers, operating system and environment, where applicable: N/A

TrueSkill is a ranking metric developed by Microsoft for game matchmaking. Unlike ELO which just measures one agent's strength, TrueSkill can measure both strength and stability. Each player starts with mu=25.000 and sigma=8.333. Former one (mu) measures strength and the latter one (sigma) measures stability. After receiving payoffs of one matching, mu and sigma will be updated accordingly from the TrueSkill API. Final agent's score can be defined as mu - 3 * sigma to take both strength and stability into consideration.

Currently this metric is missing in the league demo. It would be better to add it.
@PaParaZz1 PaParaZz1 added enhancement New feature or request P1 Issue that should be fixed within a few weeks labels Aug 4, 2021
@PaParaZz1 PaParaZz1 self-assigned this Aug 5, 2021
@PaParaZz1 PaParaZz1 mentioned this issue Aug 10, 2021
Merged
3 tasks
@PaParaZz1
Copy link
Member

TrueSkill demo result in tensorboard Screen Shot 2021-08-10 at 3 23 42 PM

@zxzzz0
Copy link
Author

zxzzz0 commented Sep 3, 2021

This issue has been solved in PR #22.

@zxzzz0 zxzzz0 closed this as completed Sep 3, 2021
@PaParaZz1 PaParaZz1 mentioned this issue Sep 3, 2021
Closed
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@PaParaZz1 PaParaZz1

Labels
enhancement New feature or request P1 Issue that should be fixed within a few weeks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PaParaZz1 @zxzzz0