Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmark performance #97

Merged
merged 4 commits into from Dec 17, 2020
Merged

Update benchmark performance #97

merged 4 commits into from Dec 17, 2020

Conversation

Derek-Wds
Copy link
Contributor

Description

Motivation and Context

How Has This Been Tested?

  • Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
  • If you are adding a new feature, test on your own test scripts.

Screenshots of Test Results (if appropriate):

  1. Pipeline test:
  2. Your own tests:

Types of changes

  • Fix bugs
  • Add new feature
  • Update documentation

| TFT | Alpha158 (with selected 20 features) | 0.0344±0.00 | 0.2071±0.02| 0.0103±0.00 | 0.0632±0.01 | 0.0638±0.00 | 0.5845±0.8| -0.1754±0.02 |
Copy link
Collaborator

@you-n-g you-n-g Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isTFT‘s annual return so stable while it's information ratio so variable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wendi calculated the information ratio's std in a wrong way. It should be 0.08 instead of 0.8. I have just updated it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I calculated the mean of two 5-runs and typed by hand, and missed a "0"...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wendili-cs make sure your rank IC. TFT has high IC & annual return while it's IC is low.
Please double check it though such results ares possible.

@you-n-g
Copy link
Collaborator

you-n-g commented Dec 16, 2020

@Derek-Wds Are these results generated by repeating 20 times of experiments?

@Derek-Wds
Copy link
Contributor Author

Derek-Wds commented Dec 17, 2020

@Derek-Wds Are these results generated by repeating 20 times of experiments?

It's not for now. The running for Alpha360 hasn't finished yet. I will update the results when the running's done.

@you-n-g you-n-g merged commit 1c0bb2f into microsoft:main Dec 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants