New KL Divergence Upgrade #2
magiccodingman
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I've not been posting many new models because my code has gone through another big upgrade. I didn't want to continue posting new models on architecture I thought I'd upgrade.
TLDR, I think I've successfully added KL Divergence. System still uses PPL, but it weighs a hybrid score for brain damage by weighting KLD by 70% and PPL by 30%. Thank you to everyone who provided feedback on KLD.
Additionally I've made a big upgrade to how the system determines what to build to:
1.) Decrease the number of samples required to build.
2.) Increase the odds of finding better results.
On some of my early tests, compared to the original numbers achieved on the uploaded models. I'm now reliably seeing new hybrid mixes that achieve 20% increased TPS and 30% to 70% better precision retention compared to the old system. These increased results are not necessarily due to the KL divergence but significant changes made to how the system predicts combinations and determines hybrids to sample.
Should hopefully start dropping more models again with an updated wiki on the new process soon.
Beta Was this translation helpful? Give feedback.
All reactions