-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A bunch of improvements for the classification skill #50
base: master
Are you sure you want to change the base?
A bunch of improvements for the classification skill #50
Commits on Dec 24, 2023
-
Improve utils.logs.print_dataframe()
1. Make number of rows to print a configurable parameter (default to 5). 2. Add parameter to enable index column printing. This implementation prints index of the original data frame if the passed data frame is derived.
Configuration menu - View commit details
-
Copy full SHA for dbcd295 - Browse repository at this point
Copy the full SHA dbcd295View commit details
Commits on Jan 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for c30988b - Browse repository at this point
Copy the full SHA c30988bView commit details -
agents.Agent: Estimate accuracy after the last prompt change.
Otherwise we don't know whether it improved the accuracy or not. We should eventually introduce smarter learning strategies. E.g. simple ones like not accepting changes that make accuracy worse. Or complex ones with genetic algorithms like in FunSearch.
Configuration menu - View commit details
-
Copy full SHA for 4fb234e - Browse repository at this point
Copy the full SHA 4fb234eView commit details -
agents.Agent: Skip skill output with accuracy above the threshold.
Accuracy threshold was ignored and unused originally which made training quite difficult in practical scenarios.
Configuration menu - View commit details
-
Copy full SHA for 5c4cb6b - Browse repository at this point
Copy the full SHA 5c4cb6bView commit details -
Fix environment feedback for skills with multiple outputs.
When providing feedback to the model, mention which output is wrong. Otherwise the model doesn't have enough information which of the outputs is correct/incorrect.
Configuration menu - View commit details
-
Copy full SHA for 77cb85e - Browse repository at this point
Copy the full SHA 77cb85eView commit details -
agents.Agent: Improve the teacher model prompt.
1. Phrase the prompt in more imperative manner liked by GPT models. 2. Instruct the teacher model to avoid unnecessary rephrasing of the prompt. With GPT-4 this makes it to make a lot less unnecessary changes. When a skill has multiple outputs, each skill output rewrite also changes wording of all the other outputs unnecessary distorting and degrading their performance. This phrasing significantly reduces such distortion but doesn't remove it completely. Running training cycles on each skill output separately solves this completely but is much much slower. Another potential solutions (I haven't tried it yet) is to collect feedback for all outputs and apply it all in a single go. More testing with real data is needed here.
Configuration menu - View commit details
-
Copy full SHA for 480a781 - Browse repository at this point
Copy the full SHA 480a781View commit details
Commits on Jan 6, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 689387b - Browse repository at this point
Copy the full SHA 689387bView commit details