Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new command for testing existing model behavior #85

Closed
russellb opened this issue Mar 22, 2024 · 3 comments · Fixed by #125
Closed

Create new command for testing existing model behavior #85

russellb opened this issue Mar 22, 2024 · 3 comments · Fixed by #125
Assignees

Comments

@russellb
Copy link
Member

Create a new command to the bot that takes all of the added questions in the given taxonomy PR and captures existing answers from the model. The results should be published similar to the generate output we already do.

I'm not sure if lab provides a shortcut to this. Adding one would make this a little easier on us. Otherwise, I assume we can feed input into lab chat to do it a bit more manually.

We could also talk to the OAI endpoint directly, but I think it's better to go through lab to make sure we're using the same default prompt.

@russellb russellb self-assigned this Mar 29, 2024
@russellb
Copy link
Member Author

First implementation using lab depends on instructlab/instructlab#765

@russellb
Copy link
Member Author

russellb commented Apr 2, 2024

First implementation using lab depends on instruct-lab/cli#765

This is merged.

@russellb
Copy link
Member Author

russellb commented Apr 2, 2024

First implementation using lab depends on instruct-lab/cli#765

This is merged.

However, it won't actually work for the bot because lab test currently only works on Mac. Here's the issue for adding it to Linux: instructlab/instructlab#659

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant