categorize internal jobs and expose a mechanism for the user to specify/override what model/api gets used for different categories

[dmarx]

followup thought from #36

I haven't looked into this yet to see if it's already a configuration option, but it would be useful if the user could specify different models for different components of this system. concretely, gpt-3.5-turbo would probably be sufficient for the validation steps. If the action could be configured such that the more expensive API (gpt4) is used only for the main generations and a cheaper API could be used for stuff like the validation steps, that would potentially have a lot of benefits (cheaper, faster, less likely to get rate limited).

[irgolic]

Currently the model config option applies to all LLM calls. All agents should have their own model, as well as other config options. I'll open a PR implementing the refactor soon.

[irgolic]

Sorry for the long pause, I've implemented a refactor that allows agents to have their own config options, but haven't yet implemented picking different models for different actions. This will come with a refactor into an "action choice engine".

I did some testing, and I experienced a pretty bad performance drop with gpt-3.5-turbo, even for parsing steps. Although, earlier this week we changed how we were specifying instructions in guardrails, and it seems to greatly improve gpt-3.5-turbo performance; I'll try again to see if we can offload parsing steps to it.

[dmarx]

yeah forget gpt-3.5-turbo. I think it's just too unreliable for code generation. may as well focus on using whatever tools are most likely to work rather than wrestling with 3.5