Draft llamafile support

[k8si]

Background

This draft PR is a step toward enabling the use of local models in AutoGPT by adding llamafile as an LLM provider.

Related issues:

• Instructions for local models #6336
• Make it easier to use local LLMs by decoupling AutoGPT from dependence on OpenAI function calling #6947

Changes 🏗️

For full documentation of the changes involved in this PR, as well as instructions for running AutoGPT with the llamafile backend, see autogpts/autogpt/llamafile-integration/README.md.

Summary of changes:

• Add minimal implementation of LlamafileProvider, a new ChatModelProvider for llamafiles. Currently it just extends OpenAIProvider and only overrides methods that are necessary to get the system to work at a basic level.
• Add mistral-7b-instruct-v0 to OpenAIModelName/OPEN_AI_CHAT_MODELS registries. This is the only model currently supported by LlamafileProvider because this is the only model I tested anything with.
• Misc changes to app configuration to enable switching between openai/llamafile providers. In particular, added config field LLM_PROVIDER that, when set to 'llamafile', will use LllamafileProvider in agents rather than OpenAIProvider.

Limitations:

• Only tested with (quantized) Mistral-7B-Instruct-v0.2
• Only tested with a single AutoGPT 'task' ("Tell me about Roman dodecahedrons")
• Did not attempt extensive refactoring of existing components; I just added special cases as necessary
• Haven't added any tests for new classes/methods

PR Quality Scorecard ✨

• Have you used the PR description template?   +2 pts
• Is your pull request atomic, focusing on a single change?   +5 pts
• Have you linked the GitHub issue(s) that this PR addresses?   +5 pts
• Have you documented your changes clearly and comprehensively?   +5 pts
• Have you changed or added a feature?   -4 pts
  • Have you added/updated corresponding documentation?   +4 pts
  • Have you added/updated corresponding integration tests?   +5 pts
• Have you changed the behavior of AutoGPT?   -5 pts
  • Have you also run agbenchmark to verify that these changes do not regress performance?   +10 pts

[netlify]

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	e426766
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/66229c95467f0e0008b6ec35

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Swiftyos]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Review

⏱️ Estimated effort to review [1-5]	4, due to the complexity and breadth of the changes introduced, including new model provider integrations, extensive modifications to configuration and provider logic, and the addition of new scripts and documentation. The PR touches multiple core components and introduces a new LLM provider, which requires careful review to ensure compatibility and correctness.
🧪 Relevant tests	No
🔍 Possible issues	Possible Bug: The method check_model_llamafile in configurator.py uses api_credentials.api_base.get_secret_value() which might expose sensitive information in error messages. This could lead to security risks if the error messages are logged or displayed in an environment where unauthorized users can view them.
	Possible Bug: In LlamafileProvider, the method _create_chat_completion hard-codes the seed for reproducibility, which might not be desirable in all use cases and could limit the functionality of the model in generating diverse responses.
🔒 Security concerns	Sensitive information exposure: The method check_model_llamafile potentially exposes sensitive API base URLs in exception messages, which could be a security risk if these messages are logged or improperly handled.

Code feedback:

---

relevant file	autogpts/autogpt/autogpt/app/configurator.py
suggestion	Consider removing or masking sensitive information such as api_base from error messages in check_model_llamafile to prevent potential leakage of sensitive data. [important]
relevant line	raise ValueError(f"llamafile server at {api_credentials.api_base.get_secret_value()} does not have access to {model_name}. Please configure {model_type} to use one of {available_model_ids} or use a different llamafile.")

---

relevant file	autogpts/autogpt/autogpt/core/resource/model_providers/llamafile.py
suggestion	Remove the hard-coded seed in _create_chat_completion or make it configurable via method parameters or configuration settings to allow for more dynamic behavior. [important]
relevant line	kwargs["seed"] = 0

---

---

✨ Review tool usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

• When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

• With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.