Local LLM Backend #116
Local LLM Backend #116
Conversation
llm-chain-local/src/executor.rs
Outdated
| } | ||
| } | ||
|
|
||
| pub fn load<M: llm::KnownModel + 'static>( |
There was a problem hiding this comment.
Which one? The lifetime? If I remove the lifetime, it breaks and says the parameter M may not live long enough...
There was a problem hiding this comment.
Ah meant the whole function :)
There was a problem hiding this comment.
Ah, it doesn't need to be published, but it is currently being used.
llm-chain-local/src/executor.rs
Outdated
| ))?; | ||
|
|
||
| let params = invocation_options.unwrap_or_default().into(); | ||
| let llm: Box<dyn llm::Model> = match model_type.as_str() { |
There was a problem hiding this comment.
Could this be done upstream?
There was a problem hiding this comment.
So you mean exposing a load option that takes an enum variant referring to the model type?
There was a problem hiding this comment.
meant the mapping of strings to model types :)
llm-chain-local/src/options.rs
Outdated
| #[derive(Debug, Clone, Serialize, Deserialize, Default)] | ||
| pub struct PerExecutor { | ||
| /// Optional path to the LLM. | ||
| pub model_path: Option<String>, |
There was a problem hiding this comment.
Could this be a Path instead perhaps?
There was a problem hiding this comment.
Without trying too hard, a first attempt fails because Path cannot be Sized, which appears to be required for Option.
There was a problem hiding this comment.
Ah yes. How about PathBuf. Should be Sized.
| /// Optional path to the LLM. | ||
| pub model_path: Option<String>, | ||
| /// Optional type (e.g. LLaMA, GPT-Neo-X) of the LLM. | ||
| pub model_type: Option<String>, |
There was a problem hiding this comment.
No ideas to have an enum and a fallback? To let know what models are supported?
I mean something like:
enum ModelType {
LLama,
GPTNeoX,
Custom(String)
}There was a problem hiding this comment.
The model architecture enum isn't currently serializable (although I don't see why it can't be), so I'm leaving this a string for now.
There was a problem hiding this comment.
Hopefully this will be changed soon rustformers/llm#200
danforbes
force-pushed
the
dfo/feat/local-llm
branch
from
36b2b23
to
9c05a5d
Compare
danforbes
force-pushed
the
dfo/feat/local-llm
branch
from
9c05a5d
to
ccf9051
Compare
This seems to work in simple cases, but it's a very naive implementation and I'm happy to accept any suggestions to improve it.
Note that this is using a feature branch of the
llmcrate rustformers/llm@main...danforbes:llama-rs:dfo/chore/llm-chaincc: @philpax