Reconsider llm chatgpt command and general command design

[simonw]

Options for an interactive chat mode:

• It's a new command - llm chat -m gpt-4 for example. This feels a bit odd since the current default command is actually llm chatgpt ... and llm chat feels confusing.
• It's part of the default command: llm --chat -4 starts one running.

Maybe the llm chatgpt command is mis-named, especially since it can be used to work with GPT-4.

I named llm chatgpt that because I thought I'd have a separate command for bard and for llama and so-on, and because I thought the other OpenAI complete APIs (the non-chat ones, like GPT-3) may end up with a separate command.

Originally posted by @simonw in #6 (comment)

[simonw]

I need a top-level design that supports the following:

• Leaves space for commands like llm bard and llm claude and suchlike - some of these may be added using plugins.
• llm "prompt goes here" default command can be configured to use different models and model settings
• Different classes of model have different options - some models might not have a system prompt for example
• Compatible with the "alias" mechanism I want to add for saving prompt templates.

[simonw]

Maybe llm chatgpt becomes llm openai.

I need to figure out how the completion APIs like GPT-3 will work within that llm openai command though.

[sderev]

What do you think of keeping llm chatgpt while also adding llm openai?

• llm chatgpt calls by default gpt-3.5-turbo
• llm chatgpt -4 calls gpt-4
• llm openai --davinci-003
• llm openai --gpt-3.5-turbo
• llm openai --gpt-4
• etc.

Or a longer version (but maybe better for clarity):

• llm openai --model gpt-4 "prompt"
• llm openai --model davinci-003 "prompt"

As for the "chat mode", it could be done by using the flag -i or --interactive.

Also, llm would use llm chatgpt by default, but this could be changed in a configuration file.

[sderev]

For ChatGPT, there's also the context length to consider:

• llm chatgpt would still call the "basic" gpt-3.5-turbo model
• llm chatgpt -16k
• llm chatgpt -4 -32k
• llm openai --gpt-3.5-turbo-16k
• llm openai --gpt-4-32k-0613
• etc.

[simonw]

I'm going to create:a new default command called prompt - which can take the user's preferences into account before passing on to some other command (just openai for the moment).

UPDATE: No, that doesn't work - because I need all of the various options and arguments on the command to be available on that default command too. So I should stick with openai as the default.

[simonw]

Changed my mind again - I think llm prompt is indeed the way to go here.

I'm going to make that the default command, and have it expose a subset of functionality that I expect to be common across all models - it will accept a prompt and a model and run that prompt.

If you need to do something specialized with custom options, you can use llm name-of-model instead. llm prompt will be the lowest common denominator.

[simonw]

I want to make streaming mode the default - I'm fed up of forgetting to add -s to everything. I don't see any harm in it as a default, people can turn it off with --no-stream if they really want to.

[simonw]

Here's the set of options for the prompt command now:

llm/llm/cli.py

Lines 32 to 51 in 68c3848

	@click.argument("prompt", required=False)
	@click.option("--system", help="System prompt to use")
	@click.option("-m", "--model", help="Model to use")
	@click.option("--no-stream", is_flag=True, help="Do not stream output")
	@click.option("-n", "--no-log", is_flag=True, help="Don't log to database")
	@click.option(
	"_continue",
	"-c",
	"--continue",
	is_flag=True,
	flag_value=-1,
	help="Continue the most recent conversation.",
	)
	@click.option(
	"chat_id",
	"--chat",
	help="Continue the conversation with the given chat ID.",
	type=int,
	)
	@click.option("--key", help="API key to use")

(I just removed the -4 option).

Do these make sense as a set of options for any generic model?

Looking at them in turn:

• prompt - yes, all LLMs should take a prompt
• --system - only relevant to ChatGPT and GPT-4 at the moment, but I think it's OK to leave it in here
• -m/--model - which model to use, fundamental
• --no-stream - if we stream by default this makes sense, plus it's OK if it is a no-op
• -n/--no-log - yes, all models should log by default and have an opt-out
• -c/--continue - only makes sense for "chat" oriented models, can throw an error otherwise
• --chat - ditto
• --key - yes for models which use an API key, ignored for others

I'm happy with these as the standard set of options. I don't think it's too harmful that some of them won't make sense for every model and should return errors if used incorrectly.

[simonw]

As part of the templates feature I'll be adding a -t/--template name-of-template option, which will definitely work for all models.

• Mechanism for storing prompt templates #23

[simonw]

Further work on this will happen here:

• Plugin hook: register_models #53