Make Auto-GPT aware of it's running cost

[Torantulino]

Auto-GPT is expensive to run due to GPT-4's API cost.

We could experiment with making it aware of this fact, by tracking tokens as they are used and converting to a dollar cost.

This could also be displayed to the user to help them be more aware of exactly how much they are spending.

[0xcha05]

could use this?
https://github.com/openai/tiktoken

[Torantulino]

My thoughts exactly.

If someone wants to tackle this.

Suggested Implementation:

• ALL API calls should be centralised through a single method that counts tokens and adds to a dollar amount based on the model used.
• Store this in its own file and remove openai imports from everywhere else so that the API can't be called accidentally.

Displaying

• Total dollar amount spent should be constantly displayed to the user.
• Total dollar amount spent could be displayed to the AI. Perhaps it could even be given a budget, and the dollar amount would count down as it spends money. This could create a sense of urgency in a HustleGPT style task.

If you'd like to work on this (please do!), assign it to yourself so we don't both work on the same thing.

[0xcha05]

I can pick this, but first, I'll spend a little time on #4 .

[0xcha05]

I'm going to start work on this now, @Torantulino . please assign this to me so others don't work on the same issue.
I'll start with a fresh fork.

[ryanpeach]

I agree with all @Torantulino's ideas here. In that case, it might be a reason to use a real TUI instead of a simple set of print statements for this application. It could have a top-bar that contains running price for example.

I'd also make a flag in the cli that allows for rate limiting the application: Sleep 10 seconds between requests, for example. Some tasks the AI may be given may not require frequent polling. For example I'm using the bot for research, and notifications of major news events. Major news event searching may only need the bot to check the internet every day. This polling frequency option will be critical for giving some applications amazing speed, and others sort of dumb infrequent inexpensive speed.

[masterismail]

hey @0xcha05 are you currently working on this issue?

[Vwing]

I went ahead and implemented this in my own fork (branch "running_cost"). I'll submit a pull request later tonight.

Here it is in action, but also an example of it not working quite right.

I think I'll prompt engineer it a bit before submitting the pull request. I find that it works better if I send its remaining budget in the system message context, and yell at it if it goes over budget

    create_chat_message(
        "system", f"Your remaining budget is ${remaining_budget:.3f}" + (" BUDGET EXCEEDED! SHUT DOWN." if remaining_budget < 0  else "")),

[Vwing]

I'll submit a pull request later tonight.

I'm very close, now. Auto-GPT is beginning to behave with the appropriate behavior of hurrying up when the budget is nearly depleted, and shutting itself down when the budget is exceeded.

Too tired to finish this tonight, though. I'll give another update tomorrow night.

[Vwing]

Okay! It's working, now. It prints the running cost, the AI is aware of the running cost, and the user can optionally provide a budget (of which the AI is also aware, and behaves appropriately).

Please refer to my pull request. #762

[Vwing]

Woohoo, it's merged in! 🎉

@rob-luke would you be interested in tackling the final task of making the total running cost visible to the user?

See #762 (comment)

[ntindle]

@Vwing shoot me a DM on the discord :)

[rob-luke]

Thanks for tackling this issue @Vwing.
I would be pleased to add the feature of making the running costs visible to the user, I can get that done this week. But if someone else starts to tackle it first, let me know so we don't duplicate work 🚀

[Pwuts]

Fixed in #762

[rob-luke]

@Pwuts The final aspect of the initial problem description is addressed in #3313 which isn't merged yet.

[Boostrix]

has anybody thought about exposing this in a shell prompt-style manner at runtime, per agent (task/job) ?