How do we know AUTOMATICALLY if a prompt improved Auto-GPT ?

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

Currently someone needs to intentionally modify the code to say " I have beaten challenge A".
But it's possible someone makes an improvement on challenge A but also improves challenge B.

We need to attempt challenges anytime there is a prompt change.

Examples 🌈

No response

Motivation 🔦

No response

[Boostrix]

for starters, by keeping track of the costs spent to arrive at a solution ?
In other words, at least steps/API tokens + time ?

In the future, maybe by tracking CPU/RAM utiization as well.

But in general we should gather data for different prompts so that we can use gnuplot to plot performance for each version/commit.

And we should probably start by using GPT to come up with N mutations for a given task (that we know works) and then use those as a baseline for future benchmarking

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.