Build the framework to be able to create a failing test simply from the logs/DEBUG people report

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

In order to improve Auto-GPT we have to fix user's bugs quicker and more efficiently.

To do that, we want to create a battery of failing tests that showcase some of Auto-GPT's inconsistent behaviors or bugs.

We want to create tests simply from the issues people report:

• step 1 the user reports an issue and puts the logs/DEBUG folder as a zip
• we download the ZIP, put it in our testing suite and "replay" the bug.
• we create a pull request and once it gets merged, the community can fix the bug.

This issue requires good understanding of pytest

DM me on discord : merwanehamadi

Join Auto-GPT's channel so we can talk about Auto-GPT: https://discord.gg/autogpt

Examples 🌈

No response

Motivation 🔦

No response

[anonhostpi]

Man your competence in Software QA is through the roof.

This probably seems like a basic question to you, but the design principle behind these tests is to try to test any bug fixes against the situation that discovered them?

[anonhostpi]

If that is the case, this framework might be helpful for towards y'alls efforts of making AutoGPT capable of self-improvement -link to my gists tracking y'alls issues/PRs/discussions

[Boostrix]

we want to create a battery of failing tests that showcase some of Auto-GPT's inconsistent behaviors or bugs.

To get this started, one of the lowest-hanging fruits is probably the write_to_file issue where dupe-checking sees in the log file that the file was previously written by another agent instance, and fails to write/update the file because of that - despite the file possibly no longer existing (or despite it being outdated).

Details: #3514

[waynehamadi]

@anonhostpi that's it ! this is the place I want to to go to:

• User finds an issue
• User uploads his replay
• we create the "Fix Auto-GPT" challenge for it
• people solve it
• we know it won't appear again because a test protects it.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.