How can we help the core team with this huge backlog?

[nponeccop]

Duplicates

• I have searched the existing issues

Summary 💡

No response

Examples 🌈

No response

Motivation 🔦

There are 170 open PRs as of 2023-04-16

[nponeccop]

Ok, let me explain the current release process first. It's may be described elsewhere but let's keep a complete description here so it's easier to see the big picture.

Part 1

There are presently 5 people with write access to the repo:

• @Torantulino - the boss
• The rapid merging team of @nponeccop @p-i- and @richbeales
• @BillSchumacher is another merger having access to the mergers private room but who hasn't participated in a spree yet

There are also other people in the private maintainers-chat room in Discord:

@coditamar @merwanehamadi @0xArty

The invitation process goes through @p-i- (@pi in Discord). There is a discord channel in Readme, it's public with 10k+ members but there are core team rooms, so the invitation is to get in those.

There are 2 github branches as of today, master and stable. Approximately once a day we freeze the current master, smoke-test it for 1 hours and merge to stable. Tag and release.

Public PRs are not against stable but against master. All internal work is done through PRs too, we don't even have rights to push to master. So, there are time windows when we merge public PRs and when we don't, so the team (mostly @BillSchumacher as he is incredibly productive, but the rest of us submit too) can work and iron out their PRs without the noise introduced by the constant merges.

[nponeccop]

Part 2

The constant merges phase is presently organized the following way:

• @nponeccop asks the core devs if nobody is developing and a merger person is available to merge. Presently there are two, @richbeales and @p-i-
• @nponeccop Sends notifications to people with conflicted PRs by looking through his mail notification log.
• The PR authors fix the conflicts.
• Once there are no conflicts, @nponeccop checks for obvious signs of a bad CI, such as non-atomic CIs fixing many issues. If the basic quality is ok, he approves the PR and CI
• Once CI is red, he re-notifies the PR owners
• Once CI is green he notifies the merger by sending B5 1234 clean or B5 1234 clean but dubious or B5 1234 clean one-liner to the merger. One-liner PRs (which are not necessarily one line but simple to review) are often merged out of band.
• The merger then works on his own:
  • Reviews the Python code and how the PR matches the project priorities (@nponeccop don't actually care what the code is doing, he is a human PR linter)
  • Bounces the PR for fixes (basically passing it to @nponeccop indirectly through mail notifications from GitHub)
  • May label PR as not worth merging at the moment even if it's otherwise clean
  • Actually merges if it's ok

[nponeccop]

The bottleneck is presently in the mergers, on days when both @richbeales and @p-i- are available, we are able to process all the PRs in which their owners resolve the conflicts.

So the help can be of 3 types:

• another merger. So it's available when the other mergers are not
• Someone to help the PR authors by pushing to their branches (needs to become a maintainer for Auto-GPT)
• Someone to help the PR authors by cloning their branches and submitting entirely new clean PRs (doesn't need any collaboration or permissions from the core team)

[Pwuts]

I can help with reviewing/moderating/merging DevOps specific issues/PRs (Docker, devcontainers, GitHub workflows/actions). I do have significant experience with Python but not very recent, so I want to refresh that before offering to actively contribute in that area.

[Pwuts]

As for the way this project is organized, I'd suggest to:

• use master as the stable branch
• set up a dev/development branch
  • with a merge queue to better accommodate the large number of daily contributions
• limit direct commits to the stable and development branches
• use version tags on master to indicate tested, stable versions
  • this requires better test coverage, which should also be a high priority imo
• set up a project with some automation to better coordinate teamwork
• require PRs to link the issues they close/solve and related/conflicting PRs, as a way to push contributors to search for similar, existing work

The biggest challenges I'm seeing right now:

• contributions that are valuable but do not fit within the current scope of the core project, demonstrating the need for plugin support
• a huge backlog of issues and PRs, the size and clutter of which can discourage to search for existing issues/PRs before submitting one's own, even though many have already been solved

[AndroidDev77]

Master should really be the stable branch, that's part of the reason why you are getting so many issue tickets against it. Also the fact that stable is broken right now, so people are resorting to master.

[p-i-]

I'm pi#8377 on Discord fwiw.

[nponeccop]

@Pwuts @AndroidDev77 Yes we will migrate to a more standard workflow once we iron out more important issues. @merwanehamadi is actively pushing this workflow too, but the current workflow is what the rapid merging team has experience with. The release process is not polished yet, we had delays for 3 hours already without any public or core merges. So IMO let's have releases go smoothly and then decide on branching. There is a branching discussion in dev-threads in Discord, too.

[ntindle]

Would love to help with this as well. Have significant familiarity with DevOps for projects and managing github repos

[waynehamadi]

@ntindle I would love to talk to you, can you join us on discord ?
https://discord.gg/autogpt

[ntindle]

I’m in the discord with the same handle. Would love to chat

[Boostrix]

not wanting to beat the dead horse here, but how about exporting those issues and letting Auto-GPT look at issues and merge requests, to work out some of the lowest hanging fruits and some of the most worthwhile pull requests to get merged ? Likewise, it might make sense to look at the activity/track record of some of the contributors, to get more people involved in reviewing/integrating useful stuff.

I mean, let's face it, the OP's got a point: the project got rather popular in a really short time span, so maybe that would actually be an opportunity to use recursion - aka let Auto-GPT manage its own project ?

I am sure the team of existing contributors could come up with "objectives & goals" that the "project management" GPT module could then use to work out what to prioritize next, based on resources/constraints (who's got actively involved, available merge/pull requests, overall github activity of people making requests/suggestions etc) to work out the rest.

And it would actually be a rather compelling way to "demo" what AutoGPT is capable of.
If that's something that people might want to consider, coming up with a new github plugin would be the next logical step - to actually provide AutoGPT with access to the project: https://docs.github.com/en/rest?apiVersion=2022-11-28

In parallel, existing contributors could come up with a dedicated ai_settings.yaml file based on their objectives/goals for the project.

This should then be maintained as part of the repo, so that it can be reviewed/refined over time

Overall, if this is an option, it might make sense to restrict code files to <= 8192 tokens so that the system can actually deal with those (over time) ;-)

The basic idea can be seen here: #3480
And it's been brought up previously in several comments:

#15 (comment)

A cool extension to this idea would be having autogpt spin up an instance of its self like every couple hours, crawl all the current prs, and build a sandboxed version of itself with each new pr merged. Then it could determine either through some combination of unit tests, benchmarking and it evaluating its own code quality, whether this pr was anything beneficial. This could unclog the massive amounts of prs being made and hopefully only let the good ideas shine through. Some problems I see though are people trying to inject malicious code, however, if adequately sandboxed this may not be an issue.

#15 (comment)

A thought on how to implement this would be building a python class or set of functions that can open PRs. Using pygithub the script could pull all prs and iterates though them. A place to start could be a set of tests that must pass else the bot reviews and comments on the pr and closes it. If the pr is only a comment or test, the bot can try to build a python script to satisfy the given functionality. The functionality of having tests pass would help avoid the ai spinning cycles on poor code. I think having the bot work off of a directive submission process, something like "create a python script that calls llama and returns the result in Json" would really kick off the process. Plus crowd source ideas. A 'suggested ideas' section of only text or markdown may be an option. Or we could utilize GitHub issues, pull all issues, look for a specific format. CREATE: some idea. The script would need a key to submit prs, anyone can do that. But to read, review and merge prs we would need a key from someone with write access. It could merge to a dev branch so it doesn't break stuff too bad.

#15 (comment)

I would love to talk with maintainers. We have been creating Reviewpad to help with that. You can specify multiple PR automations and policies and on top of that you can use GPT-4 to summarize and very soon automatically approve PRs.

[Boostrix]

huge backlog of issues and PRs, the size and clutter of which can discourage to search for existing issues/PRs before submitting one's own, even though many have already been solved

Indeed, that's a very good observation (still struggling to make heads and tails of everything).

With this number of open PRs and given the popularity of the project, how about a more liberal approach to handle this degree of activity ?

For instance, some contributors could be given access to help with plugins only, others with commands only, whereas others could work on agents - just to declutter the whole thing a little more, but also to have/foster experts.

Also, "just" reviewing tests, plugins or commands will be much easier a task than reviewing potentially any PR.
The underlying infrastructure/APIs would still be maintained by the core team, so just talking about components using this API (again: tests, plugins, commands come to mind).

It would also be easier for other collaborators to see if someone is trying to review something outside their area of expertise.
Having 200+ open PRs seems like a potential waste of manpower/time and like a sure way to sooner or later see the project being forked (if that hasn't already happened?).

Besides, some of the more complex changes are probably better developed in parallel using branches, until the dust settles ?

Please don't get me wrong, but I suppose the problem is only going to be more pronounced 6-8 weeks from now ...

For future reference: at 240 open PRs, the project stats currently show:
Watch: 1.4k / Fork:23k / Star: 120k

In other words: for roughly 1k forks, we're currently seeing ~10 open PRs ...

[Pwuts]

Update: we have recently been joined by a number of enthusiastic catalysts who help us navigate and clear the backlog. We still have to put some work into clarifying the path to contribution though.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.