Consider whether or not to migrate bugs.python.org source code to this repo

[ncoghlan]

(Migrating this proposal from http://psf.upfronthosting.co.za/roundup/meta/issue655 )

When CPython migrated to GitHub, the bugs.python.org code stayed on hg.python.org in order to remain consistent with upstream Roundup.

It would be nice if the bugs.python.org maintenance could be split somehow such that most CPython specific changes could be managed using the same process as other core-workflow tools, even if there were some lower level issues that still required making changes to a modified version of Roundup managed in Mercurial.

[ewdurbin]

We have a handful of repositories to consider:

https://hg.python.org/tracker/roundup/

This is our fork of roundup. It should remain as up to date as possible with upstream at http://hg.code.sf.net/p/roundup/code which is also a mercurial repository. I think it's fine that it remain on hg.python.org until such a time that the upstream repo is compatible with migrating it to GitHub or similar.

https://hg.python.org/tracker/python-dev/

The templates, extensions, detectors, and modules that customize bugs.python.org. I'm 100% on board with bringing that over to GitHub!

Other hosted trackers

We still also host:

Jython's tracker

https://hg.python.org/tracker/jython/ We could similarly migrate the customizations into this repository along side the CPython tracker.

Meta tracker

https://hg.python.org/tracker/meta/ which it sounds like goes away per #1

Setuptools tracker

https://hg.python.org/tracker/setuptools/ which seems defunct in favor of the issue tracker at https://github.com/pypa/setuptools/issues. Perhaps we should archive/statcify the existing tracker?

Roundup's Tracker

Not currently source controlled. I'd like to see this migrated into this repository as well.

[ncoghlan]

Making this repo a replacement for https://hg.python.org/tracker/python-dev/ sounds excellent to me, since most of the events that core-workflow folks are going to be interested in should already exist at that level.

I filed #4 to cover doing something about the legacy setuptools issue tracker.

Running http://bugs.jython.org/ and http://issues.roundup-tracker.org/ out of this repo would also be fine by me, although I'd also ask if it might make more sense to give them their own repos (as that way the webhook triggers for CI and automated redeployments would more naturally align between the repositories and the live sites).

[ezio-melotti]

I would prefer to leave python-dev on hg.p.o at least for now. If python-dev is moved here, the other instances should be moved as well, since I often have to apply the exact same patch on all 4 of them.
Using the same vcs for both the instances and roundup would also be nice.

[ncoghlan]

While maintaining consistent workflows is definitely preferable, we already have a major inconsistency between bugs.python.org, CPython itself, and the rest of the CPython workflow automation that means we can't just send a PR for bugs.python.org the way we can for our other workflow tools, and that creates a significant barrier to contribution for the bugs.python.org instance. It also makes it significantly more difficult to set up CI for bugs.python.org, since we can't just use Travis CI the way we do for CPython and other workflow repos.

While there are some rumblings along the lines of "Well maybe we should just stop using bugs.python.org entirely, then", I personally consider that to be a more drastic solution than we actually need. Instead, the lowest impact option we currently have available to mitigate the impact of the workflow inconsistency is to shift where it occurs to be between the bugs.python.org instance repo (which would move to GitHub), and the Roundup fork (which would remain on hg.python.org unless/until Roundup itself migrated to using git, whether on GitHub, GitLab, or somewhere else).

This does have flow-on effects for the other Roundup instances that the PSF hosts, and definitely shifts the burden of the inconvenience down the stack to the folks maintaining the rolling fork, but has the major advantage that most bugs.python.org contributions will be able to flow through a simplified version of the contribution process documented in the CPython developer guide.

It will only be when folks decide they want/need to modify Roundup itself that they'll need to start looking at Mercurial and non-PR based contribution processes (and even there, we could potentially mitigate the difference by making BitBucket the master copy of the Roundup fork and accepting PRs there, and switching hg.python.org over to being a PSF-hosted mirror).

[brettcannon]

@ezio-melotti What are the 4 instances? setuptools, jython, b.p.o and what?

And are these other instances providing people-power to help support their instances? If the answer is "no" then I think it might be time to ask them to consider migrating to any of the other various free open source hosting platforms that come with an issue tracker rather than have knock-on effects of trying to support them.

[ezio-melotti]

b.p.o, meta-tracker, jython, setuptools.

AFAIK no one is working on them, but b.p.o and meta are ours and setuptool is not longer used.
Jython is the only one that should be actively used.
I've updated those instances every time I did an update on Roundup and/or there was some security issue, but other than that they don't require any particular maintenance.

[ewdurbin]

http://issues.roundup-tracker.org itself is also hosted on bugs.python.org

[ezio-melotti]

While it is hosted on the same machine, it's not part of the 4 instances we maintain, and it's currently maintained by the Roundup devs.

[brettcannon]

OK, so maybe we should reach out to the Jython devs and let them know we are no longer going to push changes to their fork for them and they will need to have someone do it on their behalf? That will bring the number of instances down to one (assuming this replaces the meta tracker).

[ezio-melotti]

I'm not sure that would be the best course of action. The other instances require so little maintenance that they are not a burden (at least to me). For instance, the Jython instance got a dozen of commits in the last 6 years, and those changesets are applied to all the instances without changes.
I think it's best if we keep all the active instances together, and we keep maintaining them together -- in fact, I also think it would be better to add the roundup-tracker instance to the others.
Moreover, I believe none of the Jython developers have access to b.p.o or even care much about what happens to their instance as long as it keeps working fine.

[brettcannon]

@ezio-melotti but will it be a burden if we start to get more people involved? It just sounded like it was a burden in your earlier comment is all. I mean if you want to take it on to backport all changes that land in b.p.o's code then I have no objection.

[ezio-melotti]

I'm not sure I understand exactly what you mean, so I'll try to summarize the situation and my position.
It's a long post and I also included several things that are OT for this issue, but I'm having a hard time following all the issues/threads/discussion that came up lately and thought it would be good to have a summary somewhere (new issues/threads can then be started elsewhere if necessary).

TL;DR imho the vcs is not what's stopping people, we need to improve the docker image so that it's easier to set up a tracker and test things and we need more people that understand Roundup and are able to review patches. Moving to github will also make my life slightly harder for the time being.

Regarding our Roundup fork:

• it would be easier to maintain/update if it's kept on hg, since upstream uses hg too;
• it should not get any changes (especially b.p.o-specific ones), since we don't want to diverge from upstream;
• if we change anything, it should be backported upstream, so it would also be better to discuss the changes with upstream first. One such change is adding the rest api;
• getting more people involved in this would be helpful, but moving to github:
  • doesn't necessarily means more people will show up to help;
  • makes things more complicated for the aforementioned reasons;

Regarding the instances:

• there are currently 4 instances we maintain (b.p.o, meta-tracker, jython, setuptools) + roundup-tracker that is hosted on the same machine and manually updated by the Roundup devs;
• setuptools and the meta-tracker could be made read-only, bringing the number of instances down to 2+1;
• the roundup-tracker instance could join the instances we maintain, bringing the total to 3;
• most of the work is done on the b.p.o instance, the others are very low maintenance, to the point that I wouldn't mind maintaining the roundup-tracker instance too;
• getting more people involved in improving the b.p.o instance would be helpful, but moving the instances to github:
  • doesn't necessarily means more people will show up to help;
  • makes things more complicated if roundup is still on hg;

Regarding the tasks and where contributors would be useful:

• the tasks are:
  • provide an easy way to run the tracker locally;
  • writing new features for the b.p.o instance;
  • writing new features for our fork of roundup;
  • reviewing and approving the features;
  • backporting roundup features upstream;
  • updating roundup and our fork;
  • updating the instances;
  • deploying changes;
• the areas where help would be useful would be:
  • reviewing and approving features (especially big ones like the rest api or anything involving security);
  • providing an easy way to run the tracker locally (the docker image is a good starting point, but should be improved/made official/documented);
  • backporting things upstream (the roundup devs have helped out with this);
  • writing new features for the b.p.o instance and our fork;
  • the other tasks are just occasional busy-work that I can handle alone;
• to make the development easier, the steps would be:
  1. improve the docker image so that everyone can easily set up a tracker and test/review changes;
  2. find people that are able to review the patches;
  3. find people that can develop new features, with an adequate number of reviewers to match;
  4. possibly automate/streamline deploying;

For the future, I think we should follow these steps:

1. wait for the upcoming release of Roundup 1.6 and update our fork;
2. update our instances and possibly apply outstanding patches (e.g. the github login);
3. review and apply the rest api patch;
4. test/fix the rest api patch and backport upstream once stable (possibly before Roundup 2.0);
5. upgrade to Roundup 2.0, switch to Python 3, possibly ditch our fork and start following main upstream releases;
6. continue to improve our instance as we see fit;

In parallel to this we can also:

• keep improving the docker image;
• decide if/when/what move to github (also depending on what Roundup decides to do) and what to do with the meta tracker and setuptools instances, but I would wait after step 2;

[brettcannon]

So I'm hearing:

1. Better Dockerfile
2. Fix stuff
3. Get off of our fork with Roundup 2.0
4. Keep fixing stuff forever

So who wants to help make step 1 happen? And are people okay with helping make that happen on hg.python.org and the current development process there? If anyone answers "yes" to the first question but "no" to the second then please speak up.

[ezio-melotti]

Note that the dockerfile is already on GitHub at https://github.com/python/docker-bpo (thanks to @soltysh) and that should be the most recent version (there used to be a couple of clones on BitBucket too, including https://bitbucket.org/ezio_melotti/docker-b.p.o/ and the original version by @introom).
The repo has a few open issues, so that should be a good starting point for anyone that wants to help.

Getting out of our fork is something I've been working on since we switched to Mercurial, and we are not far now, but the rest api patch will push us further apart until is backported upstream.

[Mariatta]

I personally think that as long as this project is in mercurial, without any CI, and without good way to even review patches, then it is not ready to start accepting lots of patches and fixes from new contributors.

I've read https://wiki.python.org/moin/TrackerDevelopment , and I still couldn't figure out how to upload a patch, let alone how to help review patches.

[ncoghlan]

None of the issues Ezio raised are the ones I am trying to solve with this proposal: the one I care about is that folks that are already actively contributing to CPython, the main PSF website, and to other core development workflow tools are not able to meaningfully contribute to bugs.python.org maintenance without:

1. Learning a new, now mostly undocumented, source code management workflow (the CPython dev guide no longer covers this)
2. Talking about proposed patches in the abstract, without any tooling to help with line-by-line commenting
3. Figuring out how to run Docker locally, rather than pushing a patch and letting pre-merge CI figure out whether or not it broke anything

That's a problem, since bugs.python.org is essentially our core workflow engine, and without a straightforward way to fix small niggling problems, folks are unlikely to be motivated to progress towards tackling the bigger structural problems with the current maintenance process - they're more likely to invest their time in working out how to eliminate bugs.python.org from the workflow entirely.

[ezio-melotti]

@ncoghlan: What kind of tooling/workflow would you suggest then (other than just GitHub)? In the past (before the switch to GitHub) I wanted to fix things on the python.org website and couldn't figure out how to do it.
ISTM that having CI figure out whether or not things broke is not as straightforward, since many of the changes require user interaction and are not tested from the code. If things are not tested manually on a local instance, having a new online instance for a live test tracker might be a solution, but maybe you have something else in mind.

@Mariatta:

I personally think that as long as this project is in mercurial, without any CI, and without good way to even review patches, then it is not ready to start accepting lots of patches and fixes from new contributors.

In order to create these patches they would still need to figure out how to set up and run the tracker locally, understanding how Roundup works, what to change, where, and how, how to debug it, and how to test their changes. This is IMHO a much higher wall than learning how to use mercurial (learning hg clone and hg diff > patch.diff is enough to create the patch) or having a CI and inline reviews. This is also why I put "improving the docker image" as one of the most important steps above.

I've read https://wiki.python.org/moin/TrackerDevelopment , and I still couldn't figure out how to upload a patch, let alone how to help review patches.

The workflow is the same that we used on CPython for years: get a local clone, change the code, create a diff, upload it to the (meta)tracker, have someone review/try the patch, and, if it works, commit and push it.
Now that CPython uses a different workflow, it's not as obvious and this workflow should probably be documented somewhere.

[ncoghlan]

@ezio-melotti Given @soltysh's work on OpenShift/Kubernetes compatibility, then folks should be able to deploy demo instances from their public forks to the OpenShift Online starter tier, and potentially also to the Google/Azure/Amazon Kubernetes offerings (GKE/AKS/EKS). That's a lot more powerful than only being able to test locally, since it means folks can also share UI changes for review and comment before they deploy them. However, even without that, moving to a PR based workflow means that anyone can pull the PR down and run it up locally in Docker, rather than having to download the patch and apply it manually. (Essentially, you'd gain access to this aspect of the CPython workflow: https://devguide.python.org/gitbootcamp/#editing-a-pull-request-prior-to-merging )

It would even be possible for the CI to do a full local deployment using Docker, and then start setting up some basic automated behavioural tests using something like behave or pytest-bdd. The key stepping stone towards doing that would be to write out the kinds of things that a human should be checking for in a manual test, and then asking whether or not a test script could be checking for those things instead.

Getting back to the learning curve question, though, "How to run your own modified copy of the issue tracker" and "How to check if the issue tracker is still working correctly after making changes" are important points, but they're also intrinsically part of the task "I want to make changes to the issue tracker".

"How to use our legacy patch-based development workflow" isn't the same kind of learning activity: it's a "We know there are better workflows available, because we migrated CPython to them, but we simply haven't got around to modernising this one yet" task. As such, it's a lot more demotivating as a barrier to contribution than being asked to learn how to run or test the service that you're aiming to work on.

As such, rather than attempting to smooth out that part of the learning curve, it makes more sense to take that step out entirely, leaving only the intrinsic barriers to entry (i.e. actually being able to run a test instance).

[willingc]

@soltysh's Dockerfile is excellent. It's very straightforward to get up and running. Docker has improved a great deal on its support of MacOS and Windows in the past year, and installing docker is no longer the same hurdle it once was.

One thing that would improve the experience using the Dockerfile would be to include some dummy data that would aid in testing that UI changes are working as expected.

IMHO, the good work by @ezio-melotti and others on bugs.python.org would be even more valuable if the source and Dockerfile were all in one place and consistent with the workflows of CPython.

[ncoghlan]

That actually reminds me of something I've set up at work: a pipenv managed "dev environment" type arrangement, where git clone + pipenv shell gets you a working environment with everything else you need (up to and including ansible).

If we updated https://github.com/python/docker-bpo to include Mercurial inside the Docker image, then the initial repository cloning and later updating could be handled as additional helper scripts inside the image.

The change review process would still be fairly awkward, though :(

[ammaraskar]

For my 2 cents as someone who has recently tried to contribute code to add logging in with Github, here's some brief thoughts:

• The documentation here https://wiki.python.org/moin/TrackerDevelopment was actually fairly thorough and set up was not too hard.

• What the documentation lacked, and what Mariatta pointed out is that there's no description at all on where to submit patches. The only reason I knew to do an hg diff and submit the patch is because I remembered the old CPython workflow. But even so, with this being used as the meta tracker, it was unclear where to submit the patch, so I posted it as a comment here: Support logging in with GitHub #7 (comment)

• As someone who has done a fair bit of web dev and now had a cursory look at the bpo source code, it would be fairly involved to add automated CI testing, even the type proposed above. The project is fairly outdated and not built with this in mind. With the level of contributions it currently receives, this is a fairly unreachable goal. You would really need someone to volunteer a fair bit of time to make it happen.

[bitdancer]

On Thu, 28 Jun 2018 11:17:18 -0700, Ammar Askar ***@***.***> wrote: * As someone who has done a fair bit of web dev and now had a cursory look at the bpo source code, it would be fairly involved to add automated CI testing, even the type proposed above. The project is fairly outdated and not built with this in mind. With the level of contributions it currently receives, this is a fairly unreachable goal. You would really need someone to volunteer a fair bit of time to make it happen.

Yes, roundup was developed long ago, and has actually held up surprisingly well, considering that. roundup is a separate project from bugs.python.org, and there are a few (not many!) developers working on it in addition to Ezio's occasional contributions. They talked on the -dev mailing list about a possible git migration at one point. No idea where they are at now, since the mailing lists started requiring a "confirm you are still interested" response and I didn't bother. I still hang out on their irc channel, though (#roundup on oftc). All of which is to say that a move of roundup itself to git would benefit more than just cpython, and possibly also pull in some additional knowledgeable folks if someone wanted to spearhead it. That, however, is separate from the *instance* definitions and the testing of the same. I don't see why testing the b.p.o instance would be any different a problem than testing any other web site, but I've never written automated tests for a web site so I really don't know.

…

--David

[ammaraskar]

That, however, is separate from the instance definitions and the
testing of the same. I don't see why testing the b.p.o instance would be
any different a problem than testing any other web site, but I've never
written automated tests for a web site so I really don't know.

Right, when talking about bpo up there, I really mean roundup + the bpo instance as a whole.

[Mariatta]

I have just discovered the repo https://github.com/psf/bpo-roundup today. Is this now the official codebase for bpo, and it's no longer on mercurial? It would be great if all relevant documentation can be updated. For example, the page https://wiki.python.org/moin/TrackerDevelopment seems outdated.
We may also need to consolidate the issues in this repo (python/bugs.python.org) and transfer them to psf/bpo-roundup

[berkerpeksag]

bpo-roundup is our fork of Roundup. Repository for bugs.p.o's Roundup instance is located at https://github.com/psf/bpo-tracker-cpython

[Mariatta]

Ok then should we retire this repo (python/bugs.python.org) and transfer all open issues to psf/bpo-tracker-cpython and start pointing people there? It's confusing that we have the two repos right now.
Note that only those with write access to both python/bugs.python.org and psf/bpo-tracker-cpython can transfer issues, and I don't have such privilege.

[berkerpeksag]

Sounds good to me. I can move issues over to the new repo. Archiving/redirecting it to the new location may require someone from the infra team.

[bitdancer]

No, I don't think so. I haven't been involved in tracker development for a while, so I may be out of date, but this is my understanding: Our roundup setup comes in two pieces: there is our modifications to the base roundup code (our fork, as Berker mentioned), and there is the code that constitutes our *instance*, which is a set of template and code files that implement bugs.python.org on top of the base roundup code. Those are the two code repositories Berker metioned, IIUC. It is often hard to tell where a bug or feature needs to be addressed (the roundup codebase or the bugs.python.org codebase). Therefore we have always had a "meta tracker" to track bugs related to bugs.python.org. The repo you are asking about is the replacement for the old meta-tracker, which was itself a roundup instance. So no, unless things have changed more than I think, this repo is doing exactly what it should be doing: holding the issues related to bugs.python.org, and nothing else (no code). A "repo" named 'bugs.python.org' is a much more obvious place for people to post bugs/feature requests than the other two repos, assuming one could even figure out which one to post to :) You will note that the two code repos (both under 'psf', not 'python') do not have any issues, which is as it should be.

…

--David PS: I haven't read back through it, but after a quick scan I think that if psf/bpo-roundup is a fork of the github repo for roundup being maintained by the roundup team, then this issue can probably be closed once the README of this repo is updated accordingly.

[berkerpeksag]

Ezio can correct me if I'm wrong, but I think we are close to ditching our Roundup fork so perhaps we can now combine meta tracker and source code of bugs.p.o instance in the same repo.

[Mariatta]

Even if you're still keeping Roundup fork, I just think the concept of meta-tracker is less relevant on GitHub. You can always transfer issues to the other repo easily if necessary. Additionally you're able to cross reference other GitHub repo in your PR.

Having this repo as the "meta-tracker" right now feels odd to me since us core devs aren't really the maintainers of roundup or b.p.o. If anything perhaps core devs should be added to the https://github.com/psf/bpo-tracker-cpython repo so we can help maintain it.

[ezio-melotti]

https://github.com/psf/bpo-roundup contains the Roundup code + our changes in the bugs.python.org branch ("our fork").

https://github.com/psf/bpo-tracker-cpython is the instance used for https://bugs.python.org/
https://github.com/psf/bpo-tracker-roundup is the instance for https://issues.roundup-tracker.org/
https://github.com/psf/bpo-tracker-jython is the instance for https://bugs.jython.org/

These 3 instances are run on top of the code of the bpo-roundup repo.

https://github.com/python/bugs.python.org (this repo) was meant to replace the old meta-tracker. The old meta-tracker tracked issues related to all the other 4 repos (back then there were actually 5 instances, our Roundup fork, and a couple of ancillary repos, but most issues were either about our fork, the b.p.o instance, or affected all the instances).

Once we switch to GitHub issues, we won't need the bpo-tracker-cpython repo anymore, but bpo-roundup will have to stay around unless issues.roundup-tracker.org and bugs.jython.org also switch. Merging the issues from this repo into the bpo-roundup repo might be ok then, with the understanding that it will cover both our fork and the two remaining instances.