Does this project need a fork or new maintainers?

[WolfgangFahl]

There has been no new release since 2020-05. I am currently having trouble with using it with Mediawiki 1.39 and before i am adding another issue I'd love to know whether this is worthwhile and will lead to any resolution.

As of 2023-02-07 there are 9 open Pull requests and 36 open issues. I don't see an milestones configure and there is no open discussion.

Who could change this situation and how?

[waldyrious]

Hi Wolfgang — thanks for raising this issue, and for your interest in the project :)

For the record, I do have administrative permissions in the project, and am aware of the issues and pull requests that have been submitted over time, but I don't have sufficient knowledge of the codebase to be able to make technical decisions regarding them. For what it's worth, whenever a decision has been within the bounds of my ability or confidence level, I did merge the relevant patches (see e.g. #242 (comment), #261, and #269).

That said, I definitely agree we need additional maintainers, but unfortunately I'm not aware of any candidates. I'd be happy to grant collaborator rights to people who make a compelling case for having both the willingness to maintain the project and the technical know-how to do so well (e.g. one of the past contributors to the project). Speaking of which, besides @danmichaelo and @btongminh, who are currently inactive, the people with the most commits to the repo are @lukasjuhrich, @marcfrederick and @AdamWill. Pinging them in case they might be interested.

By the way, I will attend this year's Wikimedia Hackathon, so I will also make it one of my goals there to scout the Wikimedia technical community for possible maintainers to this project (it would be especially helpful to get the ability to publish releases in PyPI).

Let me know if there's anything else you can think of that would help!

[AdamWill]

Hey, thanks for pinging me. I (and the Fedora project through me) do have a fairly strong interest in mwclient; I maintain python-wikitcms, a project which rather quixotically treats the Fedora wiki as a "test case management system" (basically a way of storing test cases and storing, tabulating and displaying associated test results). This is a key component of our overall Fedora release process; it's still the only place where we actually reconcile manual and automated test results, and a key part of deciding whether to ship a Fedora release is going through the results for the current release candidate in the wiki. python-wikitcms is used (among other things) for creating the initial result pages, having our automated test systems submit their results to them, and generating alternative views on the data like https://openqa.fedoraproject.org/testcase_stats/37/ .

python-wikitcms is implemented by subclassing bits of mwclient, basically, and it makes fairly heavy use of somewhat idiosyncratic mwclient designs like the page/category generator stuff, so rewriting it to use a different mediawiki client library would be quite an undertaking.

I don't know if I know the project well enough to be the main/only maintainer, but I'd certainly be willing to join with others in a maintainership group. I could probably help with 'meta' stuff like modernizing the build process including publishing on pypi; I do try to stay on top of the state of the art in that area, though there are various approaches and any maintainership group might have to agree on a favorite.

[waldyrious]

Hey! Thanks for chiming in :) and quite quickly, too!

It seems to me you know the internals of the project better than I do, at least, so having you as part of the maintenance team would already be an improvement! 😄 Besides, your past contributions have been generally well received and received little to no requested changes, which is also a very good sign that your understanding of the project is aligned with what its vision has been so far.

I will add you as a member of the organization. At the very least, that will increase our bus number by 1, so it seems justified.

I'll keep the issue open nevertheless, as additional maintainers would be welcome.

[AdamWill]

Thanks. I do try and take care to understand bits I'm touching when I touch them, but there's a lot of the project I've never really touched :)

[waldyrious]

That's fine! Being bold when we're confident, and knowing when to be cautious otherwise, is an excellent trait for participants in a collaborative project ;)

[WolfgangFahl]

@ALL - thanks for the productive discussion. I intend to be at Wikimedia Hackathon in Athens too and besides my wikidata interest this might be another project i might get involved with.

[waldyrious]

Awesome, @WolfgangFahl, looking forward to seeing you there.

Hey @AdamWill, thanks for accepting the invitation to join the org. Would you be willing to make your membership public? That way people would have a way to figure out who they could contact in case they want to reach out to maintainers.

[AdamWill]

@waldyrious , can you check the project settings and enable Github Actions? I've sent a PR to switch from travis-ci to Actions for CI. It's not very difficult. thanks!

edit: also, adding me to the org doesn't seem to have given me any maintainer privs - I don't have a button to merge PRs, for instance. Is that what you expected? Not sure if you intended to make me a maintainer yet or not. :D

[AdamWill]

btw, so far as pypi goes, @yuvipanda is listed as an admin on pypi and seems to be active on github, so maybe he can help getting more folks added?

[waldyrious]

edit: also, adding me to the org doesn't seem to have given me any maintainer privs - I don't have a button to merge PRs, for instance. Is that what you expected? Not sure if you intended to make me a maintainer yet or not. :D

Huh, that's weird. I dug a bit and it seems GitHub now has a permissions model where org members only get access to the repos if explicitly added to a team within the org. I have no idea what is the point of adding someone as an org member if that doesn't give them any permissions 🤨 Anyway, I have added you to the "Maintainers" team in the org — please check if you now have access to the settings.

[waldyrious]

I've also invited @yuvipanda to the org :)

[AdamWill]

I now have a merge button for PRs, and I have access to Settings, but I seem to have a restricted view there and still can't turn Actions on...in a repo I own I have 13 options in the left-hand sidebar, on this repo I only have two, "Moderation options" and "Pages". I guess I'm at some intermediate level. I don't need full access, just someone to change the settings on the Actions page; I use "Allow all actions and reusable workflows", "Require approval for first-time contributors", and "Read and write permissions" (though we probably don't actually need write). Stricter rules for the first two sections are intended to prevent people abusing pull requests to do crypto mining and stuff, but I've never had issues with that and we can always tighten them down if it happens...

[waldyrious]

I guess I'm at some intermediate level. I don't need full access

No, that's OK. I did want to test GitHub's "Maintain" permissions, because it's not exactly clear how full-featured it is (especially since there are two additional levels below it, "Triage" and "Write", which I assume are even more limited). I have now changed the "Maintainers" team in the mwclient org to have "Admin" permissions, so you should now have access to all the settings. Please confirm (and sorry for the back-and-forth 😅)

[AdamWill]

Thanks! I'll sit down and poke stuff in the morning.

[AdamWill]

So I've merged the outstanding PRs that were easy to review, the five remaining are more complicated. Outstanding:

• Get someone in a position to publish new releases to pypi
• Drop Python 2 support and code that's only there for Python 2 support, modernizations like format strings (optional)
• Update the build system to use https://pypi.org/project/build/ or https://python-poetry.org/ or something (I just use build and a simple release script for my projects, but I know some folks love poetry), move as much config as possible to pyproject.toml and support build straight out of pyproject.toml, but also keep setup.py for compatibility
• Revise the coverage/quality checking a bit: for my projects I like to use https://github.com/Bachmann1234/diff_cover , which does coverage and quality diffing (using pylint or pycodestyle or flake8; I tend to use pylint) right 'inline' in tox; the nice thing about that is it's not tied to the CI system, you can easily port between CI systems since all the CI has to do is run tox. These days I also like to use https://github.com/psf/black .

Any thoughts on those ideas, or other ones? @WolfgangFahl , if you file an issue with whatever problems you had with 1.39, I promise to take a look!

[RheingoldRiver]

Hey @waldyrious thanks for the ping & for getting involved, it's great to see this project coming back to life! I had actually sort of "forked" this project a few years ago by way of writing a wrapper for it that had alternative methods as well as improved retries that caught some exceptions mwclient misses, and a Cargo client, see mwcleric.

Currently I'm on a break from work and not actively involved with any MediaWiki projects, though, I'm in between having a couple surgeries from this past October to this upcoming July. So, I'm not sure if I would actively contribute much, but I would be happy to participate in discussions and help to review PRs. So, if you want to add me as a contributor for that I'd be happy to, but i'm not sure if I'd make any major contributions in the near future. If not, no hard feelings, and I'll reach out again later this year if my situation changes!

[waldyrious]

Nice, that's great to hear, @RheingoldRiver :) I hope you get a speedy recovery from the surgeries.

No worries about not contributing code — being around to review PRs or chime in on discussions is already pretty valuable! I'll go ahead and invite you to the org. Thanks for sticking around and offering to help!

[RheingoldRiver]

Sounds good - I accepted the invitation, and looking forward to doing what I can!

[AdamWill]

One other idea I'd like to look at: functional tests. There is an official mediawiki container that seems easy to deploy; I think it'd be very useful to have tests that deploy that container and then test mwclient against it, so we know whether we're working against a real live mediawiki instance. That would allow us to onboard new mwclient releases by just extending the functional tests to run against the new release...

[marcfrederick]

Hi @waldyrious thanks for getting in touch! I'm currently not involved with any MediaWiki projects anymore and rather busy with my studies, but would be happy to help where I can (Review some PRs/Do some work on the documentation) when I find the time. Maybe it would even be useful to add some type-hints after Python 2 support is dropped, as mentioned by @AdamWill?

[waldyrious]

That's great to hear @marcfrederick — any improvements are more than welcome 😊 thanks for the willingness to help! I'll send you an invitation as well.

[marcfrederick]

Thanks @waldyrious, I've accepted the invitation. Looking forward to help where I can 😊

[waldyrious]

Awesome! @marcfrederick and @RheingoldRiver, if you don't mind, I'd ask you to make your membership of the org public, so that people who are not members can see that you're part of it.

[WolfgangFahl]

@waldyrious please also invite me - i'd be willing to work on the pypi and CI parts and README badges and the like.

[yzqzss]

One other idea I'd like to look at: functional tests. There is an official mediawiki container that seems easy to deploy; I think it'd be very useful to have tests that deploy that container and then test mwclient against it, so we know whether we're working against a real live mediawiki instance. That would allow us to onboard new mwclient releases by just extending the functional tests to run against the new release...

Hi, I have deployed different versions of the MediaWiki site from 1.16 to 1.39 for testing. Please feel free to use them, either to edit entries or to upload simple test files.

See: mediawiki-client-tools/mediawiki-dump-generator#7 (comment)

Also, the oldest version of the official docker image available is 1.27, which is still "too new" for webmasters who are lazy to upgrade. :-)

[AdamWill]

thanks a lot! I tend to like tests that are entirely self-contained and can run without a network connection, because Fedora package builds run without a network connection (most distros do the same), but hey, it might turn out that's the best choice. Various things are on fire in Fedora ATM so I haven't had time to do any more big mwclient stuff, but hopefully soon.

edit: hmm, to do an entirely offline test using the docker image we'd have to provide that image as an input anyway, I suppose...oh well, I'll burn this bridge when I get to it ;)

[WolfgangFahl]

@yzqzss You can also use a dockerized version of the mediawiki sites which can also be used in the CI.
See https://github.com/WolfgangFahl/pymediawikidocker where this is fully automated and where the extensions to be used may be specified while at so usecases revolving around problems with extensions might be tested.

See https://wiki.bitplan.com/index.php/Pymediawikidocker#Examples for examples

See https://github.com/WolfgangFahl/pymediawikidocker/actions for the CI usage
Defined in https://github.com/WolfgangFahl/pymediawikidocker/blob/main/.github/workflows/build.yml
Which simply uses

scripts/install
scripts/test

as the CI commands - which starts python commands to run the whole show
using the excellent https://github.com/gabrieldemarmiesse/python-on-whales library

[AdamWill]

Hey folks! Did we get any progress on access to pypi yet?

[WolfgangFahl]

If i get invited i might be able to work on the CI and pypi

[AdamWill]

As this has been sitting around for a while and we've landed some fairly big changes that really could do with a new pypi release, I've tried sending out an email to all current github and pypi maintainers (except one I couldn't find an email address for) to see if we can get at least one of the active github maintainers access to pypi.

[NguoiDungKhongDinhDanh]

@AdamWill If Danmichaelo the maintainer is this Danmichaelo, we are in luck. He is active until recently (Jan 3) according to GUC.

[AdamWill]

Yes, but he's not super responsive to mwclient stuff these days :D However, yuvipanda kindly gave @waldyrious access in response to my email, so if he can take care of it or add some of the rest of us new maintainers as well, we might be able to finally do a new release :)

[waldyrious]

Hi all! First of all, sorry for my lengthy absence from the project! I have finally caught up on the dozens of GitHub notifications I had pending for mwclient, and am able to chime in on a few threads, including this one. But before that, I just want to thank @AdamWill and @marcfrederick for their active maintenance of the project since we last interacted in this thread.

As for the pypi access, I'm happy to say I have just accepted YuviPanda's invitation to join the mwclient project there — and I didn't even have to do anything else because he added @AdamWill too :) so we have finally surpassed that long-term block and will be able to release new versions.

Wolfgang — I have seen of your request for maintainer access, but would prefer if you first got a few more contributions under your belt, via regular PRs. As I mentioned above, for Adam, Marc and River, I based my decision on their past contributions to the project.

But that said, I don't feel like I am able to (or should) make such maintenance decisions for the project from now on, given my low involvement — which, unfortunately will likely continue for the foreseeable future. So I'll go ahead and make Adam and Marc owners in the GitHub org, so that they can fully manage the project —including adding additional maintainers— without depending on me.

Finally, I want to say that this project feels very much alive now, under the stewardship of Adam and Marc, so I believe this issue can be closed (especially now the current maintainers also have access to pypi). WDYT?

[AdamWill]

Thanks waldy!

Yes, I agree closing this issue seems reasonable. There are various good ideas lying around in it, but we should probably file each of those separately for better organization :)

I think we probably have enough stuff in place to go ahead and cut a new release soon - with the useful bugfixes we've merged from various folks, and the cutoff of Python 2.x support. I'll update the ticket about that and try to co-ordinate with Marc on doing a release.

[waldyrious]

Awesome, @AdamWill — looking forward to seeing all that! I'll go ahead and close this issue, then.

[WolfgangFahl]

The idea was to get a new release so it's good that this is coming soon. For testing you might want to use https://github.com/WolfgangFahl/pymediawikidocker which allows you to setup a whole farm of dockerized mediawikis in one go to test different combinations of extensions and versions.

[AdamWill]

yes, that looks useful indeed. thanks for the pointer.

[WolfgangFahl]

Any progress? Will you be at the Wikimedia Hackathon in Tallinn?

[AdamWill]

sorry, I've been busy with Fedora 40 release cycle stuff so hadn't had any time to get back here. But yes, there has been some progress, we updated some CI stuff and merged some patches. I think we could do a release now, all we really need is a release pipeline; I don't want to hand-craft it, so I wanted to set up a GHA pipeline that will automatically do all the release steps when we tag a release, but I need a day or two of clear space for that.

I won't be at that event, no (I'm on the west coast of North America and I try to avoid unnecessary transatlantic travel), not sure if anyone else will be.