Bring wiki to github

[waldyrious]

The wiki currently lives here: http://sourceforge.net/apps/mediawiki/mwclient. It should be moved to https://github.com/btongminh/mwclient/wiki.

There are only 3 users with edits, so it's probably ok to just copy the pages here with a quick note about the original authors in the bottom, just like what was made with the issues from sourceforge.

If someone wants to investigate a way to convert a mediawiki wiki to a git repo (github wikis are git repositories), that would be nice as well, but just an extra, so don't sweat it :)

[waldyrious]

Here's a list of pages to move. Anyone is invited to move one or two pages, so we can gradually get this done without massive work by a single person. The conversion basically consists in replacing mediawiki syntax with markdown (''→*, '''→**, ==→##, [url title]→[title](url), etc.), replacing template calls with the actual template content (they're basically italic text, nothing fancy) and adding a migration footer (see the already converted pages to copy and adapt the notice code).

• Documentation
  → Documentation
• [Main Page](https://sourceforge.net/apps/mediawiki/mwclient/index.php?title=Main Page)
  → Home
• MaximumRetriesExceeded
  → MaximumRetriesExceeded
• MwClientError
  → MwClientError
• [Namespace index](https://sourceforge.net/apps/mediawiki/mwclient/index.php?title=Namespace index)
  → [Namespace index](../wiki/Namespace index)
• Page
  → Page
• Page.can
  → Page.can
• Page.delete
  → Page.delete
• Page.edit
  → Page.edit
• Page.namespace
  → Page.namespace
• Page.save
  → Page.save
• README
  → README.rst
• REFERENCE
  → REFERENCE.md
• Site
  → Site
• Site.Categories
  → Site.Categories
• Site.Images
  → Site.Images
• Site.Pages
  → Site.Pages
• [Site. init](https://sourceforge.net/apps/mediawiki/mwclient/index.php?title=Site. init)
  → Site.init
• Site.allpages
  → Site.allpages
• Site.login
  → Site.login
• [Tutorial:Creating a page listing all pages in a category](https://sourceforge.net/apps/mediawiki/mwclient/index.php?title=Tutorial:Creating a page listing all pages in a category)
  → [Tutorial:Creating a page listing all pages in a category](../wiki/Tutorial:Creating a page listing all pages in a category)
• [Tutorial:Replacing a string for every page in a category](https://sourceforge.net/apps/mediawiki/mwclient/index.php?title=Tutorial:Replacing a string for every page in a category)
  → [Tutorial:Replacing a string for every page in a category](../wiki/Tutorial:Replacing a string for every page in a category)

It might also be a good idea to include relevant information from

• this tutorial
  → [Tutorial:Querying Wikipedia with mwclient](../wiki/Tutorial:Querying Wikipedia with mwclient)

[waldyrious]

Notifying @DCoetzee and @dbolser since they collaborated in the sourceforge wiki.

[danmichaelo]

Hm, most of these pages are descriptions of classes and methods. I believe it would be easier to keep up-to-date if these were generated from comments in the source code. What do you think about using Sphinx and readthedocs.org? The latter is quite cool, since it hooks up with GitHub. I made a very simple and incomplete test and the result is here. Currently the source-code comments are not optimized for Sphinx, but I could work with that.

[waldyrious]

That would be great! I've never worked with Shpinx before. How can I help? What do I need to know?

[danmichaelo]

I've not really worked with it before either, just done some testing, so I'm not really sure :) I guess I can continue setting up the structure and start moving info from the wiki into the source code, and if it's a lot of work we can share it. But if you have a look at #21 that's great. Then I can send in another pull request for code-cleanup, to have a better starting point.

[waldyrious]

Let me see if I got this correctly: so we setup Sphynx by creating/customizing the appropriate configuration files, and then hook it to readthedocs, which periodically rebuilds the API documentation without any human intervention. If that's the case, does that mean that the documentation folder doesn't actually contain the generated files? That's not necessarily a bad thing especially if the documentation comes from the code; I'm just asking.

Do you see any way to have a similar process based on docco/pycco and gh-pages (i.e. hosting the pages directly on mwclient.github.io)? I think that would be nicer. I've read a bit about git hooks, but it seems github only supports "webhooks", not running arbitrary programs such as pycco (which makes sense). Client-side hooks could work for this but I'm not sure we can setup the repo so that anyone who clones it gets the hooks as well... any ideas?

[danmichaelo]

That's right. You provide a configuration file and some "templates", and can then build the documentation by typing make html in the docs folder. It ends up in the docs/_build folder, which we could of course put under version control, but I think it's better not to since it's not source (which is why I added _build to my .gitignore). Readthedocs can build the documentation automatically after each commit using a commit hook (or manually pressing a button).

Looks lite it's possible to host directly on github, but I'm not sure if that's necessarily an advantage. Readthedocs provides versioning and a nicer stylesheet than the default. I'm not familiar with pycco, but it looks much less powerful than Sphinx (well, minimalism can be a good thing, but not too much of it :))

[waldyrious]

I actually have no bias towards hosting on github rather than readthedocs; I only mentioned it because RTD seems to only support sphynx, and I couldn't find any other similar documentation hosting service which could perhaps have different requirements. Indeed, I would prefer a simpler approach such as pycco (what powerful features would we need from sphynx, btw?); that's why I asked about "self"-hosting + local hooks, as that would be equivalent to automatically run make html (for pycco, specifically, it'd be pycco mwclient/*.py) in our local machines, after each commit. I am afraid it's not possible to propagate hooks across clones, though, and was hoping you'd prove me wrong :)

[waldyrious]

Indeed, hook propagation isn't possible. Sphinx it is, then. Can I start moving wiki documentation into the code, or are you doing something locally?

[danmichaelo]

Nope, not yet, so feel free to start. Btw. it's the autodoc extension that pulls in data from the docstrings.

[waldyrious]

I tried reading that page but it seemed to require quite a lot of custom documentation markup to get autodoc to work. Apparently I wasn't the only one thinking the same. Fortunately, the solution mentioned on that page has since been integrated as the sphinx-apidoc tool, which is meant to be used with autodoc.

This tutorial seems to imply that no special documentation format is required when adding docstrings to be extracted with apidoc+autodoc. But surely there must be some way to, say, describe parameters, and I assume rST formatting is parsed as well. Do you have any insight regarding this? Once I know how to add the docstrings I will start adding the information from the wiki to the actual code.

[waldyrious]

btw, the coverage extension also seems like a useful one. I wonder if it could somehow be integrated into the tests, or something.

[danmichaelo]

My impression is that there are few standards when it comes to Python documentation... I tried to look at some other projects now. The MongoDB driver for instance seems to have a good style:

https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/mongo_client.py

It uses :Parameters: and then lists the parameters as a normal list, and the result is this:

http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#module-pymongo.mongo_client

Coverage testing is nice. I'm not sure how it implements with Sphinx, but I'm using coverage with py.test in mwtemplates (name is of course inspired by mwclient). Btw. I'm somewhat uncertain how we should write unit tests for mwclient. It seems like those who are really into unit testing prefer to mock http requests, but it seems like a lot of work to make mockups for all the API methods, and I'm not sure if I see the point of doing it. Tests will run faster and be less dependent on external factors, yes, but on the other hand it's interesting to test that actual requests work. Write requests are difficult though. Any thoughts?

[waldyrious]

Update: mwclient is now managed through an organization, currently managed by @btongminh, @danmichaelo and me. We can now set up the ReadTheDocs hook and try to import some documentation from the sourceforge wiki. Do you want to do that, @danmichaelo?

I did look into the mongodb example but I'm a little unsure about the keywords and formatting. For example, the list here, while formatted similarly to the mongodb example here, doesn't seem to be interpreted the same way by Sphynx. I'm not sure whether that has to do with using the "parameters" keyword before it, or something else; I'd be more comfortable to start from a mwclient-specific example and work up using the readthedocs output for feedback.

Btw, I have no opinions on the unit tests.

[btongminh]

Also, the old wiki should be updated to point to Github. The files in the
SF svn should probably also redirect to Github some way. Waldir, did I
already give you admin access on SF?

On Wed, Jul 3, 2013 at 11:57 PM, Waldir Pimenta notifications@github.comwrote:

Update: mwclient is now managed through an organization, currently managed
by @btongminh https://github.com/btongminh, @danmichaelohttps://github.com/danmichaeloand me. We can now set up the ReadTheDocs hook and try to import some
documentation from the sourceforge wiki. Do you want to do that,
@danmichaelo https://github.com/danmichaelo?

I did look into the mongodb example but I'm a little unsure about the
keywords and formatting. For example, the list herehttps://github.com/danmichaelo/mwclient/blob/sphinx/mwclient/client.py#L489,
while formatted similarly to the mongodb example herehttps://github.com/mongodb/mongo-python-driver/blob/master/pymongo/mongo_client.py#L117,
doesn't seem to be interpreted the same way by Sphynx. I'm not sure whether
that has to do with using the "parameters" keyword before it, or something
else; I'd be more comfortable to start from a mwclient-specific example and
work up using the readthedocs output for feedback.

Btw, I have no opinions on the unit tests.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/12#issuecomment-20447515
.

[waldyrious]

@btongminh nope, not yet — I was going to ask for it after the migration was finished, ayway ;)

[btongminh]

Done now.

On Fri, Jul 5, 2013 at 6:45 AM, Waldir Pimenta notifications@github.comwrote:

@btongminh https://github.com/btongminh nope, not yet — I was going to
ask for it after the migration was finished, ayway ;)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/12#issuecomment-20501170
.

[waldyrious]

Thanks! I'll make sure to update it as soon as things are settled around here :)

[waldyrious]

This has to be completed before June 19th, since SourceForge will be discontinuing the mediawiki hosting service on that date and migrating it to a new service. I don't know how much detail is preserved in the migration, e.g. page history.

[danmichaelo]

Ok, taking note of the date

[waldyrious]

Update: I've just moved another bunch of pages, and edited the wiki a little (updated the main page, converted the "Documentation" page to the sidebar, since it was an index anyway, etc). Right now the only pages that still need to be moved are those of the Site class and its methods :)

[waldyrious]

Ok, all pages have now been moved! 🎉

I included @jace's tutorial as well. @pfctdayelise's demo code was suggested here, but it's not really a tutorial, so I'm not sure it would either fit with the others, or be useful considering the rest of the documentation is quite comprehensive actually (thanks @DCoetzee!).

@danmichaelo I'll open a new issue for discussing the setup of in-code documentation.

[jace]

Thanks for keeping those pages alive.