Querying more than (500/5000) results in an automated way?

[lovelaced]

Hi,

I'm attempting to get the results of a query (categorymembers) and the number of results are more than 500. I'm cool with doing multiple queries, but is there a way to continue where I left off? I know the "cmcontinue" param is available in raw_res in the categorymembers function, but I'm not sure if I can leverage it directly to get the results I want, or if I'm missing something.

For example, let's say I want to use the user default max (500) to get a list of all the pages that exist in a category, but there's 8000 pages in the category. Is it possible to loop a query to get all the pages?

[barrust]

Can you provide me with the category you are looking to pull from? The category tree does get all records but this could be something added in.

[lovelaced]

I'm looking at this:
http://practicalplants.org/wiki/Category:Plant

Thanks, I gave categorytree a try as well but it only gives me a short list of pages, strangely (the entire first page and halfway through the second page, up to Anemone altaica, which is 500 entries).

{'Plant': {'depth': 0, 'sub-categories': {}, 'links': ['Abelia triflora', ... , 'Anemone altaica'], 'parent-categories': []}}

The categorymembers query returns 500 entries as well if I set "results" to anything higher than 500. I'm calling it like this:

all_plant_names = plantwiki.categorymembers("Plant", results=8000, subcategories=False)

Forgive me if I'm overlooking something obvious. If there's something that needs changing here I'm also happy to contribute.
Thanks so much for your help!

[barrust]

You should be able to get all the category members using the following:

from mediawiki import MediaWiki

wiki = MediaWiki('http://practicalplants.org/w/api.php')
all_plant_names = wiki.categorymembers('Plants', results=None, subcategories=False)  

After a bit of poking around, it seems as though pymediawiki is looking for continue in the resulting request where the api you are hitting is sending back query-continue. I will need to look into it some more, but if you replaced continue with query-continue it should work (~ line 590)

if 'continue' not in raw_res or last_cont == raw_res['continue']:
                break

It may take me some time to get this working and to see if this is a long-term change or if the query-continue is an older construct.

[barrust]

@lovelaced I seemed to have tracked down the issue to a difference between actions and props and how they handle the continue parameter. I have a fix in the works with a test to ensure that it is resolved. I hope to push it to PyPi in the coming day. If you would like to try the fix it is currently in the hotfix/query-continue branch.

With this, if you set the number of results to return to None, it will pull back all the category members.

[barrust]

I pushed the change to pypi as version 0.3.17; if you would upgrade pymediawiki and test again, that would be great! I actually used this category in the test suite.

[barrust]

@lovelaced I am going to close this issue. If something is still not working, please reopen and let me know!