Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get out of the infinite loop in listing titles #309

Closed
nemobis opened this issue May 7, 2018 · 3 comments
Closed

Get out of the infinite loop in listing titles #309

nemobis opened this issue May 7, 2018 · 3 comments

Comments

@nemobis
Copy link
Member

nemobis commented May 7, 2018

Switch to the --xmlrevisions option when the titles cannot be retrieved:

Loading config file...
Resuming previous dump process...
Title list is incomplete. Reloading...
Loading page titles from namespaces = all
Excluding titles from namespaces = None
16 namespaces found
    Retrieving titles in the namespace 0
...........    5426 titles retrieved in the namespace 0
    Retrieving titles in the namespace 1
.    487 titles retrieved in the namespace 1
    Retrieving titles in the namespace 2
.    142 titles retrieved in the namespace 2
    Retrieving titles in the namespace 3
.....    2105 titles retrieved in the namespace 3
    Retrieving titles in the namespace 4
.    89 titles retrieved in the namespace 4
    Retrieving titles in the namespace 5
.    20 titles retrieved in the namespace 5
    Retrieving titles in the namespace 6
..    808 titles retrieved in the namespace 6
    Retrieving titles in the namespace 7
.    1 titles retrieved in the namespace 7
    Retrieving titles in the namespace 8
.    103 titles retrieved in the namespace 8
    Retrieving titles in the namespace 9
.    1 titles retrieved in the namespace 9
    Retrieving titles in the namespace 10
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
@nemobis
Copy link
Member Author

nemobis commented May 7, 2018

Another test case (wiki with 4k articles and 12k pages, public, 1.23.6):

Analysing http://cunnan.sca.org.au/api.php
Loading config file...
Resuming previous dump process...
Title list is incomplete. Reloading...
Loading page titles from namespaces = all
Excluding titles from namespaces = None
16 namespaces found
    Retrieving titles in the namespace 0
...........    5426 titles retrieved in the namespace 0
    Retrieving titles in the namespace 1
.    487 titles retrieved in the namespace 1
    Retrieving titles in the namespace 2
.    142 titles retrieved in the namespace 2
    Retrieving titles in the namespace 3
.....    2105 titles retrieved in the namespace 3
    Retrieving titles in the namespace 4
.    89 titles retrieved in the namespace 4
    Retrieving titles in the namespace 5
.    20 titles retrieved in the namespace 5
    Retrieving titles in the namespace 6
..    808 titles retrieved in the namespace 6
    Retrieving titles in the namespace 7
.    1 titles retrieved in the namespace 7
    Retrieving titles in the namespace 8
.    103 titles retrieved in the namespace 8
    Retrieving titles in the namespace 9
.    1 titles retrieved in the namespace 9
    Retrieving titles in the namespace 10
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

@nemobis
Copy link
Member Author

nemobis commented May 21, 2018

This last one seems to be a problem of capitalisation. The API tells to continue from a lowercases title but then it normalises it to uppercase and continues from a previous point:
http://cunnan.lochac.sca.org/api.php?action=query&list=allpages&apfrom=returnto&apnamespace=10&format=json&aplimit=500

@nemobis
Copy link
Member Author

nemobis commented Jun 2, 2018

So far I've not seen the bug any more, although 4bc41c3 increases the memory usage significantly in that phase.

@nemobis nemobis closed this as completed Jun 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant