SemanticMediaWikiImporter.php not ending when all data is exported

[jthingelstad]

Now that issue #1 is closed I reran the importer to generate data. It should have stopped with 21,516 rows (http://wikinosh.com/wiki/Category:Food) but the file went to 22,900 rows and then I CTRL-C'd the task.

http://wikinosh.com/miga/apps/wikinosh/Food.csv

Looking at the contents of that file, it seems to not be entirely sequential. Actually, doing a

grep "Agar Seaweed" Food.csv

on that CSV shows the same data over and over. Ugh. Not sure what is causing this issue, it's likely some SMW issue. The net result is that the importer never stops and runs forever.

If you want to test yourself, this is my import settings:

<?php
$gImportFileName = "Food.csv";
$gImportSpecialAskURL = "http://wikinosh.com/wiki/Special:Ask";
$gImportCategoryName = "Category:Food";
$gImportFields = array(
        'Name' => '_name',
        'Calories' => 'Has calories',
        'Fat' => 'Has fat',
        'Carbohydrates' => 'Has carbohydrates',
        'Protein' => 'Has protein'
);

[yaronkoren]

Yes, that's pretty bad - it looks like Special:Ask simply ignores values of "offset" higher than 5000. I don't know if there's any easy way to fix that. If not, you could always use the standard MediaWiki importer, as sad as that sounds...

[jthingelstad]

Oi... from SMW_QueryProcessor.php (line 611)

            $params['offset'] = array(
                    'type' => 'integer',
                    'default' => 0,
                    'negatives' => false,
                    'upperbound' => 5000 // TODO: make setting
            );

So, yeah. :-(

[jthingelstad]

There is an existing bugzilla issue on this.

https://bugzilla.wikimedia.org/show_bug.cgi?id=49203

[yaronkoren]

Thanks for the good detective work! I updated the docs accordingly.