Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rebuildData.php gives error: BIGINT UNSIGNED value is out of range in smw_prop_stats.usage_count #4926

Open
Malvineous opened this issue Feb 7, 2021 · 8 comments
Open

rebuildData.php gives error: BIGINT UNSIGNED value is out of range in smw_prop_stats.usage_count #4926

Malvineous opened this issue Feb 7, 2021 · 8 comments
Labels
bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error

Comments

@Malvineous
Copy link

Malvineous commented Feb 7, 2021 •
edited by kghbln

Setup

  • SMW version: 3.2.2
  • MW version: 1.35.1
  • PHP version: 7.3
  • DB system (MySQL, Blazegraph, etc.) and version: MariaDB 10.3.27

Issue

I am upgrading SMW from 2.5 to 3.2.2 and upon completion the wiki came up saying I was missing an upgrade key. I ran populateHashField.php which got the wiki going but the SMW elements were blank. I thought maybe I have to rebuild the data, which I did by running rebuildData.php however this fails half way through each time, with or without --auto-recovery.

I had already run the usual maintenance/update.php so I tried SMW's setupStore.php which seemed to output much the same as update.php did, all of which completed without any problem. However I still can't get rebuildData.php to complete and the copy of the wiki I am testing this on still appears to be missing data in the SMW properties.

The message is long and a bit difficult to decipher so I included it all:

Rebuilding semantic data ...
   ... selecting 1 to 13793 IDs ...
   ... updating ...                                     1492 / 13798 ( 11%)


Wikimedia\Rdbms\DBTransactionStateError from line 1501 of /../w/includes/libs/rdbms/database/Database.php: Cannot execute query from SMW\SQLStore\PropertyStatisticsStore::setUsageCount while transaction status is ERROR
#0 /../w/includes/libs/rdbms/database/Database.php(1220): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...')
#1 /../w/includes/libs/rdbms/database/Database.php(2457): Wikimedia\Rdbms\Database->query('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#2 /../w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update('`smw_prop_stats...', Array, 'p_id = 13209', 'SMW\\SQLStore\\Pr...', Array)
#3 /../w/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call('update', Array)
#4 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Connection/Database.php(536): Wikimedia\Rdbms\DBConnRef->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...', Array)
#5 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(210): SMW\MediaWiki\Connection\Database->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...')
#6 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(118): SMW\SQLStore\PropertyStatisticsStore->setUsageCount(13209, Array)
#7 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(164): SMW\SQLStore\PropertyStatisticsStore->addToUsageCount(13209, -1)
#8 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyTableUpdater.php(97): SMW\SQLStore\PropertyStatisticsStore->addToUsageCounts(Array)
#9 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(374): SMW\SQLStore\PropertyTableUpdater->update(1493, Object(SMW\Parameters))
#10 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(226): SMW\SQLStore\SQLStoreUpdater->doFlatDataUpdate(Object(SMW\SemanticData))
#11 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(316): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(Object(SMW\SemanticData))
#12 /../w/extensions/SemanticMediaWiki/src/Store.php(234): SMW\SQLStore\SQLStore->doDataUpdate(Object(SMW\SemanticData))
#13 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(415): SMW\Store->updateData(Object(SMW\SemanticData))
#14 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(294): SMW\DataUpdater->updateData()
#15 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(198): SMW\DataUpdater->runUpdate()
#16 /../w/extensions/SemanticMediaWiki/src/ParserData.php(454): SMW\DataUpdater->doUpdate()
#17 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(303): SMW\ParserData->updateStore()
#18 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(216): SMW\MediaWiki\Jobs\UpdateJob->updateStore(Object(SMW\ParserData))
#19 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(137): SMW\MediaWiki\Jobs\UpdateJob->parse_content()
#20 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(94): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#21 /../w/extensions/SemanticMediaWiki/src/SQLStore/Rebuilder/Rebuilder.php(239): SMW\MediaWiki\Jobs\UpdateJob->run()
#22 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(466): SMW\SQLStore\Rebuilder\Rebuilder->rebuild(1493)
#23 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(383): SMW\Maintenance\DataRebuilder->doUpdateById(1493)
#24 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(222): SMW\Maintenance\DataRebuilder->rebuildAll()
#25 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(198): SMW\Maintenance\DataRebuilder->rebuild()
#26 /../w/maintenance/doMaintenance.php(107): SMW\Maintenance\rebuildData->execute()
#27 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once('/srv/web/mediaw...')
#28 {main}

Wikimedia\Rdbms\DBQueryError from line 1699 of /../w/includes/libs/rdbms/database/Database.php: Error 1690: BIGINT UNSIGNED value is out of range in '`keenwiki`.`smw_prop_stats`.`usage_count` - 1' (localhost)
Function: SMW\SQLStore\PropertyStatisticsStore::addToUsageCount
Query: UPDATE  `smw_prop_stats` SET usage_count = usage_count - 1,null_count = null_count - 0 WHERE p_id = 13209

#0 /../w/includes/libs/rdbms/database/Database.php(1323): Wikimedia\Rdbms\Database->getQueryException('BIGINT UNSIGNED...', 1690, 'UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...')
#1 /../w/includes/libs/rdbms/database/Database.php(1223): Wikimedia\Rdbms\Database->executeQuery('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#2 /../w/includes/libs/rdbms/database/Database.php(2457): Wikimedia\Rdbms\Database->query('UPDATE  `smw_pr...', 'SMW\\SQLStore\\Pr...', 128)
#3 /../w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update('`smw_prop_stats...', Array, 'p_id = 13209', 'SMW\\SQLStore\\Pr...', Array)
#4 /../w/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call('update', Array)
#5 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Connection/Database.php(536): Wikimedia\Rdbms\DBConnRef->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...', Array)
#6 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(112): SMW\MediaWiki\Connection\Database->update('smw_prop_stats', Array, Array, 'SMW\\SQLStore\\Pr...')
#7 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php(164): SMW\SQLStore\PropertyStatisticsStore->addToUsageCount(13209, -1)
#8 /../w/extensions/SemanticMediaWiki/src/SQLStore/PropertyTableUpdater.php(97): SMW\SQLStore\PropertyStatisticsStore->addToUsageCounts(Array)
#9 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(374): SMW\SQLStore\PropertyTableUpdater->update(1493, Object(SMW\Parameters))
#10 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStoreUpdater.php(226): SMW\SQLStore\SQLStoreUpdater->doFlatDataUpdate(Object(SMW\SemanticData))
#11 /../w/extensions/SemanticMediaWiki/src/SQLStore/SQLStore.php(316): SMW\SQLStore\SQLStoreUpdater->doDataUpdate(Object(SMW\SemanticData))
#12 /../w/extensions/SemanticMediaWiki/src/Store.php(234): SMW\SQLStore\SQLStore->doDataUpdate(Object(SMW\SemanticData))
#13 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(415): SMW\Store->updateData(Object(SMW\SemanticData))
#14 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(294): SMW\DataUpdater->updateData()
#15 /../w/extensions/SemanticMediaWiki/src/DataUpdater.php(198): SMW\DataUpdater->runUpdate()
#16 /../w/extensions/SemanticMediaWiki/src/ParserData.php(454): SMW\DataUpdater->doUpdate()
#17 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(303): SMW\ParserData->updateStore()
#18 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(216): SMW\MediaWiki\Jobs\UpdateJob->updateStore(Object(SMW\ParserData))
#19 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(137): SMW\MediaWiki\Jobs\UpdateJob->parse_content()
#20 /../w/extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(94): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#21 /../w/extensions/SemanticMediaWiki/src/SQLStore/Rebuilder/Rebuilder.php(239): SMW\MediaWiki\Jobs\UpdateJob->run()
#22 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(466): SMW\SQLStore\Rebuilder\Rebuilder->rebuild(1493)
#23 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(383): SMW\Maintenance\DataRebuilder->doUpdateById(1493)
#24 /../w/extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(222): SMW\Maintenance\DataRebuilder->rebuildAll()
#25 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(198): SMW\Maintenance\DataRebuilder->rebuild()
#26 /../w/maintenance/doMaintenance.php(107): SMW\Maintenance\rebuildData->execute()
#27 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once('/srv/web/mediaw...')
#28 {main}
[52e992da63a5b67df4e6f3f0] [no req]   Wikimedia\Rdbms\DBTransactionError from line 1519 of /../w/includes/libs/rdbms/database/Database.php: Explicit transaction still active. A caller may have caught an error. Open transactions: sql/transaction/update
Backtrace:
#0 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1642): Wikimedia\Rdbms\Database->assertNoOpenTransactions()
#1 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2150): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /../w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1661): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#3 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(244): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#4 /../w/includes/libs/rdbms/lbfactory/LBFactorySimple.php(133): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(246): Wikimedia\Rdbms\LBFactorySimple->forEachLB(Closure, array)
#6 /../w/includes/libs/rdbms/lbfactory/LBFactory.php(305): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /../w/maintenance/doMaintenance.php(135): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string)
#8 /../w/extensions/SemanticMediaWiki/maintenance/rebuildData.php(294): require_once(string)
#9 {main}
@Malvineous Malvineous added the bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error label Feb 7, 2021
@akuckartz
Copy link

@Malvineous Did you follow these suggestions? https://www.semantic-mediawiki.org/wiki/Help:Upgrade/Upgrade_from_SMW_2.5_to_SMW_3.0

@Malvineous
Copy link
Author

I did, but I'm not sure how accurate the steps are because it tells you to run php extensions/SemanticMediaWiki/maintenance/setupStore.php -v but when I do that it complains Unexpected option v! I ran it without the -v which is where it came up with the same output as the end of update.php did, so I think update.php runs setupStore.php anyway. That page was where I found to run populateHashField.php to fix the update hash error, and everything works up until I get to the rebuildData.php step which fails with the above exception.

@Malvineous
Copy link
Author

I tried altering extensions/SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php at line 164 where the error was, and added an extra if condition around it:

+if ($addition > 0) {
     $success = $this->addToUsageCount( $pid, $addition ) && $success;
+}

This fixed the error and allowed the process to complete. Not sure exactly what it has done or the implications of skipping that line for the negative values.

@cure
Copy link

cure commented Apr 15, 2021

I did, but I'm not sure how accurate the steps are because it tells you to run php extensions/SemanticMediaWiki/maintenance/setupStore.php -v but when I do that it complains Unexpected option v!

I ran into the same problem. Documentation bug?

@kghbln
Copy link
Member

kghbln commented Apr 15, 2021

Indeed, "setupStore.php" does not have a v option. Sorry for this. Amended in the docu.

@robis24
Copy link

robis24 commented Jul 27, 2021

I had the same error,
Fixed it by rebuilding everything with running
php setupStore.php --delete

After this, proceed as if re-installing SMW anew by first running
php setupStore.php

@krabina
Copy link
Contributor

krabina commented Jan 3, 2023 •
edited

We ran into the same problem in an upgrade from 1.31 to 1.35 (SMW 3 to 4.x) and improved the situation by adding the following code in line 110

SemanticMediaWiki/src/SQLStore/PropertyStatisticsStore.php

Line 110 in 23b1044

'p_id' => $pid 

                                        'p_id' => $pid,
                                        'usage_count + 10000000' . ( $usageVal > 0 ? '+ ' : '- ' ) . $this->connection->addQuotes( abs( $usageVal ) ) . ' >= 10000000',
                                        'null_count + 10000000' . ( $nullVal > 0 ? '+ ' : '- ' ) . $this->connection->addQuotes( abs( $nullVal ) ) . ' >= 10000000'

Maybe someone can test this or come up with an even better solution?

@kghbln
Copy link
Member

kghbln commented May 17, 2024 •
edited

I got this on https://sandbox.semantic-mediawiki.net/wiki/Utilisateur:WinfredGeake28, a previously non-existing page for the following setup:

Product Version
MediaWiki 1.35.14 (ef29848)
PHP 7.4.33 (apache2handler)
MariaDB 10.5.23-MariaDB-0+deb11u1
Semantic MediaWiki 4.1.2 (5c94879) 20:01, 29 July 2023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cure @Malvineous @akuckartz @kghbln @robis24 @krabina