Skip to content
This repository has been archived by the owner on Jun 7, 2023. It is now read-only.

SUS-1620 | Introduced User::touch() method to bump the getTouched() value using memcached #12923

Merged
merged 3 commits into from
May 16, 2017

Conversation

macbre
Copy link
Contributor

@macbre macbre commented May 16, 2017

https://wikia-inc.atlassian.net/browse/SUS-1620

User::invalidateCache will no longer try to update user.user_touched on shared database which was prone to deadlocks. Instead it will just set a key in memcache (via new User::touch method). This will greatly reduce the number of such queries (we currently make 640k of these every day, that's 68% of all UPDATE queries on shared db master node). invalidateCache is usually called when an edit / upload is made (via User::addWatch method) or when a user is created from an auth token (User::newFromToken method).

user_touched field is still updated when User::saveSettings is called (e.g. preferences are saved).

User::getTouched will now return the greater of these - user_touched field from user table / value stored in memcache (set via User::touch)

macbre@dev-macbre:~$ ./eval.sh 
> $u = User::newFromName('Macbre')
Unstubbing $wgAuth on call of $wgAuth::getCanonicalName from User::newFromName

> echo $u->getName()
Macbre

> echo $u->getTouched()
Query plpoznan (DB user: wikia_maint) (10) (slave): SELECT /* User::idFromName CommandLineInc - fcfd0fd0-87fc-44e9-b349-b4a368730599 */  user_id  FROM `wikicities_c1`.`user`  WHERE user_name = 'Macbre'  LIMIT 1  
memcached: get(dev-macbre-plpoznan:user:id:119245)
memcached: MemCache: sock i:0; got dev-macbre-plpoznan:user:id:119245
memcached: get(dev-macbre-wikicities:user_touched:v1:119245)
memcached: MemCache: sock i:0; got dev-macbre-wikicities:user_touched:v1:119245
User: got user 119245 from cache
memcached: get(dev-macbre-wikicities:user-quicktouched:id:119245)
memcached: set dev-macbre-wikicities:user-quicktouched:id:119245 (STORED)
20170516115729

> $u->invalidateCache()
memcached: set dev-macbre-wikicities:user-quicktouched:id:119245 (STORED)
memcached: MemCache: delete dev-macbre-plpoznan:user:id:119245 (DELETED)
User: loading options for user 119245 from override cache.
memcached: client: serializing data as it is not scalar
memcached: client: compressing data; was 10981 bytes is now 3259 bytes
memcached: set dev-macbre-plpoznan:user:id:119245 (STORED)
User: user 119245 stored in cache
memcached: set dev-macbre-wikicities:user_touched:v1:119245 (STORED)
Shared user: updating shared user_touched
memcached: MemCache: delete GLOBAL:Wikia\Service\User\Permissions\PermissionsServiceImpl:permissions-groups:119245 (DELETED)

> echo $u->getTouched()
20170516115756

> $u->saveSettings()
User: got user 119245 from cache
User: loading options for user 119245 from override cache.
Connecting to geo-db-dev-db-master.query.consul plpoznan...
DatabaseBase::query: Writes done: UPDATE  `wikicities_c1`.`user` SET user_name = 'Macbre',user_real_name = '',user_email = 'macbre@wikia-inc.com',user_email_authenticated = '20090126130335',user_touched = '20170516115817',user_token = '96f6670ef02fea4d2b2c3ade065d9aaf',user_email_token = '1f92a97d8d27e0ca2a6fecc3bc2c57d3',user_email_token_expires = '20090711205751' WHERE user_id = '119245'
Query plpoznan (DB user: wikia_maint) (11) (master): UPDATE /* User::saveSettings CommandLineInc - fcfd0fd0-87fc-44e9-b349-b4a368730599 */  `wikicities_c1`.`user` SET user_name = 'Macbre',user_real_name = '',user_email = 'macbre@wikia-inc.com',user_email_authenticated = '20090126130335',user_touched = '20170516115817',user_token = '96f6670ef02fea4d2b2c3ade065d9aaf',user_email_token = '1f92a97d8d27e0ca2a6fecc3bc2c57d3',user_email_token_expires = '20090711205751' WHERE user_id = '119245'
...
User: user 119245 stored in cache
memcached: set dev-macbre-wikicities:user_touched:v1:119245 (STORED)
Shared user: updating shared user_touched
LBFactory_Wikia::getSectionForWiki: section c1, wiki
Query plpoznan (DB user: wikia_maint) (17) (master): UPDATE /* Title::invalidateCache CommandLineInc - fcfd0fd0-87fc-44e9-b349-b4a368730599 */  `page` SET page_touched = '20170516115812' WHERE page_namespace = '2' AND page_title = 'Macbre'
{"appname":"mediawiki","@timestamp":"2017-05-16T11:58:12.358842+00:00","@message":"Important table write - SQL UPDATE /* Title::invalidateCache CommandLineInc - fcfd0fd0-87fc-44e9-b349-b4a368730599 */  `page` SET page_touched = '20170516115812' WHERE page_namespace = '2' AND page_title = 'Macbre'","@fields":{"app_name":"mediawiki","app_version":"dev","datacenter":"poz","environment":"dev","wiki_dbname":"plpoznan","wiki_id":"5915","maintenance_file":"/usr/wikia/source/app/maintenance/eval.php","maintenance_class":"CommandLineInc","client_ip":"127.0.0.1","span_id":"79fcdc83-2905-4bd3-9835-a39d6ef6af15","trace_id":"fcfd0fd0-87fc-44e9-b349-b4a368730599"},"@exception":{"class":"Exception","message":"","code":0,"file":"/usr/wikia/source/app/includes/db/Database.php:3732","trace":["/usr/wikia/source/app/includes/db/Database.php:3673","/usr/wikia/source/app/includes/db/Database.php:977","/usr/wikia/source/app/includes/db/Database.php:1897","/usr/wikia/source/app/includes/Title.php:4467","/usr/wikia/source/app/includes/User.php:3206","/usr/wikia/source/app/maintenance/eval.php(88) : eval()'d code:1","/usr/wikia/source/app/maintenance/eval.php:88"]},"@context":{"method":"Title::invalidateCache","elapsed":0.010771989822388,"num_rows":1,"transaction_level":0,"cluster":"c1","server":"geo-db-dev-db-master.query.consul","server_role":"master","db_name":"plpoznan","db_user":"wikia_maint"}}
memcached: set dev-macbre-plpoznan:page_touched:2_Macbre (STORED)

> echo $u->getTouched()
20170516115817

macbre and others added 3 commits May 16, 2017 12:04
…alue using memcached

* This lets some callers avoid the heavyweight invalidateCache() method

Backporting wikimedia/mediawiki@87d2eba (MW 1.25)
… query

* This method is used for clearing the User cache as well
  as bumping the value of getTouched() for HTTP 304 logic.
  These do not need to do the actual user_touched update.
* This also avoids problems with setting mTouched but
  deferring the update. That confused the CAS update
  logic since it expects mTouched to be in the DB.

Backporting wikimedia/mediawiki@5b4bd16 (MW 1.26)
@@ -130,12 +130,18 @@ class User implements JsonSerializable {
/** @name Cache variables */
//@{

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAJOR Remove this commented out code. rule

@WikiaTech
Copy link

SonarQube analysis reported 96 issues

  • MAJOR 58 major
  • MINOR 7 minor
  • INFO 31 info

Watch the comments in this conversation to review them.

Top 10 extra issues

Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

  1. MAJOR User.php: File "User.php" has 4687 lines, which is greater than 1000 authorized. Split it into smaller files. rule
  2. MAJOR User.php#L75: Class "User" has 219 methods, which is greater than 20 authorized. Split it into smaller classes. rule
  3. MAJOR User.php#L148: Remove this commented out code. rule
  4. MAJOR User.php#L155: Remove this commented out code. rule
  5. MAJOR User.php#L427: Remove this commented out code. rule
  6. MAJOR User.php#L506: Reduce the number of returns of this function 5, down to the maximum allowed 3. rule
  7. MAJOR User.php#L577: Remove this commented out code. rule
  8. MAJOR User.php#L677: Reduce the number of returns of this function 4, down to the maximum allowed 3. rule
  9. MAJOR User.php#L680: Reduce the number of conditional operators (4) used in the expression (maximum allowed 3). rule
  10. MAJOR User.php#L748: Replace this legacy wfMsg* function call with one of the new i18n functions. rule

@@ -130,12 +130,18 @@ class User implements JsonSerializable {
/** @name Cache variables */
//@{
var $mId, $mName, $mRealName,
$mEmail, $mTouched, $mToken, $mEmailAuthenticated,
$mEmail, $mToken, $mEmailAuthenticated,
$mEmailToken, $mEmailTokenExpires, $mRegistration, $mGroups, $mOptionOverrides,
$mCookiePassword, $mEditCount, $mAllowUsertalk;
var $mBirthDate; // Wikia. Added to reflect our user table layout.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

INFO Move this trailing comment on the previous empty line. rule

@@ -130,12 +130,18 @@ class User implements JsonSerializable {
/** @name Cache variables */
//@{
var $mId, $mName, $mRealName,
$mEmail, $mTouched, $mToken, $mEmailAuthenticated,
$mEmail, $mToken, $mEmailAuthenticated,
$mEmailToken, $mEmailTokenExpires, $mRegistration, $mGroups, $mOptionOverrides,
$mCookiePassword, $mEditCount, $mAllowUsertalk;
var $mBirthDate; // Wikia. Added to reflect our user table layout.
//@}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAJOR Remove this commented out code. rule

@macbre macbre requested review from a user, drozdo, mszabo-wikia and pwojcik86 May 16, 2017 12:09
@macbre macbre changed the title Sus 1620 handle user invalidate cache deadlocks SUS-1620 | Introduced User::touch() method to bump the getTouched() value using memcached May 16, 2017
@macbre macbre merged commit fa72dc2 into dev May 16, 2017
@macbre macbre deleted the SUS-1620-handle-user-invalidateCache-deadlocks branch May 16, 2017 12:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
4 participants