Introduce single user and seen users sync #30485

butonic · 2018-02-14T12:40:13Z

Restart of #28212, but without additional logging. Based on #30412

We had to add a findUserIds method to the Account mapper to iterate over all users in batches. callForSeenUserstakes a callback but also initializes the user objects filling caches everywhere, an unwanted side effect that only slows down the sync.

We also introduced Iterators for Iterating over all users of a backend as well as seen users only. The case for a single user uses an ArrayIterator.

One change in behavior: to get a total count you now have to use --showCount. Counting might be expensive. So we disable it by default ...

Tasks for another PR:

on single user sync show before & after (requires bypassing the cache in the user manager)
show only changed values for synced users or highlight them
reflect changes in documentation

codecov · 2018-02-20T09:41:34Z

Codecov Report

Merging #30485 into master will increase coverage by 0.16%.
The diff coverage is 75.75%.

@@             Coverage Diff              @@
##             master   #30485      +/-   ##
============================================
+ Coverage     62.29%   62.46%   +0.16%     
- Complexity    18208    18232      +24     
============================================
  Files          1142     1145       +3     
  Lines         68210    68289      +79     
  Branches       1234     1234              
============================================
+ Hits          42494    42655     +161     
+ Misses        25355    25273      -82     
  Partials        361      361

Flag	Coverage Δ	Complexity Δ
#javascript	`52% <ø> (+0.02%)`	`0 <ø> (ø)`	⬇️
#phpunit	`63.65% <75.75%> (+0.17%)`	`18232 <38> (+24)`	⬆️

Impacted Files	Coverage Δ	Complexity Δ
core/Migrations/Version20170221114437.php	`0% <0%> (ø)`	`2 <1> (ø)`	⬇️
lib/private/User/Sync/AllUsersIterator.php	`100% <100%> (ø)`	`4 <4> (?)`
lib/private/User/Sync/SeenUsersIterator.php	`100% <100%> (ø)`	`4 <4> (?)`
lib/private/User/Sync/UsersIterator.php	`100% <100%> (ø)`	`5 <5> (?)`
lib/private/User/AccountMapper.php	`77.77% <100%> (+4.3%)`	`31 <6> (+6)`	⬆️
core/Command/User/SyncBackend.php	`55.17% <52.38%> (+55.17%)`	`42 <15> (+6)`	⬆️
lib/private/User/SyncService.php	`74.83% <92.85%> (+1.71%)`	`56 <3> (-1)`	⬇️
apps/files/lib/App.php	`25% <0%> (-12.5%)`	`3% <0%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54cd970...c14eb12. Read the comment docs.

tomneedham · 2018-02-20T10:01:17Z

Blocked by #30412

tomneedham · 2018-03-13T10:40:26Z

I will rebase

tomneedham · 2018-03-13T10:46:10Z

Rebased and squashed

tomneedham · 2018-03-13T13:06:04Z

@ownclouders rebase

ownclouders · 2018-03-13T13:06:06Z

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

ownclouders · 2018-03-13T13:06:24Z

Automated rebase with GitMate.io was successful! 🎉

tomneedham · 2018-03-13T13:28:43Z

We need to add unit tests for the iterators and the accountMapper findByUserIds

tomneedham · 2018-03-29T11:48:26Z

Rebased

tomneedham · 2018-04-03T13:12:40Z

rebase attempt #4 ......

PVince81 · 2018-04-03T16:16:57Z

@ownclouders rebase

ownclouders · 2018-04-03T16:16:59Z

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

ownclouders · 2018-04-03T16:17:22Z

Automated rebase with GitMate.io was successful! 🎉

PVince81 · 2018-04-03T19:38:27Z

tests pass now, only codecov is unhappy

butonic · 2018-04-04T10:00:35Z

@ownclouders rebase please

ownclouders · 2018-04-04T10:00:38Z

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

tomneedham · 2018-04-04T10:00:48Z

codecov was showing weird results

ownclouders · 2018-04-04T10:00:58Z

Automated rebase with GitMate.io was successful! 🎉

PVince81 · 2018-04-04T15:04:36Z

@tomneedham please review the last additions then merge

mrow4a

I tried this command already, and can confirm it worked properly, at least some time ago. Iterator seemed to work properly, and all tests are there. I just have one doubt in the query itself as mentioned below

mrow4a · 2018-04-06T09:56:15Z

lib/private/User/AccountMapper.php

+		$qb = $this->db->getQueryBuilder();
+		$qb->select('user_id')
+			->from($this->getTableName())
+			->orderBy('user_id'); // needed for predictable limit & offset


mrow4a · 2018-04-06T09:59:23Z

lib/private/User/AccountMapper.php

+		if ($hasLoggedIn === true) {
+			$qb->andWhere($qb->expr()->gt('last_login', new Literal(0)));
+		} else if ($hasLoggedIn === false) {
+			$qb->andWhere($qb->expr()->eq('last_login', new Literal(0)));


Is all fine here? In both cases you have Literal(0) @butonic

ohh, sorry just noticed gt and eq.

lock · 2019-07-31T11:02:08Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

butonic added enhancement 2 - Developing labels Feb 14, 2018

butonic self-assigned this Feb 14, 2018

butonic requested a review from tomneedham February 14, 2018 12:40

butonic force-pushed the single-user-sync-take-two branch from 1cf0296 to 96df587 Compare February 14, 2018 12:50

tomneedham added 3 - To Review and removed 2 - Developing labels Feb 20, 2018

tomneedham approved these changes Feb 20, 2018

View reviewed changes

ownclouders added the status/STALE label Feb 28, 2018

ownclouders removed the status/STALE label Mar 13, 2018

tomneedham force-pushed the single-user-sync-take-two branch from 2580334 to 0730372 Compare March 13, 2018 10:44

tomneedham changed the title ~~[WIP] Introduce single user and seen users sync~~ Introduce single user and seen users sync Mar 13, 2018

tomneedham added this to the development milestone Mar 15, 2018

butonic force-pushed the single-user-sync-take-two branch from 0730372 to 3615f08 Compare March 15, 2018 11:59

ownclouders added the status/STALE label Mar 23, 2018

tomneedham added the p1-urgent Critical issue, need to consider hotfix with just that issue label Mar 26, 2018

ownclouders removed the status/STALE label Mar 26, 2018

tomneedham force-pushed the single-user-sync-take-two branch from 3615f08 to f65b9bb Compare March 29, 2018 11:48

tomneedham mentioned this pull request Apr 3, 2018

[FR] user:sync should have an option to only sync 'seen' users #29644

Closed

tomneedham force-pushed the single-user-sync-take-two branch from f65b9bb to e671cd0 Compare April 3, 2018 13:12

ownclouders force-pushed the single-user-sync-take-two branch from e671cd0 to 07a4bf9 Compare April 3, 2018 16:17

introduce user:sync [--uid foo|--seenOnly|--showCount]

d1f84d6

ownclouders force-pushed the single-user-sync-take-two branch from 07a4bf9 to d1f84d6 Compare April 4, 2018 10:00

butonic added 2 commits April 4, 2018 13:59

test exception is logged

1bca4c8

add SyncBackendTest

c14eb12

PVince81 added the backport-request label Apr 4, 2018

mrow4a reviewed Apr 6, 2018

View reviewed changes

tomneedham approved these changes Apr 6, 2018

View reviewed changes

tomneedham merged commit 151b1b8 into master Apr 6, 2018

tomneedham deleted the single-user-sync-take-two branch April 6, 2018 10:56

tomneedham mentioned this pull request Apr 6, 2018

[stable10] Introduce seen and single user sync command line features #31025

Merged

ownclouders mentioned this pull request Apr 6, 2018

Allow single user sync of new users #31028

Closed

PVince81 removed the backport-request label Apr 10, 2018

ownclouders mentioned this pull request Apr 19, 2018

Confusing behaviour with single user syncing if the user doesn't exists #31207

Closed

lock bot locked as resolved and limited conversation to collaborators Jul 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce single user and seen users sync #30485

Introduce single user and seen users sync #30485

butonic commented Feb 14, 2018 •

edited

codecov bot commented Feb 20, 2018 •

edited

tomneedham commented Feb 20, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 13, 2018

ownclouders commented Mar 13, 2018

ownclouders commented Mar 13, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 29, 2018

tomneedham commented Apr 3, 2018

PVince81 commented Apr 3, 2018

ownclouders commented Apr 3, 2018

ownclouders commented Apr 3, 2018

PVince81 commented Apr 3, 2018

butonic commented Apr 4, 2018

ownclouders commented Apr 4, 2018

tomneedham commented Apr 4, 2018

ownclouders commented Apr 4, 2018

PVince81 commented Apr 4, 2018

mrow4a left a comment •

edited

mrow4a Apr 6, 2018

mrow4a Apr 6, 2018

mrow4a Apr 6, 2018 •

edited

lock bot commented Jul 31, 2019

Introduce single user and seen users sync #30485

Introduce single user and seen users sync #30485

Conversation

butonic commented Feb 14, 2018 • edited

codecov bot commented Feb 20, 2018 • edited

Codecov Report

tomneedham commented Feb 20, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 13, 2018

ownclouders commented Mar 13, 2018

ownclouders commented Mar 13, 2018

tomneedham commented Mar 13, 2018

tomneedham commented Mar 29, 2018

tomneedham commented Apr 3, 2018

PVince81 commented Apr 3, 2018

ownclouders commented Apr 3, 2018

ownclouders commented Apr 3, 2018

PVince81 commented Apr 3, 2018

butonic commented Apr 4, 2018

ownclouders commented Apr 4, 2018

tomneedham commented Apr 4, 2018

ownclouders commented Apr 4, 2018

PVince81 commented Apr 4, 2018

mrow4a left a comment • edited

Choose a reason for hiding this comment

mrow4a Apr 6, 2018

Choose a reason for hiding this comment

mrow4a Apr 6, 2018

Choose a reason for hiding this comment

mrow4a Apr 6, 2018 • edited

Choose a reason for hiding this comment

lock bot commented Jul 31, 2019

butonic commented Feb 14, 2018 •

edited

codecov bot commented Feb 20, 2018 •

edited

mrow4a left a comment •

edited

mrow4a Apr 6, 2018 •

edited