Central user account table arriving ..... #27165

Open
wants to merge 8 commits into
from

Projects

None yet

4 participants

@DeepDiver1975
Member
DeepDiver1975 commented Feb 15, 2017 edited

Related Issue

fixes #23558

Missing

  • unit test execution is 3 tomes slower compared to the implementation we had before 😢
  • handle permission flags like: can set display et
  • store password reset token in the account table
  • store language setting in the account table as well
  • migration of local users - #27205
  • migration of ldap users - #27205
  • migration of saml users - #27205
  • migration or ..... users - #27205
  • ldap integration - #27205

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
@DeepDiver1975 DeepDiver1975 added this to the 10.0 milestone Feb 15, 2017
@mention-bot

@DeepDiver1975, thanks for your PR! By analyzing the history of the files in this pull request, we identified @butonic, @VicDeo and @PVince81 to be potential reviewers.

+ public function changeSchema(Schema $schema, array $options) {
+ $prefix = $options['tablePrefix'];
+ $table = $schema->createTable("{$prefix}accounts");
+ $table->addColumn('id', 'integer', [
@PVince81
PVince81 Feb 15, 2017 Collaborator

let's go big ? bigint ?

+ 'notnull' => false,
+ 'length' => 255,
+ ]);
+ $table->addColumn('quota', 'string', [
@PVince81
PVince81 Feb 15, 2017 Collaborator

Now seeing this, I remember that we already have a quota value in the oc_preferences table. But not as easy to retrieve as a single select on this table here.

Might need to think of migrating it.

+ 'notnull' => false,
+ 'length' => 32,
+ ]);
+ $table->addColumn('backend', 'string', [
@PVince81
PVince81 Feb 15, 2017 Collaborator

also add a backend_id which is the user id as seen by the backend ? unless we think it is always the same as the "user_id" column. That is, assuming that multiple backends can't provide the same user id.

@DeepDiver1975
DeepDiver1975 Feb 15, 2017 Member

Maybe not that clear but this is - as of now - the class name of the backend

@@ -94,7 +100,7 @@ public function getBackends() {
* @param \OCP\UserInterface $backend
*/
public function registerBackend($backend) {
- $this->backends[] = $backend;
+ $this->backends[get_class($backend)] = $backend;
@PVince81
PVince81 Feb 15, 2017 Collaborator

I think in the end this would only be called in the background job (or on-demand) as we won't need the backend any more for every operation since we can rely on the account table.

@DeepDiver1975
DeepDiver1975 Feb 15, 2017 Member

we need the backend for login, inital setup of new accounts (auto provision mode) and import scenarios

- } else {
- return new User($ownerId, null);
- }
+ return $this->userManager->get($ownerId);
@PVince81
PVince81 Feb 16, 2017 Collaborator

This will break with federated share owners which are not resolveable through the user manager. That's why we create a "virtual" user...

@DeepDiver1975
DeepDiver1975 Feb 16, 2017 Member

Hmmmm - ok - then we need a second implementation of IUser for this case.
User requires an instance of Account and AccountMapper to be passed in ...

@PVince81
PVince81 Feb 17, 2017 Collaborator

Indeed, that would align with this idea: #25815 (comment)

@PVince81
PVince81 Feb 17, 2017 Collaborator

Later on I wonder if we should also have a "federated users backend" which would deliver such users, they'd also appear in the account table under the "federation" backend for quick lookup. (but that sounds redundant from the system address book)

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

something to think about - for sure ....

+ 'notnull' => false,
+ 'length' => 32,
+ ]);
+ $table->addColumn('backend', Type::STRING, [
@mrow4a
mrow4a Feb 16, 2017 Contributor

@DeepDiver1975 Why it is a String? Shouldnt it be INT representing foreign key? Why do we allow NULL here?

@mrow4a
mrow4a Feb 16, 2017 Contributor

Is it connected to 0: initial?

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

Because it's the php class name

@mrow4a
mrow4a Feb 17, 2017 edited Contributor

How do you ensure, that if the backend is deleted, all related records are also deleted. Will we do single transaction doing that to ensure atomicity and consistency? Is backend primary key somewhere? This smells like possible inconsistency later, if some developer is not careful in the future.

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

If a backend is not enabled or has been deleted the user can no longer login. Deleting a user is an explicit operation and nothing we should do 'just because the backend is no longer there'

@mrow4a
mrow4a Feb 17, 2017 edited Contributor

I wound ensure this not null, and add another state 4: backend-deleted or mark user as deleted, in a single transaction - also not sure about this. Actualy this sound quite dangerous for me, that someone can enable and disable backends and we still have a hanging relation pointing to nowhere. Not sure how to resolve it not to brake consistency

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

yes - nut null makes sense here - thx

@mrow4a
mrow4a Feb 17, 2017 Contributor

Ok, because I assume we dont have a new table holding backends, we just use classnames to do that? Makes sense.

+ 'length' => 64,
+ ]);
+ $table->addColumn('home', Type::STRING, [
+ 'notnull' => false,
@mrow4a
mrow4a Feb 16, 2017 Contributor

Why do we allow null here?

@mrow4a
mrow4a Feb 16, 2017 Contributor

Is it connected to 0: initial?

+ $table->addColumn('state', Type::SMALLINT, [
+ 'notnull' => true,
+ 'default' => 0,
+ 'comment' => '0: initial, 1: enabled, 2: disabled, 3: deleted'
@mrow4a
mrow4a Feb 16, 2017 Contributor

0: initial, I start loving it. Is it connected with last_login, backend and home?

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

I'm not yet fully sure if we need 0:initial ...

lib/private/User/AccountMapper.php
+ $qb->select('*')
+ ->from($this->getTableName())
+ ->where($qb->expr()->iLike($fieldName, $qb->createNamedParameter('%' . $this->db->escapeLikeParameter($pattern) . '%')))
+ ->orderBy($fieldName, $orderBy ? 'ASC' : 'DESC');
@mrow4a
mrow4a Feb 16, 2017 edited Contributor

orderBy is quite expensive operation, do we need to do that by default? I think it makes sense only for displayname, isnt it?

@mrow4a
mrow4a Feb 16, 2017 edited Contributor

Why dont we use LIMIT on DB level? What findEntities is exactly doing? Please be aware of that it will go over all never-logged-in accounts in LDAP also, doing sequential read on the disk for pages for all of them!

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

Limit is on db level. And if we want pagination we need to sort on db level.
Orderby is not expensive is there is an index.
(Need to add it 😆)

@mrow4a
mrow4a Feb 17, 2017 edited Contributor

Internaly in DB, in order to do limit, you have to finish all others. Adding OrderBy basically tells the plan generator that you need a whole picture of the table, thus whole scan (it is enforced by sort requiring full view on the input - blocking operator). LIMIT just tells the sort to stop sorting. Of course, unless you have index, found good reference https://www.percona.com/blog/2006/09/01/mysql-order-by-limit-performance-optimization/. Just wanted to ensure that we wont do anything crazy here :>

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

The users are expected to be sorted - this has to be done in the db - no other option available

lib/private/User/AccountMapper.php
+ ->where($qb->expr()->iLike($fieldName, $qb->createNamedParameter('%' . $this->db->escapeLikeParameter($pattern) . '%')))
+ ->orderBy($fieldName, $orderBy ? 'ASC' : 'DESC');
+
+ return $this->findEntities($qb->getSQL(), $qb->getParameters(), $limit, $offset);
@mrow4a
mrow4a Feb 16, 2017 edited Contributor

We are passing here $limit. Why do we first scan whole the table and then limit that on PHP level? What findEntities is exactly doing, is it belonging to the query above or we extract logic from DB to PHP?

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

It is on db level. See the code inside find entries

+ $prefix = $options['tablePrefix'];
+ $table = $schema->createTable("{$prefix}accounts");
+ $table->addColumn('id', Type::BIGINT, [
+ 'autoincrement' => true,
@mrow4a
mrow4a Feb 17, 2017 Contributor

Is it a primary key or foreign key somewhere? I might have a question here about user uniqueness, where backend tries to add user which already exists, and how do we distinguish between them

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

auto increment always forces to be the primary key

@mrow4a
mrow4a Feb 17, 2017 Contributor

Yes, this is true

+ 'notnull' => false,
+ 'length' => 255,
+ ]);
+ $table->addColumn('user_id', Type::STRING, [
@mrow4a
mrow4a Feb 17, 2017 edited Contributor

Is it a primary key or foreign key somewhere? I might have a question here about user uniqueness, where backend tries to add user which already exists, and how do we distinguish between them. I dont also understand why we have two IDs. DO we have redundant tables? Do you have any drawing showing relations between all involved tables?

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

see the index definition - this is unique

@mrow4a
mrow4a Feb 17, 2017 Contributor

Do you have some db relations drawing of Migration feature so we can easier understand?

@mrow4a
mrow4a Feb 17, 2017 edited Contributor

I think I see the whole picture of relations of this table with others, but it is easier to look at it

@DeepDiver1975
DeepDiver1975 Feb 17, 2017 Member

Do you have some db relations drawing of Migration feature so we can easier understand?

I don't get this question. The migration feature is unrelated any db relations.

@mrow4a

I think I need to fully understand the relationships and redundancy between tables

@DeepDiver1975
Member

I think I need to fully understand the relationships and redundancy between tables

I guess so 😉

@mrow4a
Contributor
mrow4a commented Feb 21, 2017 edited

@DeepDiver1975 I did not go in details in the code yet, but seems the priority in implementation of this table will be to put as much load on table scan of oc_account table, so most of queries will just ask for records of this table, and very very seldomely do the join with e.g. ldap table (this is expensive, if both tables are big, and it will be like that). (Does it replace ldap table, or is just normalisation ?)

@PVince81
Collaborator

very very seldomely do the join with e.g. ldap table

@mrow4a no, if we had that it would be nice.
No, the old code does a PHP-level "join" by connecting directly to LDAP... or in some cases memcache. But partial searches are not possible with memcache so they always hit LDAP.

With this table partial searches can be done only in a single location: the oc_account table.

@PVince81
Collaborator

Basically we want to avoid to hit LDAP or any other user management systems at all when not needed and rely on local information.

@DeepDiver1975 DeepDiver1975 changed the title from [WIP] Central user account table arriving ..... to Central user account table arriving ..... Feb 21, 2017
@DeepDiver1975 DeepDiver1975 referenced this pull request Feb 21, 2017
Open

Syncing user backend to the account table .... #27205

3 of 15 tasks complete
DeepDiver1975 added some commits Feb 15, 2017
@DeepDiver1975 DeepDiver1975 Central user account table arriving ..... 3414c20
@DeepDiver1975 DeepDiver1975 Fixing some php errors/warnings 9fe068b
@DeepDiver1975 DeepDiver1975 During installation no UserSession is available and no database conne…
…ction - behave accordingly
224bb07
@DeepDiver1975 DeepDiver1975 Kill dummy user backend 987681f
@DeepDiver1975 DeepDiver1975 Fix unit tests ... 6dbe5a5
@DeepDiver1975 DeepDiver1975 Fix case insensitive lookup on user id dfc4c8d
@DeepDiver1975 DeepDiver1975 delete use on backend
baf7813
@DeepDiver1975 DeepDiver1975 Fix Oracle ....
78a4f06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment