Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add indexes for fields listed by slapd in the logs #729

Merged
merged 1 commit into from May 31, 2019

Conversation

Projects
None yet
6 participants
@MCMic
Copy link
Contributor

commented May 23, 2019

I get lots of suggestion of indexes by slapd in my logs:

<= mdb_equality_candidates: (cn) not indexed
<= mdb_equality_candidates: (gidNumber) not indexed
<= mdb_equality_candidates: (mail) not indexed
<= mdb_equality_candidates: (member) not indexed
<= mdb_equality_candidates: (memberUid) not indexed
<= mdb_equality_candidates: (sudoUser) not indexed
<= mdb_equality_candidates: (uidNumber) not indexed
<= mdb_equality_candidates: (uniqueMember) not indexed
<= mdb_equality_candidates: (virtualdomain) not indexed
<= mdb_substring_candidates: (sudoUser) not indexed

Since Yunohost makes it hard to edit LDAP server configuration (see YunoHost/issues#1350), the default configuration should contain indexes for the fields used by Yunohost a lot.

The problem

...

Solution

...

PR Status

...

How to test

...

Validation

  • Principle agreement 0/2 :
  • Quick review 0/1 :
  • Simple test 0/1 :
  • Deep review 0/1 :
Add indexes for fields listed by slapd in the logs
I get lots of suggestion of indexes by slapd in my logs:

 <= mdb_equality_candidates: (cn) not indexed
 <= mdb_equality_candidates: (gidNumber) not indexed
 <= mdb_equality_candidates: (mail) not indexed
 <= mdb_equality_candidates: (member) not indexed
 <= mdb_equality_candidates: (memberUid) not indexed
 <= mdb_equality_candidates: (sudoUser) not indexed
 <= mdb_equality_candidates: (uidNumber) not indexed
 <= mdb_equality_candidates: (uniqueMember) not indexed
 <= mdb_equality_candidates: (virtualdomain) not indexed
 <= mdb_substring_candidates: (sudoUser) not indexed

Since Yunohost makes it hard to edit LDAP server configuration (see YunoHost/issues#1350), the default configuration should contain indexes for the fields used by Yunohost a lot.
@decentral1se
Copy link
Contributor

left a comment

For the ones who need it:

https://ldapwiki.com/wiki/LDAP%20Indexes

LDAP is black magic to me still ...

@alexAubin
Copy link
Member

left a comment

Looks legit to me 👍
Proposing to merge soonish

@alexAubin alexAubin added this to the 3.6.x milestone May 23, 2019

@MCMic

This comment has been minimized.

Copy link
Contributor Author

commented May 23, 2019

Could you check your logs see if it suggest the same fields as mine? I may have missed one or included something specific to my usage, even if the fields listed seem pretty standard and consistent with unix/sudo usage.

@Psycojoker

This comment has been minimized.

Copy link
Member

commented May 25, 2019

Will ldap auto add those index on config reloading or should we write some kind of migration like with sql database? I really have bad knowledge of ldap :/

@Josue-T

This comment has been minimized.

Copy link
Contributor

commented May 25, 2019

Will ldap auto add those index on config reloading or should we write some kind of migration like with sql database? I really have bad knowledge of ldap :/

This should be done by the regen-conf

@Psycojoker

This comment has been minimized.

Copy link
Member

commented May 26, 2019

Will ldap auto add those index on config reloading or should we write some kind of migration like with sql database? I really have bad knowledge of ldap :/

This should be done by the regen-conf

How?

@Josue-T

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

Will ldap auto add those index on config reloading or should we write some kind of migration like with sql database? I really have bad knowledge of ldap :/

This should be done by the regen-conf

How?

When you run the regen-conf the config database is rebuilt. The code is here:
https://github.com/YunoHost/yunohost/blob/stretch-unstable/data/hooks/conf_regen/06-slapd#L22-L26

Note that the slaptest rebuild the config as described in the man:

If both -f and -F are specified, the config file will be read and converted to config directory format and written to the specified directory.

Note that the "config directory" is the "config" database.

@alexAubin

This comment has been minimized.

Copy link
Member

commented May 28, 2019

Could you check your logs see if it suggest the same fields as mine? I may have missed one or included something specific to my usage, even if the fields listed seem pretty standard and consistent with unix/sudo usage.

@MCMic I have the same messages as you on my server

@alexAubin alexAubin merged commit 0f2465a into YunoHost:stretch-unstable May 31, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Soooo I should have tested this instead of yolomergin :s But this seems to be triggering important issues when deployed on a "real" server :

  • Login as admin results in a prompt like this (c.f. the I have no name! instead of admin) :
I have no name!@machinename:$
  • Trying to log through SSH with yunohost user results in shell being closed automatically as if SSH login was disabled (password accepted, but probably shell set to /bin/false somehow...)

The slapd logs especially show this message which seems related (it appears each time an SSH login is made with a yunohost user) :

slap_global_control: unrecognized control: 1.3.6.1.4.1.42.2.27.8.5.1 

It's a bit tricky to diagnose what exactly is going on because some nscd caching is messing around (I think I got rid of it by running nscd -i hosts and nscd -i password before each tests) but it seems to be coming from this line (not sure why exactly) :

index gidNumber,uidNumber           eq
@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

I pinpointed it to index gidNumber,uidNumber

According to Josue, this is ~expected because YunoHost ldap scheme (?) doesn't really well support / handle gid and uid ...

I pushed a hotfix here (commenting the line) : 769ba57

@MCMic

This comment has been minimized.

Copy link
Contributor Author

commented Jun 4, 2019

* Login as `admin` results in a prompt like this (c.f. the `I have no name!` instead of `admin`) :
I have no name!@machinename:$

Sounds like broken/stopped LDAP no? Or maybe not correctly tied to pam/ssh?

The slapd logs especially show this message which seems related (it appears each time an SSH login is made with a yunohost user) :

slap_global_control: unrecognized control: 1.3.6.1.4.1.42.2.27.8.5.1 

This is ppolicy control. As far as I know ppolicy overlay is not used on yunohost so it is expected for slapd to not know it, but I do not know who tries to ask for it.
Is there any other configuration change which may have triggered this?

It's a bit tricky to diagnose what exactly is going on because some nscd caching is messing around (I think I got rid of it by running nscd -i hosts and nscd -i password before each tests) but it seems to be coming from this line (not sure why exactly) :

index gidNumber,uidNumber           eq

Removing that does fix the problem? I do not understand how adding an index can change behavior, unless there is a syntax error or something.

@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

Sounds like broken/stopped LDAP no? Or maybe not correctly tied to pam/ssh?

LDAP was apparently running "fine" (systemctl status showing it in green)

Is there any other configuration change which may have triggered this?

The other recent change was : #721 but AFAIK this didn't trigger any such issue ...

Removing that does fix the problem? I do not understand how adding an index can change behavior, unless there is a syntax error or something.

Yes, it does fix the issue ... and I'm also puzzled to understand why it was causing an issue in the first place :/

@Josue-T

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

This is ppolicy control. As far as I know ppolicy overlay is not used on yunohost so it is expected for slapd to not know it, but I do not know who tries to ask for it.

Maybe sudo ?

@MCMic

This comment has been minimized.

Copy link
Contributor Author

commented Jun 5, 2019

Yes, it does fix the issue ... and I'm also puzzled to understand why it was causing an issue in the first place :/

Are you sure it wasn’t just the slapd restart or the regen-conf which fixed the issue, more than the index change?

@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

Yes, I'm pretty sure, I commented the line, restarted (+cleared the cache with nscd -i hosts and nscd -i passwd), tested it, decommented the line, restarted again, tested it, and re-commented again and so on ...

@Josue-T

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2019

As I understand the issue is that in Yunohost the group is for now not cleanly managed. As you can see the admin user as no main group (and for all other user it's the same case). And it's really bad because in unix standard all user have his own main group. So the link between the issue about the authentication and the issue about group of user could be not evident. But I would just say that probably when something is not cleanly made, you might have some terrible mysterious issue !!

@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 5, 2019

More info on this :

I (and also @taziden ) noticed that the mail stack got somewhat broken in 3.6.1. Namely getting messages from MAILER-DAEMON like :

This is the mail system at host dismorphia.info.

I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.
<alex@domain.tld> (expanded from <root@domain.tld>): mail for domain.tld loops back to myself

Playing a bit more with the slapd.conf, I was able to notice that those lines seems to trigger the issue :

index  cn,mail                       eq
index  virtualdomain                 eq

Once again, that's pretty weird that just indexing the values trigger this :/ ... Apparently this kinda goes beyond the "groups are not handled properly on Yunohost" that @Josue-T describes.

@MCMic

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2019

Is it not something to do with reindexing missing ?

When using slapd.conf you are supposed to use slapindex to reindex (while slapd is stopped if I’m not mistaken).
But when using cn=config indexing is supposed to be automatic.
But since you use a mix of both I’m not sure what applies.
[EDIT] I’m also a bit troubled by the fact that the regen-conf does not stop slapd before messing with configuration database.

@alexAubin Could you try maybe splitting each attribute on its own index line? I’m not sure if it’s equivalent or not.

@taziden

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2019

I can attest that running slapindex indeed fixed the issue for me.

@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

So uh, should we run slapindex during the regenconf ? Should we also run e.g. each time we create a user and domain ? Or is it expected to be ran automatically ?

@MCMic

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2019

It’s supposed to be run when you add/remove indexes only as far as I know.
I would change the regen-conf so that it:

  1. Stops slapd
  2. Do the slapd.conf conversion
  3. Run slapindex
  4. Start slapd
@taziden

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2019

3. Run slapindex

According to the docs, it's highly recomended to run it as openldap user and not as root.

@Josue-T

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2019

We proabably just should one line here : https://github.com/YunoHost/yunohost/blob/stretch-unstable/data/hooks/conf_regen/06-slapd#L103

Maybe with sudo -u openldap ...

@alexAubin

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

Ran some tests and this indeed seems to fix it 👍

Pushed the fix in d8b086a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.