internal syncrepl searches are flagged as unindexed

[389-ds-bot]

Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/47816

• Created at 2014-06-17 20:46:44 by rmeggins (@richm)
• Closed as Fixed
• Assigned to lkrispen (@elkris)

---

For example, running syncrepl with named, gives these messages in the errors log:

Internal search - Unindexed search: source (cn=server,cn=plugins,cn=config) search base="cn=changelog" filter="(&(changenumber>=42)(targetuniqueid=xxxxxxx))" etime=0 nentries=1 notes=U

Not sure which one is unindexed - could be changenumber is not indexed for range/ordering? could be targetuniqueid is not indexed for equality?

[389-ds-bot]

Comment from mkosek at 2014-08-12 13:38:18

I would welcome this fix backported also to F20 Directory Server as FreeIPA DS log was swarmed with these error messages.

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-12 13:45:42

the problem is the missing equality index for targetuniqueid. a fix could add it to the default indexes for the changelog, but for existing deployments a reindexing would be required.
Couldn't IPA if it enables the sync repl plugin also add the index and reindexthe changelog ?

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-20 20:29:44

attachment
0001-Ticket-47816-internal-syncrepl-searches-are-flagged-.patch

[389-ds-bot]

Comment from rmeggins (@richm) at 2014-08-20 20:43:13

Do we also need to add the index to existing backends?

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-20 20:54:32

Replying to [comment:6 richm]:

Do we also need to add the index to existing backends?

I see two problems with this. First: the current update scripts will just process the *ldifs in the .../updates directory, which add or modify entries in the dse.ldif whose DN is known. For adding and index to an existing backend this would require to query the dse.ldif to find the backends and then add it. I don't see how this can be done in the updateDS code.
Second: Even if we succeed it would require a reindex, and if we can trigger it it would start to reindex all backends, with the result that the attribute didn't exist.
If we do not trigger a reindex, I think the index will be used after a restart and newly created entries will be indexed and existing ones not.

[389-ds-bot]

Comment from rmeggins (@richm) at 2014-08-20 21:17:33

Replying to [comment:7 elkris]:

Replying to [comment:6 richm]:

Do we also need to add the index to existing backends?

I see two problems with this. First: the current update scripts will just process the *ldifs in the .../updates directory, which add or modify entries in the dse.ldif whose DN is known. For adding and index to an existing backend this would require to query the dse.ldif to find the backends and then add it. I don't see how this can be done in the updateDS code.

Right. You can't do it with a .ldif file. You can do it with a .pl file. See 90subtreereindex.pl and others in ldap/admin/src/scripts

Second: Even if we succeed it would require a reindex, and if we can trigger it it would start to reindex all backends, with the result that the attribute didn't exist.

We do similar things during upgrade e.g. see 90subtreereindex.pl. I don't know if this is something that we will want to do automatically since it affects relatively few users.

If we do not trigger a reindex, I think the index will be used after a restart

A yum update/rpm update will restart the server after the updates are applied.

and newly created entries will be indexed and existing ones not.

If that is going to cause problems, then either we need to add the index and do the reindex during upgrade, or we just add the default index and tell people what they need to do to add the index to all of their existing backend instances and reindex.

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-21 15:09:48

Adding an index without reindexing makes searches fail. Here is an example:

Without index:
[21/Aug/2014:10:34:24 +0200] conn=20 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:34:24 +0200] conn=20 op=1 RESULT err=0 tag=101 '''nentries=1''' etime=0 '''notes=U'''

Adding an index for homephone, without reindexing:
21/Aug/2014:10:37:18 +0200] conn=1 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:37:18 +0200] conn=1 op=1 RESULT err=0 tag=101 '''nentries=0''' etime=0

Reindexing:
[21/Aug/2014:10:38:51 +0200] conn=1 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(homePhone=999-697-8966)" attrs=ALL
[21/Aug/2014:10:38:51 +0200] conn=1 op=1 RESULT err=0 tag=101 '''nentries=1''' etime=0

So just adding an index will result in unexpected behaviour. As this attribute is used only in the retro changelog it will not be acceptable to reindex all datbases as this can take very long.

Should we add a special script to do this only for the retro changelog backend ?

[389-ds-bot]

Comment from rmeggins (@richm) at 2014-08-21 19:36:04

Replying to [comment:9 elkris]:

So just adding an index will result in unexpected behaviour. As this attribute is used only in the retro changelog it will not be acceptable to reindex all datbases as this can take very long.

Should we add a special script to do this only for the retro changelog backend ?

Yes. And we should be able to do db2index -n changelogdb so that we only reindex the retro changelog. This should only affect people who are using the retrochangelog, and it should not take a long time.

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-21 20:29:12

ok, makes sense and avoids adding stuff to release notes, where they could be missed. It could even be db2index -n changelogdb -t targetuniqueid.

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-25 17:26:03

attachment
0001-Ticket-47816-v2-internal-syncrepl-searches-are-flagg.patch

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-25 21:22:12

]$ git merge ticket47816
Updating 50c05b4..0a69c4d
Fast-forward
Makefile.am | 2 ++
Makefile.in | 2 ++
ldap/admin/src/scripts/50targetuniqueid.ldif | 7 +++++++
ldap/admin/src/scripts/82targetuniqueidindex.pl | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
ldap/ldif/template-dse.ldif.in | 7 +++++++
5 files changed, 70 insertions(+)
create mode 100644 ldap/admin/src/scripts/50targetuniqueid.ldif
create mode 100644 ldap/admin/src/scripts/82targetuniqueidindex.pl

$ git push origin master
Counting objects: 21, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (12/12), 1.97 KiB, done.
Total 12 (delta 8), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
50c05b4..0a69c4d master -> master

[389-ds-bot]

Comment from lkrispen (@elkris) at 2014-08-26 22:23:26

committed to 1.3.2
New commits:
commit abc947a

[389-ds-bot]

Comment from lkrispen (@elkris) at 2017-02-11 22:50:14

Metadata Update from @elkris:

• Issue assigned to elkris
• Issue set to the milestone: 1.3.3 - 8/14 (August)