dogtag: search past the first 100 certificates

[HonzaCholasta]

Dogtag requires a size limit to be specified when searching for
certificates. When no limit is specified in the dogtag plugin, a limit of
100 entries is assumed. As a result, an unlimited certificate search
returns data only for a maximum of 100 certificates.

Raise the "unlimited" limit to the maximum value Dogtag accepts.

https://fedorahosted.org/freeipa/ticket/6564

[tkrizek]

With this fix, more than 100 certificates are displayed and click-able from WebUI overview. However, I'm still getting an error message pop up saying

Search result has been truncated: Configured size limit exceeded

And there is also this message at the bottom of the page:

Query returned more results than the configured size limit. Displaying the first 110 results.

[frasertweedale]

This change is working for me, including having the expected behaviour for WebUI. @tomaskrizek please provide steps to reproduce your WebUI behaviour.

[tkrizek]

@frasertweedale
I ran into this issue when I created 100 users with different user certificates:

for i in {300..400}; do
ipa user-add "test$i" --first T --last T;
openssl req -new -newkey rsa:1024 -days 365 -nodes -keyout "private$i.key" -out "cert$i.csr" -subj "/CN=test$i";
ipa cert-request "cert$i.csr" --principal "test$i" ;
done

Please let me know if you are able to reproduce the issue in this way. It might be possible some unrelated issues may be the cause here.

[frasertweedale]

@tomaskrizek yes, I can reproduce with your steps.

[frasertweedale]

@tomaskrizek @HonzaCholasta it looks like the problem is:

1. subsearches are conducted in order:

   1. _cert_search (if 'certificate' in options add key to result and "seal" it)
   2. _ca_search (actually perform the search against Dogtag, via ra.find)
   3. _ldap_search (look for local entries that have given cert in their userCertificate attr.

2. if no explicit sizelimit is requested, and if there are > 100 entries with (userCertificate=*), _ldap_search will be truncated, and this result is carried across to the final result. The cert search from Dogtag is not truncated, but the search for entries to use to filter the result may have been truncated.

The simplest way to resolve this is (I think) to forcibly execute _ldap_search with sizelimit=0.

IMO _ldap_search should also be avoided or short-circuited if none of the owner-flitering options to cert-find are given. (edit to note: this will not find certs that are in IPA LDAP but not in Dogtag, which is guess is the wrong behaviour..? So I think we just have to have sizelimit=0. I am concerned about performance impact of cert-find with many principals with certs set... but that is a separate issue).

[stlaz]

@frasertweedale if _ldap_search is performed with correct filters, sizelimit=0 is not the correct solution at least for CLI which should either follow the sizelimit argument if set or the records size limit in ipa config. It is only correct for WebUI which I believe should be setting sizelimit=0 and if it's not, I'd be looking for the bug there.

I tried to briefly go through the cert plugin code but it's a bit messy so my only hope is that the correct filter is indeed used there. On the way through it, though, I found something that seems like another size limit bug: https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L1306 -> which will not set our "unlimited" if sizelimit is set to 0. Also from there, if sizelimit is not set, we should go with ipa config sizelimit rather than having the magic do its trick somewhere else, right? Then the proposed value in options.get() could go away (be set in the cert.py module instead).

[frasertweedale]

@stlaz as I see it, the _ldap_search can potentially search all objects of a particular type (user/service/host), which have (userCertificate=*). The result is then used to filter or add to the result, depending on whether the result is "key complete" or not (indicated by the variable complete).

Anyhow I leave to Honza to comment further; he probably understands the code better than me :)

[HonzaCholasta]

I have identified some issues in search limit handling in cert-find and fixed them in an additional commit. See commit message for details.

[frasertweedale]

I think the behaviour of the command is correct now, but the Web UI is now limited
to 100 certs without (as far as I can see) a way to adjust the sizelimit or search past the first 100.

[tkrizek]

The behavior of the command seems to be correct now, but I'm also not sure about the WebUI. There seems to be a limit of 20 items when displayed in WebUI (with pagination). I'm not sure if it's possible to configure that.

@pvomacka Were there any recent changes in the WebUI pagination? Is it possible to configure how many items should be displayed?

[tkrizek]

We examined the WebUI side and it behaves as expected - the size limit is respected when viewing certificates.

[HonzaCholasta]

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/d84edc43e55c2f7c30614a4a5268aeb58e33a087
https://fedorahosted.org/freeipa/changeset/85834abad655c6aed54c0253bc194ece81d78774