Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenLDAP : TLS not working out of the box #1939

Open
TitanBalls opened this issue May 1, 2024 · 9 comments
Open

OpenLDAP : TLS not working out of the box #1939

TitanBalls opened this issue May 1, 2024 · 9 comments

Comments

@TitanBalls
Copy link

Hello,

it seems that the OpenLDAP turnkey TLS doesn't work out of the box.

[Setup]
Proxmox : 8.0.3
Template : Debian 12 - Turnkey OpenLDAP 18.0

[Issue]
After the initial setup, when trying to make an LDAP + StartTLS connection, it fails with the following error :

ldap_url_parse_ext(ldap://localhost)
ldap_create
ldap_url_parse_ext(ldap://localhost:389/??base)
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP localhost:389
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 127.0.0.1:389
ldap_pvt_connect: fd: 3 tm: -1 async: 0
attempting to connect: 
connect success
ldap_open_defconn: successful
ldap_send_server_request
ber_scanf fmt ({it) ber:
ber_scanf fmt ({) ber:
ber_flush2: 31 bytes to sd 3
ldap_result ld 0x564f7ce64b70 msgid 1
wait4msg ld 0x564f7ce64b70 msgid 1 (infinite timeout)
wait4msg continue ld 0x564f7ce64b70 msgid 1 all 1
** ld 0x564f7ce64b70 Connections:
* host: localhost  port: 389  (default)
* from: IP=127.0.0.1:37482
  refcnt: 2  status: Connected
  last used: Wed May  1 01:59:09 2024


** ld 0x564f7ce64b70 Outstanding Requests:
 * msgid 1,  origid 1, status InProgress
   outstanding referrals 0, parent count 0
  ld 0x564f7ce64b70 request count 1 (abandoned 0)
** ld 0x564f7ce64b70 Response Queue:
   Empty
  ld 0x564f7ce64b70 response count 0
ldap_chkResponseList ld 0x564f7ce64b70 msgid 1 all 1
ldap_chkResponseList returns ld 0x564f7ce64b70 NULL
ldap_int_select
read1msg: ld 0x564f7ce64b70 msgid 1 all 1
ber_get_next
ber_get_next: tag 0x30 len 12 contents:
ldap_find_request_by_msgid: msgid 1, lr 0x564f7ce680e0 lr->lr_refcnt = 1
read1msg: ld 0x564f7ce64b70 msgid 1 message type extended-result
ber_scanf fmt ({eAA) ber:
read1msg: ld 0x564f7ce64b70 0 new referrals
read1msg:  mark request completed, ld 0x564f7ce64b70 msgid 1
request done: ld 0x564f7ce64b70 msgid 1
res_errno: 0, res_error: <>, res_matched: <>
ldap_return_request: lrx 0x564f7ce680e0, lr 0x564f7ce680e0
ldap_return_request: lrx->lr_msgid 1, lrx->lr_refcnt is now 0, lr is still present
ldap_free_request (origid 1, msgid 1)
ldap_free_request_int: lr 0x564f7ce680e0 msgid 1 removed
ldap_do_free_request: asked to free lr 0x564f7ce680e0 msgid 1 refcnt 0
ldap_parse_extended_result
ber_scanf fmt ({eAA) ber:
ldap_parse_result
ber_scanf fmt ({iAA) ber:
ber_scanf fmt (}) ber:
ldap_msgfree
TLS: peer cert untrusted or revoked (0x42)
TLS: can't connect: (unknown error code).
ldap_err2string
ldap_start_tls: Connect error (-11)
	additional info: (unknown error code)
ldap_free_connection 1 1
ldap_send_unbind
ber_flush2: 7 bytes to sd 3
ldap_free_connection: actually freed

The peer cert is untrusted or revoked.

[Additional info]

As per the issue #1557 , the recommended way to use TLS is to use StartTLS. To do so, I use the following command :

ldapsearch -H ldap://localhost -x -D "cn=openldap,dc=openldap,dc=local" -W -d 1 -ZZ

where cn=openldap is my bind accout (security object) to make searches in the database.

As per the LDAP documentation, the -ZZ tag is used to initiate a unencrypted connection, and then upgrade the connection with StartTLS.

When looking into the Webmin console, in the OpenLDAP config points to the auto generated certificate for openldap and the CA.

image

@JedMeister
Copy link
Member

Hi, thanks for reporting.

As you note, it certainly does appear to be an issue with the certs:

I'm still no expert with OpenLDAP or LDAP in general and I forget the specific details. I've done some quick searching online to try to refresh my memory.

FWIW, I'm looking at the turnkey-regen-ldap-certs script (or /usr/local/bin/turnkey-regen-ldap-certs locally) against some info online - e.g. OpenLDAP docs, Ubuntu OpenLDAP wiki page, serverfault and stackoverflow) it on face value it should "just work"!?

Although now re-reading that stuff, I'm wondering if the local client has separate config to the server and it's not preconfigured? While that might make sense, it doesn't seem that we're touching the config of either the OpenLDAP client or the server at build time. So I can only assume that we're using the default cert paths used by both of them.

Also it would surprise me somewhat as I'm sure I tested it in a local VM, before pushing it the to OpenLDAP appliance build code overlay. OTOH it wouldn't be the first time that has happened (if I forget to test the script on a clean install, there can be some config I did along the way that meant the script worked on my test server, but might not on a clean install)...

So to see if there is perhaps something wrong with my script, please double check for errors that occur in the firstboot scripts:

journalctl -t inithooks

And it might also be worth manually rerunning the script. I.e.:

turnkey-regen-ldap-certs FQDN

Where FQDN is the desired domain. Note that for local testing, if you use a full FQDN the local server must be able to use that to connect to itself. As such, it should work with the hostname. By default that will be openldap, if you've set a custom one, then use that instead.

I'll try to test myself ASAP but I'm a bit snowed under ATM, so it'd be great if you could report back.

@JedMeister JedMeister added this to the 18.1 milestone May 1, 2024
@TitanBalls
Copy link
Author

TitanBalls commented May 3, 2024

Thanks for the reply.

I did the troubleshooting steps you recommended.

  1. jounalctl -t inithooks : Everything seems fine, the script runs without error.
May 03 00:53:01 openldap inithooks[619]: [01ipconfig] running
May 03 00:53:01 openldap inithooks[621]: [01ipconfig] successfully completed
May 03 00:53:01 openldap inithooks[623]: [05autogrow-fs] running
May 03 00:53:01 openldap inithooks[625]: [05autogrow-fs] successfully completed
May 03 00:53:01 openldap inithooks[627]: [09hostname] running
May 03 00:53:01 openldap inithooks[640]: [09hostname] successfully completed
May 03 00:53:01 openldap inithooks[642]: [10randomize-cronapt] running
May 03 00:53:01 openldap inithooks[645]: [10randomize-cronapt] successfully completed
May 03 00:53:01 openldap inithooks[647]: [10randomize-crontab] running
May 03 00:53:01 openldap inithooks[654]: [10randomize-crontab] successfully completed
May 03 00:53:01 openldap inithooks[656]: [10regen-sshkeys] running
May 03 00:53:01 openldap inithooks[619]: [01ipconfig] running
May 03 00:53:01 openldap inithooks[621]: [01ipconfig] successfully completed
May 03 00:53:01 openldap inithooks[623]: [05autogrow-fs] running
May 03 00:53:01 openldap inithooks[625]: [05autogrow-fs] successfully completed
May 03 00:53:01 openldap inithooks[627]: [09hostname] running
May 03 00:53:01 openldap inithooks[640]: [09hostname] successfully completed
May 03 00:53:01 openldap inithooks[642]: [10randomize-cronapt] running
May 03 00:53:01 openldap inithooks[645]: [10randomize-cronapt] successfully completed
May 03 00:53:01 openldap inithooks[647]: [10randomize-crontab] running
May 03 00:53:01 openldap inithooks[654]: [10randomize-crontab] successfully completed
May 03 00:53:01 openldap inithooks[656]: [10regen-sshkeys] running
May 03 00:53:01 openldap inithooks[671]: [10regen-sshkeys] successfully completed
May 03 00:53:01 openldap inithooks[673]: [15regen-sslcert] running
May 03 00:53:12 openldap inithooks[2452]: [15regen-sslcert] successfully completed
May 03 00:53:12 openldap inithooks[2454]: [20regen-openldap-secrets] running
May 03 00:53:18 openldap inithooks[2488]: [20regen-openldap-secrets] successfully completed
May 03 00:53:18 openldap inithooks[2492]: [20regen-phpldapadmin-secrets] running
May 03 00:53:18 openldap inithooks[2497]: [20regen-phpldapadmin-secrets] successfully completed
May 03 00:53:18 openldap inithooks[2499]: [29preseed] running
May 03 00:53:18 openldap inithooks[2506]: [29preseed] successfully completed
May 03 00:53:18 openldap inithooks[2508]: [29sudoadmin] running
May 03 00:53:18 openldap inithooks[2564]: [29sudoadmin] successfully completed
May 03 00:53:18 openldap inithooks[2566]: [29tagid] running
May 03 00:53:18 openldap inithooks[2588]: [29tagid] successfully completed
May 03 00:53:18 openldap inithooks[2590]: [30rootpass] skipping
May 03 00:53:18 openldap inithooks[2592]: [30turnkey-init-fence] running
May 03 00:53:19 openldap inithooks[2687]: [30turnkey-init-fence] successfully completed
May 03 00:53:19 openldap inithooks[2689]: [35postfix-unprivileged] running
May 03 00:53:19 openldap inithooks[3016]: [35postfix-unprivileged] successfully completed
May 03 00:53:19 openldap inithooks[3018]: [40openldap] running
May 03 00:53:36 openldap inithooks[3248]: [40openldap] successfully completed
May 03 00:53:36 openldap inithooks[3252]: [80hub-services] running
May 03 00:53:36 openldap inithooks[3254]: [80hub-services] successfully completed
May 03 00:53:36 openldap inithooks[3256]: [85secalerts] running
May 03 00:53:36 openldap inithooks[3258]: [85secalerts] successfully completed
May 03 00:53:36 openldap inithooks[3260]: [92etckeeper] running
May 03 00:53:37 openldap inithooks[3493]: [92etckeeper] successfully completed
May 03 00:53:37 openldap inithooks[3495]: [95secupdates] running
May 03 00:53:37 openldap inithooks[3498]: [95secupdates] successfully completed
May 03 00:53:37 openldap inithooks[3500]: [97turnkey-init-fence-disable] running
May 03 00:53:37 openldap inithooks[3502]: [97turnkey-init-fence-disable] successfully completed
May 03 00:53:37 openldap inithooks[3504]: [98finalize] running
May 03 00:53:37 openldap inithooks[3507]: [98finalize] successfully completed
May 03 00:53:37 openldap inithooks[3509]: [99reboot] skipping
May 03 00:53:37 openldap inithooks[3514]: [01empty] skipping
May 03 00:53:37 openldap inithooks[3515]: Inithook run completed, exiting.
May 03 00:54:37 openldap inithooks[3553]: [01empty] skipping
May 03 00:54:37 openldap inithooks[3554]: Inithook run completed, exiting.
  1. turnkey regen certs : the command works, but I still can't do a StartTLS connection. I tried multiple FQDNs just to make sure there wasn't anything weird with the fact that I use openldap.local. This didn't fix the issue, i still get the untrusted certificate error.

As a sanity check, I tried to establish a unencrypted connection with the default admin user, to make sure there wasn't any rights issue. When I change the certs and give it a new FQDN, it works correctly as long as I make sure that the FQDN is reacheable. So I modify my /etc/hosts file according ( 127.0.1.1).

One thing I noticed, is that in the webmin OpenLDAP server configuration, I can generate new SSL certificate. However, If I use that option, slapd fails to restart until I regenerate the turnkey certs with the CLI command. I am new to turnkey, so I am not sure if there is any link.

image

When I regenerate the SSL certs and try to apply the configuration, I get this issue :

Failed to apply configuration : failed :

Restarting slapd (via systemctl): slapd.serviceJob for slapd.service failed because the control process exited with error code.
See "systemctl status slapd.service" and "journalctl -xeu slapd.service" for details.
 failed!

And when I look up the systemctl status, I get this info :

 slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
     Loaded: loaded (/etc/init.d/slapd; generated)
    Drop-In: /usr/lib/systemd/system/slapd.service.d
             `-slapd-remain-after-exit.conf
     Active: failed (Result: exit-code) since Fri 2024-05-03 01:25:03 UTC; 1min 4s ago
   Duration: 9min 50.656s
       Docs: man:systemd-sysv-generator(8)
    Process: 2893 ExecStart=/etc/init.d/slapd start (code=exited, status=1/FAILURE)
        CPU: 12ms

May 03 01:25:03 openldap systemd[1]: Starting slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)...
May 03 01:25:03 openldap slapd[2898]: @(#) $OpenLDAP: slapd 2.5.13+dfsg-5 (Feb  8 2023 01:56:12) $
                                              Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
May 03 01:25:03 openldap slapd[2898]: main: TLS init def ctx failed: -1
May 03 01:25:03 openldap slapd[2898]: slapd stopped.
May 03 01:25:03 openldap slapd[2898]: connections_destroy: nothing to destroy.
May 03 01:25:03 openldap slapd[2893]: Starting OpenLDAP: slapd failed!
May 03 01:25:03 openldap systemd[1]: slapd.service: Control process exited, code=exited, status=1/FAILURE
May 03 01:25:03 openldap systemd[1]: slapd.service: Failed with result 'exit-code'.
May 03 01:25:03 openldap systemd[1]: Failed to start slapd.service - LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol).

My understanding is that the Generate SSL button is supposed to do the same thing as turnkey-regen-ldap-certs, but I might be wrong. I can restart slapd by using the turnkey-regen-ldap-certs first, and then in webmin, start the server.

I will try to generate and sign my certificate manually see if that fixes the issue. I hope that the delta will give us some insight on what is happening with the turnkey image.

Let me know if I can test other things to help out.

@TitanBalls
Copy link
Author

Is it possible that the issue is that the LDAP client is not configured correctly?

When I look into webmin, in the LDAP client, I see this :

image

When I try to install the module, it says that the module is already installed

Reading package lists...
Building dependency tree...
Reading state information...
libnss-ldapd is already the newest version (0.9.12-4).
libpam-ldapd is already the newest version (0.9.12-4).
0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.

But when I look for a ldap.conf file, I get nothing.

 find / -type f -name "*ldap*.conf"
/etc/lighttpd/conf-available/50-phpldapadmin.conf
/etc/ldapscripts/ldapscripts.conf
find: '/dev/.lxc/sys/kernel': Permission denied
find: '/dev/.lxc/sys/power': Permission denied
find: '/dev/.lxc/sys/class': Permission denied
find: '/dev/.lxc/sys/devices': Permission denied
find: '/dev/.lxc/sys/dev': Permission denied
find: '/dev/.lxc/sys/hypervisor': Permission denied
find: '/dev/.lxc/sys/fs': Permission denied
find: '/dev/.lxc/sys/bus': Permission denied
find: '/dev/.lxc/sys/firmware': Permission denied
find: '/dev/.lxc/sys/block': Permission denied
find: '/dev/.lxc/sys/module': Permission denied
find: '/dev/.lxc/proc/fs': Permission denied
find: '/dev/.lxc/proc/bus': Permission denied
find: '/dev/.lxc/proc/irq': Permission denied
find: '/dev/.lxc/proc/spl': Permission denied
find: '/dev/.lxc/proc/sys': Permission denied
find: '/dev/.lxc/proc/tty': Permission denied
find: '/dev/.lxc/proc/acpi': Permission denied
find: '/dev/.lxc/proc/scsi': Permission denied
find: '/dev/.lxc/proc/asound': Permission denied
find: '/dev/.lxc/proc/driver': Permission denied
find: '/dev/.lxc/proc/sysvipc': Permission denied
find: '/dev/.lxc/proc/pressure': Permission denied
find: '/dev/.lxc/proc/dynamic_debug': Permission denied
find: '/dev/.lxc/proc/1': Permission denied
find: '/dev/.lxc/proc/50': Permission denied
find: '/dev/.lxc/proc/94': Permission denied
find: '/dev/.lxc/proc/97': Permission denied
find: '/dev/.lxc/proc/101': Permission denied
find: '/dev/.lxc/proc/161': Permission denied
find: '/dev/.lxc/proc/232': Permission denied
find: '/dev/.lxc/proc/234': Permission denied
find: '/dev/.lxc/proc/239': Permission denied
find: '/dev/.lxc/proc/240': Permission denied
find: '/dev/.lxc/proc/241': Permission denied
find: '/dev/.lxc/proc/242': Permission denied
find: '/dev/.lxc/proc/243': Permission denied
find: '/dev/.lxc/proc/280': Permission denied
find: '/dev/.lxc/proc/281': Permission denied
find: '/dev/.lxc/proc/334': Permission denied
find: '/dev/.lxc/proc/606': Permission denied
find: '/dev/.lxc/proc/609': Permission denied
find: '/dev/.lxc/proc/612': Permission denied
find: '/dev/.lxc/proc/661': Permission denied
find: '/dev/.lxc/proc/701': Permission denied
find: '/dev/.lxc/proc/5902': Permission denied
find: '/dev/.lxc/proc/6034': Permission denied
find: '/dev/.lxc/proc/6067': Permission denied
find: '/dev/.lxc/proc/6539': Permission denied
find: '/dev/.lxc/proc/8724': Permission denied
find: '/proc/tty/driver': Permission denied
find: '/sys/kernel/debug': Permission denied
find: '/sys/fs/fuse/connections/47': Permission denied
find: '/sys/fs/fuse/connections/45': Permission denied

@JedMeister
Copy link
Member

Thanks for the additional info.

It's certainly strange that it's not finding the conf file. Perhaps it needs some more config to make the web UI work properly?

TBH, I'm a CLI guy and often forget to check via the web UI. I'll be sure to explicitly check that.

I'll have a look into it ASAP and report back (and document). If need be I'll release a new version with the required changes.

@TitanBalls
Copy link
Author

TitanBalls commented May 10, 2024 via email

@JedMeister
Copy link
Member

Hey @TitanBalls I think I know what the issue is! I haven't tested, but I'm fairly confident. Scroll to the last paragraph for the TL;DR, or read on for the context. :)

Before I go on, FYI OpenLDP itself comes direct from Debian - the package is called slapd.

The ldap.conf file should be /etc/ldap/ldap.conf - from the libldap-common package.

However, it looks like we're not explicitly installing it. See the OpenLDAP appliance specific plan (the Debian packages that are installed). Generally I would not expect that to be a problem, as in my experience ...-common packages are dependencies of other packages.

To check that, I followed the trail of dependencies. Both ldap-utils & slapd (the packages we're explicitly installing) depend on libldap-2.5-0. But instead of that depending on libldap-common as I would have expected, libldap-common is just a "recommends".

By default on Debian "recommends" (soft dependencies) are treated just like "depends" (hard dependencies) and are auto installed. But as "recommends" by nature are not required for core functionality and to try to keep our appliances lean, we configure apt to only install hard dependencies by default.

So the package that provides the ldap.conf file is not installed!

My guess is that when I developed and tested the OpenLDAP SSL/TLS script (IIRC our last major version release - based on the previous Debian version) that it was a hard dependency - so it all "just worked". But for some reason it has moved to a "recommends" in Bookworm (TKL v18.x), thus isn't being installed! Then when the updated v18.x appliance was tested for this release, my colleague (who did the update) only tested the core OpenLDAP functionality and didn't explicitly test the SSL/TLS support. I did code review and pre-release basic smoke testing, but I didn't test SSL/TLS support until you dropped in - and the rest is history. So...

TL;DR try installing libldap-common and see how you go!

apt update
apt install -y libldap-common

@TitanBalls
Copy link
Author

You were right.

When I installed libldap-common, ldap.conf was installed in /etc/ldap/. In the webmin interface, I changed the path to /etc/ldap/ldap.conf and then the client was configured.

After that , I simply edited ldap.conf to point to the right certificate TLS_CACERT /etc/ldap/tls/ca_cert.pem.

When I try the ldapsearch with the option -ZZ, the authentication goes through correctly.

To verify that, I added the option -d 2 and got a bunch of TLS exchanges like this

tls_read: want=5, got=5
  0000:  17 03 03 01 99                                     .....             
tls_read: want=409, got=409
  0000:  9b ee 78 75 [...]

which I didn't have before.

Thank you for your help, it is appreciated.

@TitanBalls
Copy link
Author

TitanBalls commented May 10, 2024

[WORKAROUND]
As per Jeremy's response,

when starting the turnkey openldap, go through the initial configuration.

Then, run the following commands :

apt update
apt install -y libldap-common

After that, modify the ldap.conf file to point to the right path /etc/ldap/tls/ca_cert.pem.

Then, either through the webmin, or through the CLI, make sure that the LDAP client is pointing to the right ldap.conf path. By default, the path is /etc/ldap.conf but should be /etc/ldap/ldap.conf .

Reboot the LDAP service, and the ldap query goes through with the -ZZ option.

@JedMeister
Copy link
Member

Awesome! Thanks again for your initial report and bonus points for confirming! 😁

I am going to reopen this issue again though.

While this thread (particularly your last post) will assist users to implement it now, IMO our appliance should include support for SSL/TLS OOTB. Especially as this is a regression really. It has supported SSL/TLS in the past.

I'll reclose it once the OpenLDAP appliance build code is "fixed". Although I probably won't prioritize rebuilding it. We still have a few appliances left to update to v18.0. Once they're all updated I'll do a batch of updates - at least those that include bugfixes.

@JedMeister JedMeister reopened this May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants