Ipaserver installation failed

[talleno]

Hi,
I install freeipa using ansible collection(v1.12.0) on rocky 8 OS.
I noticed that for a few days installation of server failed with the following error.

TASK [freeipa.ansible_freeipa.ipaclient : Install - IPA API calls for remaining enrollment parts] ***
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ipalib.errors.ACIError: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credential cache is empty)
fatal: [freeiparocky.thierryosecu.org]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 10.194.76.92 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py\", line 120, in get_package\r\n    plugins = api._remote_plugins\r\nAttributeError: 'API' object has no attribute '_remote_plugins'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/localadmin/.ansible/tmp/ansible-tmp-1705585790.0157206-834-254775280243940/AnsiballZ_ipaclient_api.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/localadmin/.ansible/tmp/ansible-tmp-1705585790.0157206-834-254775280243940/AnsiballZ_ipaclient_api.py\", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/localadmin/.ansible/tmp/ansible-tmp-1705585790.0157206-834-254775280243940/AnsiballZ_ipaclient_api.py\", line 48, in invoke_module\r\n    run_name='__main__', alter_sys=True)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\r\n    mod_name, mod_spec, pkg_name, script_name)\r\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_freeipa.ansible_freeipa.ipaclient_api_payload_vbf18ttn/ansible_freeipa.ansible_freeipa.ipaclient_api_payload.zip/ansible_collections/freeipa/ansible_freeipa/plugins/modules/ipaclient_api.py\", line 251, in <module>\r\n  File \"/tmp/ansible_freeipa.ansible_freeipa.ipaclient_api_payload_vbf18ttn/ansible_freeipa.ansible_freeipa.ipaclient_api_payload.zip/ansible_collections/freeipa/ansible_freeipa/plugins/modules/ipaclient_api.py\", line 169, in main\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/plugable.py\", line 753, in finalize\r\n    self.__do_if_not_done('load_plugins')\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/plugable.py\", line 432, in __do_if_not_done\r\n    getattr(self, name)()\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/plugable.py\", line 632, in load_plugins\r\n    for package in self.packages:\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/__init__.py\", line 952, in packages\r\n    ipaclient.remote_plugins.get_package(self),\r\n  File \"/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py\", line 128, in get_package\r\n    plugins = schema.get_package(server_info, client)\r\n  File \"/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py\", line 546, in get_package\r\n    schema = Schema(client)\r\n  File \"/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py\", line 395, in __init__\r\n    fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)\r\n  File \"/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py\", line 420, in _fetch\r\n    schema = client.forward(u'schema', **kwargs)['result']\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/rpc.py\", line 1146, in forward\r\n    return self._call_command(command, params)\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/rpc.py\", line 1122, in _call_command\r\n    return command(*params)\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/rpc.py\", line 1276, in _call\r\n    return self.__request(name, args)\r\n  File \"/usr/lib/python3.6/site-packages/ipalib/rpc.py\", line 1270, in __request\r\n    raise error_class(**kw)\r\nipalib.errors.ACIError: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credential cache is empty)\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

After some investigation, i found that installation is ok with this version of package "ipa-server-4.9.12-9.module+el8.9.0+1534+4fa0f2bf.x86_64" but failed with the latest "ipa-server-4.9.12-11.module+el8.9.0+1652+4ee71f6a.x86_64".

Installation is ok when i install ipa-server manually using ipa-server-install command.

I've done a lot of tests but I haven't been able to identify the exact cause of the problem.

Regards.
Thierry

[rjeffman]

Can you share what's on ipaserver-install.log?

[talleno]

Hi,
I share the 2 log files in attachment ipaserver-install.log and ipaclient-install.log.
In ipaserver-install.log file, i see the error :
2024-01-22T17:18:11Z DEBUG stderr=ERROR: ERROR: No kra subsystem in instance pki-tomcat.
And the following error in ipaclient-install.log:
2024-01-22T17:18:41Z DEBUG stderr=ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credential cache is empty)

ipaclient-install.log
ipaserver-install.log

[AhmedEldamaty]

I had same issue, enabling setup of kra solved the issue
You can do that by specifying ipaserver_setup_kra: true in the inventory, if you are using ini format use ipaserver_setup_kra=true

[barloff-st]

I had same issue, enabling setup of kra solved the issue You can do that by specifying ipaserver_setup_kra: true in the inventory, if you are using ini format use ipaserver_setup_kra=true

This does fix it for me. I started seeing this issue Jan 9 or 10. What changed then? FWIW, I noticed a working version we have used SSSDConfig 2.7.3 and the one with this issue used SSDConfig 2.8.2. I was not specifying a version. Only other differences were IPA module versions 4.9.10 (working) vs 4.9.12 (the not working one). gssapi version was the same in both 1.5.1.

[rjeffman]

Can you provide the output of the command rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server?

Note that this command need to be executed on the target node, not on the controller.

[talleno]

I had same issue, enabling setup of kra solved the issue You can do that by specifying ipaserver_setup_kra: true in the inventory, if you are using ini format use ipaserver_setup_kra=true

This does fix it for me. I started seeing this issue Jan 9 or 10. What changed then? FWIW, I noticed a working version we have used SSSDConfig 2.7.3 and the one with this issue used SSDConfig 2.8.2. I was not specifying a version. Only other differences were IPA module versions 4.9.10 (working) vs 4.9.12 (the not working one). gssapi version was the same in both 1.5.1.

Great!
It does fix it also for me!
Now the question is why with this new package we have to set this parameter to "true".

[talleno]

Can you provide the output of the command rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server?

Note that this command need to be executed on the targen node, not on the controller.

$ rpm -q ipa-server ipa-client 389-ds-base idm-pki-ca krb5-server
ipa-server-4.9.12-11.module+el8.9.0+1652+4ee71f6a.x86_64
ipa-client-4.9.12-11.module+el8.9.0+1652+4ee71f6a.x86_64
389-ds-base-1.4.3.37-2.module+el8.9.0+1655+39468843.x86_64
idm-pki-ca-10.14.3-1.module+el8.8.0+1160+940e4769.noarch
krb5-server-1.18.2-26.el8.x86_64

[rjeffman]

I confirmed that "there is an issue", I'm still not sure what's going on, but plan to fix it on master before next Monday.

[NavidSassan]

Hi, any news here? Can we help in any way? :)
We are having the same issue with newly installed servers, and a similar problem with existing servers (also installed with this collection) that updated from 4.9.12-9 to 4.9.12-11 (the webgui login fails with "Your session has expired. Please log in again." in the gui and 401 Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credential cache is empty) in the apache log). Weirdly enough this only happens on some servers...
Thanks for your efforts

[talleno]

Hi, any news here? Can we help in any way? :) We are having the same issue with newly installed servers, and a similar problem with existing servers (also installed with this collection) that updated from 4.9.12-9 to 4.9.12-11 (the webgui login fails with "Your session has expired. Please log in again." in the gui and 401 Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credential cache is empty) in the apache log). Weirdly enough this only happens on some servers... Thanks for your efforts

Hi,
After the first problem of installation, i was faced to a same problem on other instance of freeipa where packages had been updated. Perhaps this can help you.
After some investigation, i find that the correction of a CVE about Kerberos constrained delegation(S4U extensions). S4U Kerberos extensions required presence of MS-PAC structures in Kerberos ticket. For generation of MS-PAC we have to have SIDs.
Freeipa use idrange to generate SIDs for users, if uid of users are included in idrange. In my case we fixed uid for users but we didn't have create the corresponding idrange.
After creation of idrange and forcing the generation of SIDs, users were able to connect to webui.

[rjeffman]

The SID/MS-PAC issue may have impacted ansible-freeipa, but it is a different issue. We are facing problems to deploy the first server on CentOS 8 (and its derivatives), so there's no user yet.

I'm still looking into it, but it will take some time, as I'll have very little time next week (due to vacations).

[rjeffman]

BTW... you should NOT be using ipaserver to install a new server to an existing deployment, you should be using ipareplica. ipaserver is used to create the FreeIPA deployment, and additional servers should be added as replicas.

[NavidSassan]

Sorry if I was unclear, I was talking about different, completely separate environments. We have a few existing environments that just got updated to 4.9.12-11. I'm also are currently working on setting up a new environment where I walked into this issue. Hope you have nice holidays :)

Thanks for the hint @talleno, I will have a look

[rjeffman]

No need to be sorry, you guys uncovered a big issue we haven't seen before. Thank you for that.

My comment was just to set the proper use of the roles, in case someone misunderstand what is going on.

[abbra]

Do you still have logs from the failed target? If so, can you provide krb5kdc.log?

If not, can you for a new install attempt and get those installer logs and krb5kdc.log?

[NavidSassan]

of course, here you go:

ipaclient-install.log
ipaserver-install.log
krb5kdc.log

[abbra]

Thanks, so this is a timing issue.

Feb 01 17:58:46 n-infra01.navidsassan.xyz krb5kdc[12323](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 10.42.1.2: S4U2PROXY_EVIDENCE_TKT_WITHOUT_PAC: authtime 1706806725, etypes {rep=UNSUPPORTED:(0)} HTTP/n-infra01.navidsassan.xyz@NAVIDSASSAN.XYZ for ldap/n-infra01.navidsassan.xyz@NAVIDSASSAN.XYZ, KDC policy rejects request
Feb 01 17:58:46 n-infra01.navidsassan.xyz krb5kdc[12323](info): ... CONSTRAINED-DELEGATION s4u-client=<unknown>

KDC driver doesn't see yet that a SID generation task ran by the installer was completed and used old view (SIDs not yet available). I think we would fix this by restarting KDC after sidgen step in IPA installer.

@rjeffman, I remember @jrisc was looking in a similar issue recently, but I don't see that change merged anywhere.

[rjeffman]

PR #1206 was proposed as a fix to this issue.

It would be really nice if anyone having this issue can test the patch and report back the results on your environment. As it is related to a timing issue, none of the proposed workarounds worked on my labs.

[NavidSassan]

Using the PR #1206 the playbook runs without error, and the FreeIPA installation seems to work :D

[NavidSassan]

Will you release a new version to ansible galaxy containing the fix in the near future, or should we use the master branch?

[rjeffman]

We expect to release a new version soon.