Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use non-RFC 8009 enctypes to sign KDC checksum in PAC structure #1268

Closed
wants to merge 0 commits into from

Conversation

abbra
Copy link

@abbra abbra commented Sep 23, 2022

Active Directory does not support RFC 8009 encryption types and [MS-PAC] 2.8.2 spec outlines how cryptographic system has to be selected to produce a KDC checksum. FreeIPA 4.9.10+ defaults to RFC 8009 enctypes, resulting in a PAC buffer with a checksum rejected by Active Directory.

If a local TGT key is one of RFC 8009 encryption types, downgrade the key to use for checksum to either ENCTYPE_AES128_CTS_HMAC_SHA1_96 or ENCTYPE_AES256_CTS_HMAC_SHA1_96 if they available.

Discussed with Microsoft dochelp@ in
https://lists.samba.org/archive/cifs-protocol/2022-September/003761.html

Signed-off-by: Alexander Bokovoy abokovoy@redhat.com

@abbra
Copy link
Author

abbra commented Sep 23, 2022

@greghudson I have an alternative approach in FreeIPA by implementing dbe_search_enctype() in our KDB driver but that breaks some usecases because the key returned is used for other purposes and breaks a checksum validation for in-realm S4U2Proxy usage.

I have an example here:
http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/8b15d7de-3b38-11ed-a128-fa163e02831a/report.html
For example,

INFO     ipatests.pytest_ipa.integration.host.Host.master.IPAOpenSSHTransport:transport.py:391 RUN ['kinit', 'Administrator@ad.test']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd134:transport.py:513 RUN ['kinit', 'Administrator@ad.test']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd134:transport.py:557 Password for Administrator@ad.test: 
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd134:transport.py:217 Exit code: 0
INFO     ipatests.pytest_ipa.integration.host.Host.master.IPAOpenSSHTransport:transport.py:391 RUN ['ipa', 'ping']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd135:transport.py:513 RUN ['ipa', 'ping']
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd135:transport.py:557 ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
DEBUG    ipatests.pytest_ipa.integration.host.Host.master.cmd135:transport.py:217 Exit code: 1

this is caused by the following issue in krb5kdc.log:

Sep 23 13:07:03 master.ipa.test krb5kdc[23691](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.247: ISSUE: authtime 1663938422, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha384-192(20), ses=aes256-cts-hmac-sha1-96(18)}, Administrator@AD.TEST for HTTP/master.ipa.test@IPA.TEST
Sep 23 13:07:03 master.ipa.test krb5kdc[23691](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.247: PROCESS_TGS: authtime 0, etypes {rep=UNSUPPORTED:(0)} <unknown client> for ldap/master.ipa.test@IPA.TEST, Bad encryption type

This 'unknown client' was supposed to be HTTP/master.ipa.test service doing S4U2Proxy on behalf of Administrator@AD.TEST.

@abbra abbra force-pushed the use-correct-mspac-kdc-signature branch 2 times, most recently from c7ea9b0 to e8169b7 Compare September 23, 2022 13:07
@greghudson
Copy link
Member

Just so I'm clear: this is a KDC signature buffer that Microsoft doesn't need to validate, but the buffer size (larger than 20 bytes) nevertheless causes a rejection?

@abbra
Copy link
Author

abbra commented Sep 23, 2022

From dochelp response I believe that they check the size before parsing the buffer. If buffer is of not expected size, it is considered a failure. Since this buffer's presence is required (table in MS-PAC 2.4, https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-pac/3341cfa2-6ef5-42e0-b7bc-4544884bf399, says: "KDC (privilege server) checksum (section 2.8). PAC structures MUST contain one buffer of this type. Additional KDC checksum buffers MUST be ignored."), a failure to parse is considered a rejection.

Overally, buffer types 1, 6, 7, and 10 (0xA) MUST be present when PAC structure is present, according to the same table in MS-PAC 2.4.

@abbra
Copy link
Author

abbra commented Sep 23, 2022

A bit to add, the size check is Windows implementation detail, judging by the dochelp engineer's response in analysis of the TTT trace we provided. It might make sense, indeed: there are three possible signature types, with predefined sizes, so checking size is cheaper than performing a checksum comparison for a general case. If size is wrong, no need to parse the buffer and calculate a checksum.

In my code I use -1 for the salt. MS-PAC 2.8.2 says "The Key Usage Value MUST be KERB_NON_KERB_CKSUM_SALT (17) [MS-KILE] (section 3.1.5.9)." Should we we use a specific checksum salt in the call to KDB?

@abbra
Copy link
Author

abbra commented Sep 24, 2022

TODO: fix get_verified_pac as well -- the part where we fall back to older keys in kvno also need to take into account the right enctype.

@greghudson
Copy link
Member

I'm not sure if this is the best solution to the interop problem, but all of the solutions I can think of carry significant costs:

The current PR solution is not future-proof for new enctypes, and will fail requests (even ones for tickets that would work without PACs) if the krbtgt entry only contains aes-sha2 keys, as we can see in the CI failure. And of course it weakens the security of the realm; realm-internal S4U operations would be relying on SHA-1 checksums when the realm is configured to avoid the use of SHA-1. (This would not be a practical security concern today, as collision attacks against SHA-1 do not affect HMAC checksums. But it could be a compliance issue, and of course it's a general concern.)

Instead of looking for a key of a different enctype, we could just use the aes-sha1 checksum algorithms with the aes-sha2 keys. This would be going outside the usual bounds of RFC 3961 and would be taking advantage of laxness that we currently have in libk5crypto but might not always want to have. And it would still weaken the security of the realm. However, it would eliminate the administrative limitations on realm configuration.

Instead of using a different key type or checksum type, we could truncate the checksums, if the buffer size is the only cause of the interop issue. This would not work for non-deterministic checksum types (there currently are none, but RFC 3961 does not rule them out and there have been some in the past), and would add complexity to pac.c:verify_checksum() as we could no longer use krb5_c_verify_checksum() for truncated checksums. The weakening of the realm security would be perhaps lessened as we would only be reducing the bit strength of the SHA-2 signatures, not the algorithmic strength.

If I understand the interop issue correctly, it only applies to cross-realm TGTs issued by an MIT krb5 KDC and processed by an AD KDC in another realm. (That is, the PACs with overlarge KDC signature buffers aren't being rejected by Windows Kerberos services, only by AD KDCs.) If this is correct, we could make the behavior change conditional on issuing or processing a cross-realm TGT, at the cost of higher complexity. Such tickets usually aren't fed back to the issuing realm (though in theory it's possible to renew a cross-TGT) so the security and logistical concerns would be mitigated somewhat.

Anything we do is likely to become a historical burden if Microsoft relaxes their PAC parsing or implements aes-sha2.

@abbra
Copy link
Author

abbra commented Sep 25, 2022

Yes, current PR is merely a proposal to invoke a practical discussion. It looks like if we want to use aes-sha2 checksum for realm-internal tickets with PAC, we would still be in violation of the MS-PAC 2.8.2, even though this would be accepted by the MIT realm's KDCs that know how to issue PACs and work with aes-sha2 by default -- at this point these are MIT krb5 upstream and FreeIPA 4.9.10+ KDCs.

Samba AD, both MIT- and Heimdal-based, does not support aes-sha2 because Microsoft AD does not have supprot for aes-sha2. Any cross-realm interop with AD KDCs would need to use aes-sha1 checksum, as you noted.

How hard would be to apply the checksum change only to cross-realm TGTs? krb5_kdc_sign_ticket has knowledge of the server principal but krb5_pac_sign_ext it calls has no such knowledge so it cannot choose a different key and different checksum for krb5_c_make_checksum_iov call. I tried a change at handle_pac level:

-    if ((local_tgt_key->enctype == ENCTYPE_AES128_CTS_HMAC_SHA256_128) ||
-        (local_tgt_key->enctype == ENCTYPE_AES256_CTS_HMAC_SHA384_192)) {
+    if ((server != NULL && krb5_is_tgs_principal(server->princ)) &&
+        (signing_tgt != NULL)) {

and it seems to work for me because FreeIPA always has aes-sha1 keys when deployed with trust to Active Directory. If KDB driver could influence this choice dynamically, that would be the best. Changing kdc configuration is problematic.

@abbra
Copy link
Author

abbra commented Sep 25, 2022

though it seems to not work in all cases. I am testing with my experimental FreeIPA KDB driver that supports krb5 1.20 so it prints some additional debug information. It used to work fine before this change for in-realm cases. I am missing code paths for verification, of course:

Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): dbget_princ: ctx=0x5610ce2a2080, kerr: 0, search for HTTP/master.ipa.test, flags: 40 (client:1, referrals:0)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): dbget_princ: ctx=0x5610ce2a2080, kerr: 0, search for krbtgt/IPA.TEST, flags: 0 (client:0, referrals:0)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): issue_pac: kerr: 0, client HTTP/master.ipa.test, flags: 40 (client:1, referrals:0)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): issue_pac: kerr: 0, server krbtgt/IPA.TEST
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): issue_pac: old_pac: (nil), new_pac: 0x5610ce32d530, with pac: 1
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac[lentry]: deref_search for krbprincipalname=HTTP/master.ipa.test@IPA.TEST,cn=services,cn=accounts,dc=ipa,dc=test result kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: first_entry result lentry: 0x5610ce3ea6f0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): fill_info3: parsed principal name HTTP/master.ipa.test@IPA.TEST, ret: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): fill_info3: added asserted identity, ret: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: fill_info3 result kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: UPN_DNS_LOGON_INFO principal HTTP/master.ipa.test@IPA.TEST, kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: sid_from_pac result kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: pac_attrs_blob result kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): ipadb_get_pac: kerr: 0
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 192.168.122.203: ISSUE: authtime 1664097708, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha384-192(20), ses=aes256-cts-hmac-sha1-96(18)}, HTTP/master.ipa.test@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](info): closing down fd 11
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): dbget_princ: ctx=0x5610ce2a2080, kerr: 0, search for krbtgt/IPA.TEST, flags: 0 (client:0, referrals:0)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): dbget_princ: ctx=0x5610ce2a2080, kerr: 0, search for ldap/master.ipa.test, flags: 10 (client:0, referrals:1)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](Error): dbget_princ: ctx=0x5610ce2a2080, kerr: 0, search for HTTP/master.ipa.test, flags: 0 (client:0, referrals:0)
Sep 25 09:21:48 master.ipa.test krb5kdc[21067](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.203: 2ND_TKT_PAC: authtime 0, etypes {rep=UNSUPPORTED:(0)} HTTP/master.ipa.test@IPA.TEST for ldap/master.ipa.test@IPA.TEST, Message stream modified

@greghudson
Copy link
Member

It looks like if we want to use aes-sha2 checksum for realm-internal tickets with PAC, we would still be in violation of the MS-PAC 2.8.2

Right. It's a bad specification that doesn't permit new enctypes, but it's their spec and we took on that cost by using it in the core KDC code. For realm-internal operations we have to decide whether it's better to violate the spec or to downgrade enctypes. For cross-realm operations we obviously need to interoperate.

If KDB driver could influence this choice dynamically, that would be the best. Changing kdc configuration is problematic.

Because the KDC might know about the foreign realm's capabilities? I can't think of any other reason to consult the KDB for the KDC signature key.

I thought of another potential solution: we could derive an aes-sha1 key using krb5_c_derive_prfplus() for the KDC signature. That wouldn't be abusing the crypto library, wouldn't require an aes-sha1 key in the krbtgt principal, and wouldn't require adding complexity to pac.c as checksum truncation would. We would still be downgrading enctypes; my primary practical concern there is that I could imagine Fedora wanting to disable SHA-1 in its OpenSSL library at some point.

If we did this for all PACs then we have another upgrade like 1.20 where cross-version KDCs would reject each others' tickets for S4U operations (not the end of the world, I think). If we do this for only cross-TGT PACs then there would be much less likely to be a practical issue with upgrades, although we would still run the risk of interoperation issues with realm-internal MIcrosoft application servers.

@abbra
Copy link
Author

abbra commented Sep 26, 2022

It looks like if we want to use aes-sha2 checksum for realm-internal tickets with PAC, we would still be in violation of the MS-PAC 2.8.2

Right. It's a bad specification that doesn't permit new enctypes, but it's their spec and we took on that cost by using it in the core KDC code. For realm-internal operations we have to decide whether it's better to violate the spec or to downgrade enctypes. For cross-realm operations we obviously need to interoperate.

One realm-internal setup I can think about is running Microsoft SQL Server on a Linux host enrolled into MIT Kerberos-based realm (we have this with RHEL IdM). MS SQL Server uses S4U operations actively to refresh group membership information of users logged into the server. On the other hand, judging by https://rhel.pkgs.org/8/mssql-server-2019-x86_64/mssql-server-15.0.4249.2-1.x86_64.rpm.html, it is built against krb5-libs, so may be we are lucky here if parsing and verifying PAC is done by the library.

If KDB driver could influence this choice dynamically, that would be the best. Changing kdc configuration is problematic.

Because the KDC might know about the foreign realm's capabilities? I can't think of any other reason to consult the KDB for the KDC signature key.

FreeIPA and Samba AD have separate entries that represent trust objects on top of cross-realm principals. These entries include details about trust configuration (direction details, transitivity properties, SID filtering, encryption types supported, etc). I am thinking about future situation when FreeIPA would be able to trust another FreeIPA and we would like to not downgrade the signature.

I thought of another potential solution: we could derive an aes-sha1 key using krb5_c_derive_prfplus() for the KDC signature. That wouldn't be abusing the crypto library, wouldn't require an aes-sha1 key in the krbtgt principal, and wouldn't require adding complexity to pac.c as checksum truncation would. We would still be downgrading enctypes; my primary practical concern there is that I could imagine Fedora wanting to disable SHA-1 in its OpenSSL library at some point.

Fedora and RHEL aren't issue here: as long as Microsoft's Active Directory implementation is using SHA-1, there will be a way to enable SHA-1-based checksums and encryption types. It is, in fact, the case in RHEL 9 right now: while by default RHEL IdM uses SHA-2 encryption types, SHA-1 checksums supported and even required when trust to Active Directory is considered. This is documented in RHEL documentation:

If we did this for all PACs then we have another upgrade like 1.20 where cross-version KDCs would reject each others' tickets for S4U operations (not the end of the world, I think). If we do this for only cross-TGT PACs then there would be much less likely to be a practical issue with upgrades, although we would still run the risk of interoperation issues with realm-internal MIcrosoft application servers.

Right. I need to test realm-internal MS SQL server case. Being able to limit this change to cross-realm TGT PACs will help.

@greghudson
Copy link
Member

greghudson commented Dec 22, 2022

I have some new thoughts on this after doing the initial work on the full PAC signatures (PR #1284), although they may not lead anywhere productive:

  • privsvr checksums in TGTs are dead weight. If a cross TGT is fed to a different realm, it can't be checked. If a TGT is fed back to the realm that issued it, only the server signature needs to be checked. However, some KDCs may check it anyway (the MIT KDC does not; I think Heimdal might from my read of the code; I don't know about Windows). Perhaps we could get away with just not putting privsvr checksums in cross-realm TGTs? Probably not; if Windows KDCs are needlessly picky about the privsvr checksum size in a cross-realm TGT, we might reasonably assume that they are also needlessly picky about the privsvr checksum not existing.

  • From a security standpoint, I think the privsvr checksum in a service ticket is dead weight after the addition of the full checksum. (The full checksum does not cover the server checksum, but I don't believe that matters.) We still haven't done the testing to know if Windows application servers are picky about privsvr checksum buffer sizes, and I guess we now have the new question of whether they are picky about full checksum buffer sizes.

  • kadmin/changepw tickets exist in a kind of intermediate nether. [MS-PAC] says that the new ticket checksum shouldn't appear in a kadmin/changepw ticket, and I would guess the same will apply to the new full checksum. The MIT KDC does not currently except kadmin/changepw tickets when determining whether to verify privileged checksums like it does for TGTs. However, there is basically no reason to feed a kadmin/changepw ticket to a KDC as input (you could renew one, but why?), so this largely doesn't matter.

@cryptomilk
Copy link
Contributor

@josephsutton1 Do we have tests for the privsvr checksums to verify Windows behavior in Samba which Greg talks about?

@jsutton24
Copy link
Contributor

I don't know that they cover every edge case you're interested in, but we do have some tests for the new checksum type. See Samba commit a50a2be622afaa7a280312ea12f5eb9c9a0c41da, which adds tests in compatability_tests.py and s4u_tests.py, as well as having existing tests check for the new signature. The environment variable FULL_SIG_SUPPORT is used to indicate support for the new signature type.

@greghudson
Copy link
Member

I'd like to wrap this up soonish since it's better to handle it before the 1.21 release.

I'll make the simplifying assumption that we will use key derivation rather than key searching to force the enctype. The options are, then:

  1. Always derive an aes256-sha1 key for use as the PAC privsvr key:
  • Simplest.
  • Conforms to [MS-PAC].
  • Might be necessary to interoperate with in-realm MS services if they are intolerant in the same way AD KDCs are, but we don't know if this is true.
  • Requires a PAC flag day upgrade but we're already doing that for full checksums.
  • Requires another PAC flag day upgrade (or a complicated multi-stage upgrade process) to change the enctype later.
  1. Do (1) only for cross-TGTs:
  • Has virtually no operational impact; full checksums don't appear in cross TGTs and privsvr checksums in cross-TGTs are almost never checked (only in a ticket modification request for a cross-TGT).
  • More complicated.
  • Might not interoperate with in-realm services as mentioned above.
  1. Do (2) but let the KDB weigh in on whether it is necessary for the foreign realm.
  • The KDB may know if the foreign realm is an AD realm.
  • The only option that provides a current and complete migration path away from aes-sha1 enctypes.
  • We'd have to pick a default for dumb (in-tree) KDB modules.
  • The most complicated option.

Does anyone have preferences?

@abbra
Copy link
Author

abbra commented Jan 30, 2023

Hi @greghudson. We have been discussing this with @jrisc. I think the best approach would be a blend of (2) and (3) without an explicit interface for KDB. As you pointed in a different discussion we had, KDB driver can already inform KDC about per-principal suggestions with string attributes. In FreeIPA we can modify KDB driver to attach a particular string attribute to cross-realm principals for trust to Active Directory. We can also attach the same attribute to in-realm principals associated with clients which do not understand new encryption types. This mostly would apply to Windows clients coerced to join 'IPA domain' or to Microsoft software like SQLServer running on Linux hosts. I am still trying to understand whether the latter is a real problem for us.

If string attribute is supported, it will become easy way to configure use of aes256-sha1 key. I guess, it would also allow easier validation as well on the KDC side. What do you think?

@greghudson
Copy link
Member

That seems like a good approach, as any KDB module could use it (some automatically and some with an explicit kadmin setstr).

@abbra
Copy link
Author

abbra commented Feb 10, 2023

@greghudson any suggestion what to use for that attribute? There is already session_enctypes used for choosing a session key per principal but it is not used for the signatures' code.

@greghudson
Copy link
Member

What name, you mean? I was thinking pac_privsvr_enctype.

@abbra
Copy link
Author

abbra commented Feb 10, 2023

Ok, good enough. Thank you.

@greghudson greghudson force-pushed the use-correct-mspac-kdc-signature branch from e9289a0 to 077a7e9 Compare March 21, 2023 04:52
@greghudson
Copy link
Member

I have pushed an implementation of the design we discussed.

@abbra
Copy link
Author

abbra commented Mar 21, 2023

Thank you, this looks good to me.

@greghudson greghudson closed this Mar 27, 2023
@greghudson greghudson force-pushed the use-correct-mspac-kdc-signature branch from 077a7e9 to 654f257 Compare March 27, 2023 18:25
@abbra
Copy link
Author

abbra commented May 26, 2023

Reviving this discussion. I have extended FreeIPA to provide the hint for enctype of the cross-realm tgt and it is accepted. However, we still fail the operation for S4U2Self+S4U2Proxy case because while a service gets the correct krbtgt when asking for S4U2Self, the next S4U2Proxy request fails with

[31956] 1685102455.702982: Got cred; -1765328370/KDC has no support for encryption type

Below is a full trace output. I also have network traces and a keytab.

[root@master ~]# kinit -kt /etc/samba/samba.keytab cifs/`hostname`
[root@master ~]# KRB5_TRACE=/dev/stderr kvno -U Administrator@AD2022-C3GO.TEST -k /etc/samba/samba.keytab -P cifs/`hostname`@IPA.TEST cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST
[31956] 1685102455.702836: Getting credentials Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[31956] 1685102455.702837: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702838: Retrieving Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702839: Getting initial credentials for Administrator\@AD2022-C3GO.TEST@IPA.TEST
[31956] 1685102455.702841: Sending unauthenticated request
[31956] 1685102455.702842: Sending request (196 bytes) to IPA.TEST
[31956] 1685102455.702843: Initiating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702844: Sending TCP request to stream 10.0.198.152:88
[31956] 1685102455.702845: Received answer (150 bytes) from stream 10.0.198.152:88
[31956] 1685102455.702846: Terminating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702847: Response was from primary KDC
[31956] 1685102455.702848: Received error from KDC: -1765328316/Realm not local to KDC
[31956] 1685102455.702849: Following referral to realm AD2022-C3GO.TEST
[31956] 1685102455.702851: Sending unauthenticated request
[31956] 1685102455.702852: Sending request (212 bytes) to AD2022-C3GO.TEST
[31956] 1685102455.702853: Initiating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702854: Sending TCP request to stream 10.0.198.229:88
[31956] 1685102455.702855: Received answer (198 bytes) from stream 10.0.198.229:88
[31956] 1685102455.702856: Terminating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702857: Response was from primary KDC
[31956] 1685102455.702858: Received error from KDC: -1765328359/Additional pre-authentication required
[31956] 1685102455.702859: Identified realm of client principal as AD2022-C3GO.TEST
[31956] 1685102455.702860: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[31956] 1685102455.702861: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702862: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702863: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST using ccache KCM:0
[31956] 1685102455.702864: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702865: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702866: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[31956] 1685102455.702867: Starting with TGT for client realm: cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST
[31956] 1685102455.702868: Requesting tickets for krbtgt/AD2022-C3GO.TEST@IPA.TEST, referrals on
[31956] 1685102455.702869: Generated subkey for TGS request: aes256-cts/A3B8
[31956] 1685102455.702870: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[31956] 1685102455.702872: Encoding request body and padata into FAST request
[31956] 1685102455.702873: Sending request (1991 bytes) to IPA.TEST
[31956] 1685102455.702874: Initiating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702875: Sending TCP request to stream 10.0.198.152:88
[31956] 1685102455.702876: Received answer (1909 bytes) from stream 10.0.198.152:88
[31956] 1685102455.702877: Terminating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702878: Response was from primary KDC
[31956] 1685102455.702879: Decoding FAST response
[31956] 1685102455.702880: FAST reply key: aes256-cts/77D7
[31956] 1685102455.702881: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST with session key aes256-cts/3A2D
[31956] 1685102455.702882: TGS request result: 0/Success
[31956] 1685102455.702883: Received creds for desired service krbtgt/AD2022-C3GO.TEST@IPA.TEST
[31956] 1685102455.702884: Storing cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST in KCM:0
[31956] 1685102455.702885: Get cred via TGT krbtgt/AD2022-C3GO.TEST@IPA.TEST after requesting cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST (canonicalize on)
[31956] 1685102455.702886: Generated subkey for TGS request: aes256-cts/ED97
[31956] 1685102455.702887: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[31956] 1685102455.702889: Encoding request body and padata into FAST request
[31956] 1685102455.702890: Sending request (2490 bytes) to AD2022-C3GO.TEST
[31956] 1685102455.702891: Initiating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702892: Sending TCP request to stream 10.0.198.229:88
[31956] 1685102455.702893: Received answer (2066 bytes) from stream 10.0.198.229:88
[31956] 1685102455.702894: Terminating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702895: Response was from primary KDC
[31956] 1685102455.702896: Decoding FAST response
[31956] 1685102455.702897: FAST reply key: aes256-cts/E495
[31956] 1685102455.702898: Reply server krbtgt/IPA.TEST@AD2022-C3GO.TEST differs from requested cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST
[31956] 1685102455.702899: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@AD2022-C3GO.TEST with session key aes256-cts/B32C
[31956] 1685102455.702900: Got cred; 0/Success
[31956] 1685102455.702901: Get cred via TGT krbtgt/IPA.TEST@AD2022-C3GO.TEST after requesting cifs/master.ipa.test@IPA.TEST (canonicalize on)
[31956] 1685102455.702902: Generated subkey for TGS request: aes256-cts/226A
[31956] 1685102455.702903: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[31956] 1685102455.702905: Encoding request body and padata into FAST request
[31956] 1685102455.702906: Sending request (2429 bytes) to IPA.TEST
[31956] 1685102455.702907: Initiating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702908: Sending TCP request to stream 10.0.198.152:88
[31956] 1685102455.702909: Received answer (2175 bytes) from stream 10.0.198.152:88
[31956] 1685102455.702910: Terminating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702911: Response was from primary KDC
[31956] 1685102455.702912: Decoding FAST response
[31956] 1685102455.702913: FAST reply key: aes256-cts/E35D
[31956] 1685102455.702914: TGS reply is for Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST with session key aes256-cts/895A
[31956] 1685102455.702915: Got cred; 0/Success
[31956] 1685102455.702916: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[31956] 1685102455.702917: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702918: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702919: Storing Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST in KCM:0
cifs/master.ipa.test@IPA.TEST: kvno = 1, keytab entry valid
[31956] 1685102455.702920: Retrieving  -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702921: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST using ccache KCM:0
[31956] 1685102455.702922: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702923: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[31956] 1685102455.702924: Get cred via TGT krbtgt/IPA.TEST@IPA.TEST after requesting cifs/master.ipa.test@IPA.TEST (canonicalize on)
[31956] 1685102455.702925: Generated subkey for TGS request: aes256-cts/B39E
[31956] 1685102455.702926: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[31956] 1685102455.702928: Encoding request body and padata into FAST request
[31956] 1685102455.702929: Sending request (4968 bytes) to IPA.TEST
[31956] 1685102455.702930: Initiating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702931: Sending TCP request to stream 10.0.198.152:88
[31956] 1685102455.702932: Received answer (2261 bytes) from stream 10.0.198.152:88
[31956] 1685102455.702933: Terminating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702934: Response was from primary KDC
[31956] 1685102455.702935: Decoding FAST response
[31956] 1685102455.702936: FAST reply key: aes256-cts/1585
[31956] 1685102455.702937: TGS reply is for Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST with session key aes256-cts/22BB
[31956] 1685102455.702938: Got cred; 0/Success
[31956] 1685102455.702939: Storing Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST in KCM:0
[31956] 1685102455.702940: Getting credentials Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[31956] 1685102455.702941: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702942: Retrieving Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702943: Getting initial credentials for Administrator\@AD2022-C3GO.TEST@IPA.TEST
[31956] 1685102455.702945: Sending unauthenticated request
[31956] 1685102455.702946: Sending request (196 bytes) to IPA.TEST
[31956] 1685102455.702947: Initiating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702948: Sending TCP request to stream 10.0.198.152:88
[31956] 1685102455.702949: Received answer (150 bytes) from stream 10.0.198.152:88
[31956] 1685102455.702950: Terminating TCP connection to stream 10.0.198.152:88
[31956] 1685102455.702951: Response was from primary KDC
[31956] 1685102455.702952: Received error from KDC: -1765328316/Realm not local to KDC
[31956] 1685102455.702953: Following referral to realm AD2022-C3GO.TEST
[31956] 1685102455.702955: Sending unauthenticated request
[31956] 1685102455.702956: Sending request (212 bytes) to AD2022-C3GO.TEST
[31956] 1685102455.702957: Initiating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702958: Sending TCP request to stream 10.0.198.229:88
[31956] 1685102455.702959: Received answer (198 bytes) from stream 10.0.198.229:88
[31956] 1685102455.702960: Terminating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702961: Response was from primary KDC
[31956] 1685102455.702962: Received error from KDC: -1765328359/Additional pre-authentication required
[31956] 1685102455.702963: Identified realm of client principal as AD2022-C3GO.TEST
[31956] 1685102455.702964: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[31956] 1685102455.702965: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702966: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: 0/Success
cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST: kvno = 1, keytab entry valid
[31956] 1685102455.702967: Retrieving  -> cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702968: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST using ccache KCM:0
[31956] 1685102455.702969: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[31956] 1685102455.702970: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[31956] 1685102455.702971: Get cred via TGT krbtgt/IPA.TEST@IPA.TEST after requesting cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST (canonicalize on)
[31956] 1685102455.702972: Generated subkey for TGS request: aes256-cts/7960
[31956] 1685102455.702973: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[31956] 1685102455.702975: Encoding request body and padata into FAST request
[31956] 1685102455.702976: Sending request (5388 bytes) to AD2022-C3GO.TEST
[31956] 1685102455.702977: Initiating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702978: Sending TCP request to stream 10.0.198.229:88
[31956] 1685102455.702979: Received answer (104 bytes) from stream 10.0.198.229:88
[31956] 1685102455.702980: Terminating TCP connection to stream 10.0.198.229:88
[31956] 1685102455.702981: Response was from primary KDC
[31956] 1685102455.702982: Got cred; -1765328370/KDC has no support for encryption type
kvno: KDC has no support for encryption type cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST: constrained delegation failed
[root@master ~]# 
[root@master ~]# klist -ef
Ticket cache: KCM:0
Default principal: cifs/master.ipa.test@IPA.TEST

Valid starting       Expires              Service principal
05/26/2023 12:00:48  05/27/2023 11:17:17  krbtgt/IPA.TEST@IPA.TEST
	Flags: FIA, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192 
05/26/2023 12:00:55  05/27/2023 11:17:17  krbtgt/AD2022-C3GO.TEST@IPA.TEST
	Flags: FAT, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
05/26/2023 12:00:55  05/27/2023 11:17:17  cifs/master.ipa.test@IPA.TEST
	for client Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST, Flags: FAT, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192 

@abbra
Copy link
Author

abbra commented May 26, 2023

Network trace and keytabs from the test environment:
trace-ipa-ad-s4u.tar.gz

@greghudson
Copy link
Member

I can't view the PACs within the tickets in that network trace using Wireshark, I think because neither of the keytabs contains the cross-realm TGT key (krbtgt/AD2022-C3GO.TEST@IPA.TEST). The local krbtgt key for IPA.TEST might also be helpful.

@abbra
Copy link
Author

abbra commented May 27, 2023

Sorry, attaching an updated keytab (and the same trace for convenience) that contains all keys from both sides.
trace-ipa-ad-s4u-updated.tar.gz

My specific problem is the packet 103 that has evidence ticket with etype 20 (using AES256-CTS-HMAC-SHA384-192) instead of etype 18 (AES256-CTS-HMAC-SHA1-96).

additional-tickets: 1 item
    Ticket
        tkt-vno: 5
        realm: IPA.TEST
        sname
            name-type: kRB5-NT-PRINCIPAL (1)
            sname-string: 2 items
                SNameString: cifs
                SNameString: master.ipa.test
        enc-part
            etype: eTYPE-AES256-CTS-HMAC-SHA384-192 (20)
            kvno: 1
            cipher: 214e06cac2869f3ef1b2e0dda32fe3cbc071fe900c1c0107361b7063511c2cf2be269875…

@greghudson
Copy link
Member

Tentatively, I think this is a client problem, not a KDC problem.

The failing request is made to AD2022-C3GO.TEST, but both the header ticket and evidence tickets are issued by IPA.TEST (and not for cross-realm TGTs). The final query to the remote realm in an RBCD exchange. In an RBCD flow, the request to the remote realm should have cross-TGTs in both the header and evidence tickets, with the impersonator's PAC in the header ticket and the impersonated client's PAC in the evidence ticket. The AD KDC is probably not complaining about the PAC checksum, but rather the enc-part of the header or evidence tickets (which also use enctype 20).

In the traces, we start with an expected S4U2Self flow for an enterprise client principal: use AS-REQs to identify the realm as AD2022-C3GO.TEST and then a couple of TGS-REQs to get Administrator@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST tickets. Then things seem to go off the rails, as there is another realm identification leading up to the failing TGS request. I am not sure where we are in the code at that point, as the trace logging for client S4U operations is unfortunately a bit weak.

@greghudson
Copy link
Member

greghudson commented May 28, 2023

The mystery of the repeated realm identification is explained by looking carefully at the command line:

kvno -U Administrator@AD2022-C3GO.TEST -k /etc/samba/samba.keytab -P cifs/`hostname`@IPA.TEST cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST

kvno -P does not take an argument, so both cifs/master.ipa.test@IPA.TEST and cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST are treated as positional arguments. So kvno does:

  1. S4U2Self from Administrator@AD2022-C3GO.TEST to cifs/master.ipa.test@IPA.TEST (succeeds)
  2. S4U2Proxy to cifs/master.ipa.test@IPA.TEST (trivial; succeeds)
  3. S4U2Self from Administrator@AD2022-C3GO.TEST to cifs/master.ipa.test@IPA.TEST (again; succeeds)
  4. S4U2Proxy to cifs/ad1-c3go.ad2022-c3go.test@AD2022-C3GO.TEST (fails)

The second S4U2Proxy request fails because cross-realm S4U2Proxy must use referrals. This kvno invocation instead explicitly specifies a foreign realm for the server principal. We went back and forth on how to handle this case in the PR implementing RBCD (#912 (comment)). We decided to make this absurd-seeming S4U2Proxy request because it can succeed if the apparently-foreign service realm is actually an alias for the local realm.

Try editing the command line to read:

kvno -U Administrator@AD2022-C3GO.TEST -k /etc/samba/samba.keytab -P cifs/ad1-c3go.ad2022-c3go.test@

@abbra
Copy link
Author

abbra commented May 28, 2023

Thank you.

I tried the suggested command and it did fail on KDC side not finding cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST. I need to investigate whether this is a problem with IPA KDB.
ipa-to-ad-s4u-2.tar.gz

[root@master ~]# kinit -kt /etc/samba/samba.keytab cifs/`hostname`
[root@master ~]# KRB5_TRACE=/dev/stderr kvno -U Administrator@AD2022-C3GO.TEST -k /etc/samba/samba.keytab -P  cifs/ad1-c3go.ad2022-c3go.test@
[36090] 1685242042.915338: Getting credentials Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[36090] 1685242042.915339: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915340: Retrieving Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915341: Getting initial credentials for Administrator\@AD2022-C3GO.TEST@IPA.TEST
[36090] 1685242042.915343: Sending unauthenticated request
[36090] 1685242042.915344: Sending request (196 bytes) to IPA.TEST
[36090] 1685242042.915345: Initiating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915346: Sending TCP request to stream 10.0.198.152:88
[36090] 1685242042.915347: Received answer (150 bytes) from stream 10.0.198.152:88
[36090] 1685242042.915348: Terminating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915349: Response was from primary KDC
[36090] 1685242042.915350: Received error from KDC: -1765328316/Realm not local to KDC
[36090] 1685242042.915351: Following referral to realm AD2022-C3GO.TEST
[36090] 1685242042.915353: Sending unauthenticated request
[36090] 1685242042.915354: Sending request (212 bytes) to AD2022-C3GO.TEST
[36090] 1685242042.915355: Initiating TCP connection to stream 10.0.198.229:88
[36090] 1685242042.915356: Sending TCP request to stream 10.0.198.229:88
[36090] 1685242042.915357: Received answer (198 bytes) from stream 10.0.198.229:88
[36090] 1685242042.915358: Terminating TCP connection to stream 10.0.198.229:88
[36090] 1685242042.915359: Response was from primary KDC
[36090] 1685242042.915360: Received error from KDC: -1765328359/Additional pre-authentication required
[36090] 1685242042.915361: Identified realm of client principal as AD2022-C3GO.TEST
[36090] 1685242042.915362: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[36090] 1685242042.915363: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915364: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915365: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST using ccache KCM:0
[36090] 1685242042.915366: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915367: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915368: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[36090] 1685242042.915369: Starting with TGT for client realm: cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST
[36090] 1685242042.915370: Requesting tickets for krbtgt/AD2022-C3GO.TEST@IPA.TEST, referrals on
[36090] 1685242042.915371: Generated subkey for TGS request: aes256-cts/C06D
[36090] 1685242042.915372: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[36090] 1685242042.915374: Encoding request body and padata into FAST request
[36090] 1685242042.915375: Sending request (1991 bytes) to IPA.TEST
[36090] 1685242042.915376: Initiating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915377: Sending TCP request to stream 10.0.198.152:88
[36090] 1685242042.915378: Received answer (1909 bytes) from stream 10.0.198.152:88
[36090] 1685242042.915379: Terminating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915380: Response was from primary KDC
[36090] 1685242042.915381: Decoding FAST response
[36090] 1685242042.915382: FAST reply key: aes256-cts/3C23
[36090] 1685242042.915383: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST with session key aes256-cts/610C
[36090] 1685242042.915384: TGS request result: 0/Success
[36090] 1685242042.915385: Received creds for desired service krbtgt/AD2022-C3GO.TEST@IPA.TEST
[36090] 1685242042.915386: Storing cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST in KCM:0
[36090] 1685242042.915387: Get cred via TGT krbtgt/AD2022-C3GO.TEST@IPA.TEST after requesting cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST (canonicalize on)
[36090] 1685242042.915388: Generated subkey for TGS request: aes256-cts/29FA
[36090] 1685242042.915389: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[36090] 1685242042.915391: Encoding request body and padata into FAST request
[36090] 1685242042.915392: Sending request (2490 bytes) to AD2022-C3GO.TEST
[36090] 1685242042.915393: Initiating TCP connection to stream 10.0.198.229:88
[36090] 1685242042.915394: Sending TCP request to stream 10.0.198.229:88
[36090] 1685242042.915395: Received answer (2066 bytes) from stream 10.0.198.229:88
[36090] 1685242042.915396: Terminating TCP connection to stream 10.0.198.229:88
[36090] 1685242042.915397: Response was from primary KDC
[36090] 1685242042.915398: Decoding FAST response
[36090] 1685242042.915399: FAST reply key: aes256-cts/0873
[36090] 1685242042.915400: Reply server krbtgt/IPA.TEST@AD2022-C3GO.TEST differs from requested cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST
[36090] 1685242042.915401: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@AD2022-C3GO.TEST with session key aes256-cts/4AFA
[36090] 1685242042.915402: Got cred; 0/Success
[36090] 1685242042.915403: Get cred via TGT krbtgt/IPA.TEST@AD2022-C3GO.TEST after requesting cifs/master.ipa.test@IPA.TEST (canonicalize on)
[36090] 1685242042.915404: Generated subkey for TGS request: aes256-cts/60A5
[36090] 1685242042.915405: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[36090] 1685242042.915407: Encoding request body and padata into FAST request
[36090] 1685242042.915408: Sending request (2429 bytes) to IPA.TEST
[36090] 1685242042.915409: Initiating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915410: Sending TCP request to stream 10.0.198.152:88
[36090] 1685242042.915411: Received answer (2175 bytes) from stream 10.0.198.152:88
[36090] 1685242042.915412: Terminating TCP connection to stream 10.0.198.152:88
[36090] 1685242042.915413: Response was from primary KDC
[36090] 1685242042.915414: Decoding FAST response
[36090] 1685242042.915415: FAST reply key: aes256-cts/A0A4
[36090] 1685242042.915416: TGS reply is for Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST with session key aes256-cts/0BE9
[36090] 1685242042.915417: Got cred; 0/Success
[36090] 1685242042.915418: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[36090] 1685242042.915419: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915420: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242042.915421: Storing Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST in KCM:0
cifs/ad1-c3go.ad2022-c3go.test@: kvno = 1, keytab entry valid
[36090] 1685242043.022677: Retrieving  -> cifs/ad1-c3go.ad2022-c3go.test@ from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242043.022678: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST using ccache KCM:0
[36090] 1685242043.022679: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242043.022680: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[36090] 1685242043.022681: Get cred via TGT krbtgt/IPA.TEST@IPA.TEST after requesting cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST (canonicalize on)
[36090] 1685242043.022682: Generated subkey for TGS request: aes256-cts/A38B
[36090] 1685242043.022683: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[36090] 1685242043.022685: Encoding request body and padata into FAST request
[36090] 1685242043.022686: Sending request (4987 bytes) to IPA.TEST
[36090] 1685242043.022687: Initiating TCP connection to stream 10.0.198.152:88
[36090] 1685242043.022688: Sending TCP request to stream 10.0.198.152:88
[36090] 1685242043.022689: Received answer (463 bytes) from stream 10.0.198.152:88
[36090] 1685242043.022690: Terminating TCP connection to stream 10.0.198.152:88
[36090] 1685242043.022691: Response was from primary KDC
[36090] 1685242043.022692: Decoding FAST response
[36090] 1685242043.022693: Decoding FAST response
[36090] 1685242043.022694: Got cred; -1765328377/Server cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST not found in Kerberos database
kvno: Server cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST not found in Kerberos database cifs/ad1-c3go.ad2022-c3go.test@: constrained delegation failed

and krb5kdc.log shows the following:

May 28 02:43:42 master.ipa.test krb5kdc[31345](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 10.0.198.152: NEEDED_PREAUTH: cifs/master.ipa.test@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST, Additional pre-authentication required
May 28 02:43:42 master.ipa.test krb5kdc[31345](info): closing down fd 11
May 28 02:43:42 master.ipa.test krb5kdc[31345](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 10.0.198.152: ISSUE: authtime 1685241822, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha384-192(20), ses=aes256-cts-hmac-sha1-96(18)}, cifs/master.ipa.test@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST
May 28 02:43:42 master.ipa.test krb5kdc[31345](info): closing down fd 11
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 10.0.198.152: REFERRAL: Administrator\@AD2022-C3GO.TEST@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST, Realm not local to KDC
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): closing down fd 11
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 10.0.198.152: ISSUE: authtime 1685241822, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, cifs/master.ipa.test@IPA.TEST for krbtgt/AD2022-C3GO.TEST@IPA.TEST
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): closing down fd 11
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 10.0.198.152: ISSUE: authtime 1685241822, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha384-192(20), ses=aes256-cts-hmac-sha1-96(18)}, cifs/master.ipa.test@IPA.TEST for cifs/master.ipa.test@IPA.TEST
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): ... PROTOCOL-TRANSITION s4u-client=Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): closing down fd 11
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 10.0.198.152: LOOKING_UP_SERVER: authtime 0,  cifs/master.ipa.test@IPA.TEST for cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST, Server not found in Kerberos database
May 28 02:44:07 master.ipa.test krb5kdc[31344](info): closing down fd 11

This last TGS-REQ has the following request in packet 75:

Kerberos
    Record Mark: 4987 bytes
        0... .... .... .... .... .... .... .... = Reserved: Not set
        .000 0000 0000 0000 0001 0011 0111 1011 = Record Length: 4987
    tgs-req
        pvno: 5
        msg-type: krb-tgs-req (12)
        padata: 3 items
            PA-DATA pA-TGS-REQ
                padata-type: pA-TGS-REQ (1)
                    padata-value: 6e82064b30820647a003020105a10302010ea20703050000000000a38205696182056530…
                        ap-req
                            pvno: 5
                            msg-type: krb-ap-req (14)
                            Padding: 0
                            ap-options: 00000000
                                0... .... = reserved: False
                                .0.. .... = use-session-key: False
                                ..0. .... = mutual-required: False
                            ticket
                                tkt-vno: 5
                                realm: IPA.TEST
                                sname
                                    name-type: kRB5-NT-SRV-INST (2)
                                    sname-string: 2 items
                                        SNameString: krbtgt
                                        SNameString: IPA.TEST
                                enc-part
                                    etype: eTYPE-AES256-CTS-HMAC-SHA384-192 (20)
                                    kvno: 1
                                    cipher: 3b937ac46481907043cb2300831e547a0240518c11ba64a3a28941a8dbe38ac70b266ba7…
                            authenticator
                                etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
                                cipher: d0bfd955127b52e18d18d266fd68d4e9f6a14a8d0eb19416ddca9746d328060638dd0e20…
            PA-DATA pA-FX-FAST
                padata-type: pA-FX-FAST (136)
                    padata-value: a08206a5308206a1a1173015a003020110a10e040c3508cfbbbb452202b9288922a28206…
                        armored-data
                            req-checksum
                                cksumtype: cKSUMTYPE-HMAC-SHA1-96-AES-256 (16)
                                checksum: 3508cfbbbb452202b9288922
                            enc-fast-req
                                etype: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
                                cipher: 971a6db7fc7373666b78e4653244aea6be048cbade6923100c855e5901221edd996adcbd…
            PA-DATA pA-PAC-OPTIONS
                padata-type: pA-PAC-OPTIONS (167)
                    padata-value: 3009a00703050010000000
                        Padding: 0
                        flags: 10000000
                            0... .... = claims: False
                            .0.. .... = branch-aware: False
                            ..0. .... = forward-to-full-dc: False
                            ...1 .... = resource-based-constrained-delegation: True
        req-body
            Padding: 0
            kdc-options: 40030000
                0... .... = reserved: False
                .1.. .... = forwardable: True
                ..0. .... = forwarded: False
                ...0 .... = proxiable: False
                .... 0... = proxy: False
                .... .0.. = allow-postdate: False
                .... ..0. = postdated: False
                .... ...0 = unused7: False
                0... .... = renewable: False
                .0.. .... = unused9: False
                ..0. .... = unused10: False
                ...0 .... = opt-hardware-auth: False
                .... 0... = unused12: False
                .... .0.. = unused13: False
                .... ..1. = constrained-delegation: True
                .... ...1 = canonicalize: True
                0... .... = request-anonymous: False
                .0.. .... = unused17: False
                ..0. .... = unused18: False
                ...0 .... = unused19: False
                .... 0... = unused20: False
                .... .0.. = unused21: False
                .... ..0. = unused22: False
                .... ...0 = unused23: False
                0... .... = unused24: False
                .0.. .... = unused25: False
                ..0. .... = disable-transited-check: False
                ...0 .... = renewable-ok: False
                .... 0... = enc-tkt-in-skey: False
                .... .0.. = unused29: False
                .... ..0. = renew: False
                .... ...0 = validate: False
            realm: IPA.TEST
            sname
                name-type: kRB5-NT-PRINCIPAL (1)
                sname-string: 2 items
                    SNameString: cifs
                    SNameString: ad1-c3go.ad2022-c3go.test
            till: May 29, 2023 05:37:59.000000000 EEST
            nonce: 1630024893
            etype: 6 items
                ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
                ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA384-192 (20)
                ENCTYPE: eTYPE-CAMELLIA256-CTS-CMAC (26)
                ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA256-128 (19)
                ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
                ENCTYPE: eTYPE-CAMELLIA128-CTS-CMAC (25)
            additional-tickets: 1 item
                Ticket
                    tkt-vno: 5
                    realm: IPA.TEST
                    sname
                        name-type: kRB5-NT-PRINCIPAL (1)
                        sname-string: 2 items
                            SNameString: cifs
                            SNameString: master.ipa.test
                    enc-part
                        etype: eTYPE-AES256-CTS-HMAC-SHA384-192 (20)
                        kvno: 1
                        cipher: c1e663eb1670edb7d4c1e429c2030919f9a497053288cebecfeb3236f178e3654dac6941…

@abbra
Copy link
Author

abbra commented May 28, 2023

In the trace output it looks like a client is enforcing our realm:

[36090] 1685242043.022677: Retrieving  -> cifs/ad1-c3go.ad2022-c3go.test@ from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242043.022678: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST using ccache KCM:0
[36090] 1685242043.022679: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[36090] 1685242043.022680: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[36090] 1685242043.022681: Get cred via TGT krbtgt/IPA.TEST@IPA.TEST after requesting cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST (canonicalize on)

In addition to that, the client sets principal type to KRB5_NT_PRINCIPAL. FreeIPA expects this to be KRB5_NT_ENTERPRISE_PRINCIPAL to get aliases and referrals to work.

@greghudson
Copy link
Member

greghudson commented May 28, 2023

The referral realm is a client-side construction to work with the existing API, not a protocol concept. That trace doesn't indicate the client enforcing the local realm.

The name type may be an issue, although it would not be usual to use an enterprise name for the service principal. If this were a GSS application using a host-based name, the name type would be KRB5_NT_SRV_HST. kvno can use a service principal of this type if you use "-S cifs" and then the hostname in hte positional argument, although then you have to make sure that hostname canonicalization doesn't mess things up (e.g. set dns_canonicalize_hostname = false in [libdefaults]). Alternatively the kvno -u option can force the name type to NT_UNKNOWN.

On the KDC, referrals can be generated by the KDB layerr, or at the KDC layer using [domain_realm] configuration. Ordinarily the KDC layer only generates referrals for SRV_HST service principals, although it can be configured to generate them for NT_UNKNOWN by setting host_based_services in the kdc.conf realm configuration.

@abbra
Copy link
Author

abbra commented May 29, 2023

This was helpful, thank you. I added support for issuing a referral for the server principal and got a bit further. Now I get an issue at the S4U2Proxy request on IPA KDC side with XREALM_EVIDENCE_TICKET_MISMATCH.

ipa-to-ad-s4u-3.tar.gz

[root@master ~]# kinit -kt /etc/samba/samba.keytab cifs/`hostname`
[root@master ~]# KRB5_TRACE=/dev/stderr kvno -U Administrator@AD2022-C3GO.TEST -k /etc/samba/samba.keytab -P  cifs/ad1-c3go.ad2022-c3go.test@
[55624] 1685349076.661962: Getting credentials Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[55624] 1685349076.661963: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661964: Retrieving Administrator\@AD2022-C3GO.TEST@IPA.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661965: Getting initial credentials for Administrator\@AD2022-C3GO.TEST@IPA.TEST
[55624] 1685349076.661967: Sending unauthenticated request
[55624] 1685349076.661968: Sending request (196 bytes) to IPA.TEST
[55624] 1685349076.661969: Initiating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.661970: Sending TCP request to stream 10.0.198.152:88
[55624] 1685349076.661971: Received answer (150 bytes) from stream 10.0.198.152:88
[55624] 1685349076.661972: Terminating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.661973: Response was from primary KDC
[55624] 1685349076.661974: Received error from KDC: -1765328316/Realm not local to KDC
[55624] 1685349076.661975: Following referral to realm AD2022-C3GO.TEST
[55624] 1685349076.661977: Sending unauthenticated request
[55624] 1685349076.661978: Sending request (212 bytes) to AD2022-C3GO.TEST
[55624] 1685349076.661979: Initiating TCP connection to stream 10.0.198.229:88
[55624] 1685349076.661980: Sending TCP request to stream 10.0.198.229:88
[55624] 1685349076.661981: Received answer (198 bytes) from stream 10.0.198.229:88
[55624] 1685349076.661982: Terminating TCP connection to stream 10.0.198.229:88
[55624] 1685349076.661983: Response was from primary KDC
[55624] 1685349076.661984: Received error from KDC: -1765328359/Additional pre-authentication required
[55624] 1685349076.661985: Identified realm of client principal as AD2022-C3GO.TEST
[55624] 1685349076.661986: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[55624] 1685349076.661987: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661988: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661989: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST using ccache KCM:0
[55624] 1685349076.661990: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661991: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.661992: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[55624] 1685349076.661993: Starting with TGT for client realm: cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST
[55624] 1685349076.661994: Requesting tickets for krbtgt/AD2022-C3GO.TEST@IPA.TEST, referrals on
[55624] 1685349076.661995: Generated subkey for TGS request: aes256-cts/A660
[55624] 1685349076.661996: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[55624] 1685349076.661998: Encoding request body and padata into FAST request
[55624] 1685349076.661999: Sending request (1991 bytes) to IPA.TEST
[55624] 1685349076.662000: Initiating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662001: Sending TCP request to stream 10.0.198.152:88
[55624] 1685349076.662002: Received answer (1909 bytes) from stream 10.0.198.152:88
[55624] 1685349076.662003: Terminating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662004: Response was from primary KDC
[55624] 1685349076.662005: Decoding FAST response
[55624] 1685349076.662006: FAST reply key: aes256-cts/F4E8
[55624] 1685349076.662007: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST with session key aes256-cts/E808
[55624] 1685349076.662008: TGS request result: 0/Success
[55624] 1685349076.662009: Received creds for desired service krbtgt/AD2022-C3GO.TEST@IPA.TEST
[55624] 1685349076.662010: Storing cifs/master.ipa.test@IPA.TEST -> krbtgt/AD2022-C3GO.TEST@IPA.TEST in KCM:0
[55624] 1685349076.662011: Get cred via TGT krbtgt/AD2022-C3GO.TEST@IPA.TEST after requesting cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST (canonicalize on)
[55624] 1685349076.662012: Generated subkey for TGS request: aes256-cts/803B
[55624] 1685349076.662013: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[55624] 1685349076.662015: Encoding request body and padata into FAST request
[55624] 1685349076.662016: Sending request (2490 bytes) to AD2022-C3GO.TEST
[55624] 1685349076.662017: Initiating TCP connection to stream 10.0.198.229:88
[55624] 1685349076.662018: Sending TCP request to stream 10.0.198.229:88
[55624] 1685349076.662019: Received answer (2066 bytes) from stream 10.0.198.229:88
[55624] 1685349076.662020: Terminating TCP connection to stream 10.0.198.229:88
[55624] 1685349076.662021: Response was from primary KDC
[55624] 1685349076.662022: Decoding FAST response
[55624] 1685349076.662023: FAST reply key: aes256-cts/C283
[55624] 1685349076.662024: Reply server krbtgt/IPA.TEST@AD2022-C3GO.TEST differs from requested cifs\/master.ipa.test\@IPA.TEST@AD2022-C3GO.TEST
[55624] 1685349076.662025: TGS reply is for cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@AD2022-C3GO.TEST with session key aes256-cts/2489
[55624] 1685349076.662026: Got cred; 0/Success
[55624] 1685349076.662027: Get cred via TGT krbtgt/IPA.TEST@AD2022-C3GO.TEST after requesting cifs/master.ipa.test@IPA.TEST (canonicalize on)
[55624] 1685349076.662028: Generated subkey for TGS request: aes256-cts/CB42
[55624] 1685349076.662029: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[55624] 1685349076.662031: Encoding request body and padata into FAST request
[55624] 1685349076.662032: Sending request (2429 bytes) to IPA.TEST
[55624] 1685349076.662033: Initiating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662034: Sending TCP request to stream 10.0.198.152:88
[55624] 1685349076.662035: Received answer (2175 bytes) from stream 10.0.198.152:88
[55624] 1685349076.662036: Terminating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662037: Response was from primary KDC
[55624] 1685349076.662038: Decoding FAST response
[55624] 1685349076.662039: FAST reply key: aes256-cts/1F37
[55624] 1685349076.662040: TGS reply is for Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST with session key aes256-cts/CAC2
[55624] 1685349076.662041: Got cred; 0/Success
[55624] 1685349076.662042: Getting credentials Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST using ccache KCM:0
[55624] 1685349076.662043: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.662044: Retrieving Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.662045: Storing Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST -> cifs/master.ipa.test@IPA.TEST in KCM:0
cifs/ad1-c3go.ad2022-c3go.test@: kvno = 1, keytab entry valid
[55624] 1685349076.662046: Retrieving  -> cifs/ad1-c3go.ad2022-c3go.test@ from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.662047: Getting credentials cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST using ccache KCM:0
[55624] 1685349076.662048: Retrieving cifs/master.ipa.test@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:0 with result: -1765328243/Matching credential not found
[55624] 1685349076.662049: Retrieving cifs/master.ipa.test@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from KCM:0 with result: 0/Success
[55624] 1685349076.662050: Get cred via TGT krbtgt/IPA.TEST@IPA.TEST after requesting cifs/ad1-c3go.ad2022-c3go.test@IPA.TEST (canonicalize on)
[55624] 1685349076.662051: Generated subkey for TGS request: aes256-cts/50D2
[55624] 1685349076.662052: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[55624] 1685349076.662054: Encoding request body and padata into FAST request
[55624] 1685349076.662055: Sending request (4988 bytes) to IPA.TEST
[55624] 1685349076.662056: Initiating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662057: Sending TCP request to stream 10.0.198.152:88
[55624] 1685349076.662058: Received answer (495 bytes) from stream 10.0.198.152:88
[55624] 1685349076.662059: Terminating TCP connection to stream 10.0.198.152:88
[55624] 1685349076.662060: Response was from primary KDC
[55624] 1685349076.662061: Decoding FAST response
[55624] 1685349076.662062: Decoding FAST response
[55624] 1685349076.662063: Got cred; -1765328371/KDC can't fulfill requested option
kvno: KDC can't fulfill requested option cifs/ad1-c3go.ad2022-c3go.test@: constrained delegation failed
[root@master ~]# klist 
Ticket cache: KCM:0
Default principal: cifs/master.ipa.test@IPA.TEST

Valid starting       Expires              Service principal
05/29/2023 08:31:16  05/30/2023 08:16:48  krbtgt/AD2022-C3GO.TEST@IPA.TEST
05/29/2023 08:31:11  05/30/2023 08:16:48  krbtgt/IPA.TEST@IPA.TEST
05/29/2023 08:31:16  05/29/2023 18:31:17  cifs/master.ipa.test@IPA.TEST
	for client Administrator\@AD2022-C3GO.TEST@AD2022-C3GO.TEST

@greghudson
Copy link
Member

greghudson commented May 29, 2023

Now I get an issue at the S4U2Proxy request on IPA KDC side with XREALM_EVIDENCE_TICKET_MISMATCH.

From the protocol trace, this is the initial S4U2Proxy request from cifs/master.ipa.test@IPA.TEST to IPA.TEST. The error originates from tgs_policy.c:503, probably because is_referral is true. This test is only supposed to be applied to cross-realm S4U2Proxy requests (see line 475), which this is not(*). The is_crossrealm flag is determined in do_tgs_req.c:686:

if (!data_eq(t->header_server->princ->realm, t->sprinc->realm))
    t->flags |= KRB5_KDB_FLAG_CROSS_REALM;

In the request, the header server realm (the server realm of the ticket inside the PA-TGS-REQ padata) is IPA.TEST, and the request server realm is also IPA.TEST. So the cross-realm flag shouldn't be set, unless for some reason the KDB module put a realm other than IPA.TEST in the principal entry when the header server (krbtgt/IPA.TEST@IPA.TEST) was looked up.

(*) Obviously this is a cross-realm S4U2Proxy flow, but the specific definition of a cross-realm TGS request we use within the KDC code is a request where the TGT originated from a different realm. If IPA.TEST successfully issued a referral, the subsequent request to the AD realm would be a cross-realm request.

@abbra
Copy link
Author

abbra commented May 30, 2023

We can ignore that last one, it was my mistake of short-circuiting server referral case to client referral one in KDB code.
I've got past that and now need to understand why it sill doesn't work the second part of S4U2Proxy request for both in-realm and cross-realm target service for a user from the trusted realm. In both cases I receive S4U2Self ticket from AD DC that is not forwardable and then both MIT KDC (IPA KDC) and AD DC reject the S4U2Proxy request. The more specific detail is why AD thinks this ticket has to be non-forwardable even though we hint we support RBCD.

@abbra
Copy link
Author

abbra commented May 30, 2023

My current version gets further, I only get this denial on the final step of S4U2Proxy request from AD DC. I am yet to find out what is missing. Here is a trace:
ipa-to-ad-s4u-6.tar.gz

@greghudson
Copy link
Member

greghudson commented May 30, 2023

I thought to look at the extended error data in the KRB-ERROR, but it only raises more questions. Wireshark can't interpret it:

BER Error: Wrong field in SEQUENCE: expected class:CONTEXT(2) tag:1 but found class:UNIVERSAL(0) tag:16

Which means it found a 0x30 (SEQUENCE) where it expected 0xA1 (CONTEXT 1). Looking at the bytes by hand, I see:

30 82 01 14 (SEQUENCE, length 276)
  30 82 01 10 (SEQUENCE, length 272)
    A1 04 (CONTEXT 1, length 4)
      02 02 00 88 (INTEGER, length 2, value 136)
    A2 82 01 06 (CONTEXT 2, length 262)
      04 82 01 02 (OCTET STRING, length 258)
        A0 81 FF (CONTEXT 0, length 255)
          30 81 FC (SEQUENCE, length 252)
            A0 81 F9 (CONTEXT 0, length 249)
              30 81 F6 (SEQUENCE, length 246)
                A0 03 (CONTEXT 0, length 3)
                  02 01 12 (INTEGER, length 1, value 18)
                A2 81 EE (CONTEXT 1, length 238)
                  04 81 EB (OCTECT STRING, length 235)

and then 235 bytes of an apparently non-ASN.1 payload. There are several mysteries here:

  1. There are two layers of SEQUENCE wrapping, when only one is expected per [MS-KILE] 2.2.2 (this is likely what Wireshark is complaining about).
  2. If those CONTEXT 1 and 2 fields at the third indent level are KRB-ERROR-DATA fields (they do have the right tags and types), then the data-type is 136. Only types 2 and 3 are defined in the spec.
  3. The OCTET STRING inside the CONTEXT 2 appears to contain a DER encoding, not an NDR encoding as one would expect from a KERB-EXT-ERROR. This DER encoding begins with a CONTEXT 0 tag with no enclosing SEQUENCE, which is unusual in my experience.

@abbra
Copy link
Author

abbra commented May 31, 2023

I've seen the same issue (undocumented returned error payload) in past as well. When I asked dochelp@ about it in https://lists.samba.org/archive/cifs-protocol/2022-September/003769.html, I eventually found out it was AD DC hinting about PA types it supports -- in this case PA 136:

SEQUENCE (1 elem)
  SEQUENCE (2 elem)
    [1] (1 elem)
      INTEGER 136
    [2] (1 elem)
      OCTET STRING (258 byte) A081FF3081FCA081F93081F6A003020112A281EE0481EB4E11F53DA282655593B1412…
        [0] (1 elem)
          SEQUENCE (1 elem)
            [0] (1 elem)
              SEQUENCE (2 elem)
                [0] (1 elem)
                  INTEGER 18
                [2] (1 elem)
                  OCTET STRING (235 byte) 4E11F53DA282655593B141240C4181FA19186A197F8B4DA3D05F1088495975649D338…

@greghudson
Copy link
Member

Oh, of course. This isn't actually an advertisement of padata type support; rather, because the request used FAST TGS, the error is protected using FAST as well. The actual Microsoft extended error (if there is one) is inside the payload. It's unfortunate that Wireshark (at least the version I have installed) can't decode it, as decoding it by hand would be labor-intensive.

@abbra
Copy link
Author

abbra commented May 31, 2023

Aha, thank you. Yes, adding wireshark support for this would be great.
Anyway, I suspect it does not work because we currently cannot add IPA principal into the security descriptor of the target service on AD side. Windows UI for this uses Global Catalog to search for the principal in the AD LDAP but we don't have GC support in FreeIPA. (We do have a prototype and I might try that for completeness before resorting to other methods). But since this is one of final steps of S4U2Proxy process, I think it is the issue here.

From the checksum algorithm replacement feature side, this is now working well. Any idea when krb5 1.21 would be out? We have customers who are waking up to Microsoft's deadlines to enforce PAC full checksum enforcement and would really like to get these improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants