Version 5.1.0 is not backwards compatibility with Microsoft ADFS

[peterfarrell]

Upgrading from 5.0.0 to 5.1.0 without configuration changes to PySAML breaks connectivity using Microsoft ADFS. We are unsure what was added in 5.1.0 that causes this backwards compatibility issue other than it's related to samlp:Extensions.

Code Version

5.1.0 and 5.0.0

Expected Behavior

That 5.1.0 is backwards compatible with 5.0.0 or otherwise document an upgrade strategy.

Current Behavior

When authenticating against MS ADFS using PySAML2 5.1.0, this the error that is logged in ADFS and the authentication fails. Downgrading to PySAML2 5.0.0 fixes the issue.

Exception details:
System.Xml.XmlException: MSIS0009: The <samlp:Extensions> element was encountered. To accept extensions, you must extend the SamlProtocolSerializer.
  at Microsoft.IdentityServer.Protocols.Saml.SamlProtocolSerializer.ReadExtensions(XmlReader reader, SamlMessage message)
   at Microsoft.IdentityServer.Protocols.Saml.SamlProtocolSerializer.ReadCommonElements(XmlReader reader, SamlMessage message)
   at Microsoft.IdentityServer.Protocols.Saml.SamlProtocolSerializer.ReadAuthnRequest(XmlReader reader)
   at Microsoft.IdentityServer.Protocols.Saml.HttpSamlBindingSerializer.ReadProtocolMessage(String encodedSamlMessage)
   at Microsoft.IdentityServer.Protocols.Saml.HttpSamlBindingSerializer.CreateFromNameValueCollection(Uri baseUrl, NameValueCollection collection)
   at Microsoft.IdentityServer.Protocols.Saml.HttpRedirectSamlBindingSerializer.ReadMessage(Uri requestUrl, NameValueCollection form)
   at Microsoft.IdentityServer.Web.Protocols.Saml.HttpSamlMessageFactory.CreateMessage(WrappedHttpListenerRequest httpRequest)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlContextFactory.CreateProtocolContextFromRequest(WrappedHttpListenerRequest request, ProtocolContext& protocolContext)
   at Microsoft.IdentityServer.Web.Protocols.Saml.SamlProtocolHandler.CreateProtocolContext(WrappedHttpListenerRequest request)
   at Microsoft.IdentityServer.Web.PassiveProtocolListener.GetProtocolHandler(WrappedHttpListenerRequest request, ProtocolContext& protocolContext, PassiveProtocolHandler& protocolHandler)
   at Microsoft.IdentityServer.Web.PassiveProtocolListener.OnGetContext(WrappedHttpListenerContext context)

Possible Solution

Document what has changed between versions 5.0.0 and 5.1.0 that causes this.

Steps to Reproduce

[c00kiemon5ter]

hello, and thanks for the report.

I think I know why that happens. You should be getting this while generating an authentication request and the <AuthnRequest> should contain an empty <samlp:Extensions> element.

I will try a fix and let you know, so that you can test it.

[c00kiemon5ter]

See the latest master (f42c3f8). That should work for you. If everything is fine I can create a mini release with this item and maybe a bit of cleanup on that func.

[peterfarrell]

@c00kiemon5ter checking it now. I will report back

[peterfarrell]

@c00kiemon5ter

I installed PySAML2 from git (master) and the problem described in the issue still exists. :-(

I did some further digging and was able to get the XML so I could compare the differences between master and 5.0.0. It appears the issue is the addition of the eIDAS SAML extensions (ns2) that was added in 5.1.0.

A preferred solution is to retain backward compatibility by not enabling the EIDS extension by default. I would suspect that other people will stumble upon this issue in the future. Otherwise, if that isn't possible -- is there a way to disable the extension via configuration?

In the examples below, I have replaced identifying information or X'ed it out:

• application.example.com = consumer
• adfs.example.com = destination

Result from current code in master:

<ns0:AuthnRequest xmlns:ns0="urn:oasis:names:tc:SAML:2.0:protocol" 
                  xmlns:ns1="urn:oasis:names:tc:SAML:2.0:assertion"
                  xmlns:ns2="http://eidas.europa.eu/saml-extensions"
                  AssertionConsumerServiceURL="https://application.example.com/saml2_auth/acs/"
                  Destination="https://adfs.example.com/adfs/ls/" 
                  ID="id-XXXXX"
                  IssueInstant="2020-06-10T21:30:01Z" 
                  ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
                  Version="2.0">
    <ns1:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
        https://application.example.com/saml2_auth/acs/
    </ns1:Issuer>
    <ns0:Extensions>
        <ns2:RequestedAttributes/>
    </ns0:Extensions>
    <ns0:NameIDPolicy Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</ns0:AuthnRequest>

Result from 5.0.0

<ns0:AuthnRequest xmlns:ns0="urn:oasis:names:tc:SAML:2.0:protocol" 
                  xmlns:ns1="urn:oasis:names:tc:SAML:2.0:assertion"
                  AssertionConsumerServiceURL="https://application.example.com/saml2_auth/acs/"
                  Destination="https://adfs.example.com/adfs/ls/" 
                  ID="id-XXXXX"
                  IssueInstant="2020-06-10T21:33:10Z" 
                  ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
                  Version="2.0">
    <ns1:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
        https://application.example.com/saml2_auth/acs/
    </ns1:Issuer>
    <ns0:NameIDPolicy Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</ns0:AuthnRequest>

Thank you for your help so far. Let me know if you need anything else from me. I would be happy to help.

[c00kiemon5ter]

not enabling the EIDS extension by default

that is what the added commit (f42c3f8) is doing. With that you shouldn't be getting the Extensions, nor the RequestedAttributes elements. I can see that the right AuthnRequests are created by the tests.

[peterfarrell]

The result from master I shared above was generated using this:

import saml2
saml_client = _get_saml_client('https://application.example.com')
destination = saml_client._sso_location(binding=saml2.BINDING_HTTP_REDIRECT)
requid, request = saml_client.create_authn_request(destination=destination)
str(request)

I just re-ran it again and we're still getting an Extensions and an empty RequestedAttributes:

<ns0:AuthnRequest xmlns:ns0="urn:oasis:names:tc:SAML:2.0:protocol" 
                  xmlns:ns1="urn:oasis:names:tc:SAML:2.0:assertion"
                  xmlns:ns2="http://eidas.europa.eu/saml-extensions"
                  AssertionConsumerServiceURL="https://application.example.com/saml2_auth/acs/"
                  Destination="https://adfs.example.com/adfs/ls/" 
                  ID="id-XXXXX"
                  IssueInstant="2020-06-11T05:09:15Z" 
                  ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
                  Version="2.0">
    <ns1:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">
        https://application.example.com/saml2_auth/acs/
    </ns1:Issuer>
    <ns0:Extensions>
        <ns2:RequestedAttributes/>
    </ns0:Extensions>
    <ns0:NameIDPolicy Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</ns0:AuthnRequest>

[c00kiemon5ter]

I do not see this with the current master (f42c3f8 and newer). The only way to trigger this would be to pass the requested_attributes to create_authn_request or have a requested_attributes configuration set under services.sp section in the configuration. If those are not there, then there is no way to trigger this.

v5.1.0 got this wrong, but with the current master you should not be seeing this. I do not get the same results as you, above. Did you try this in a clean environment?

$ cd /tmp
$ git clone https://github.com/IdentityPython/pysaml2.git test-saml2
$ cd test-saml2
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install -e . ipython
$ ipython

import saml2
import saml2.config
import saml2.client

conf = saml2.config.SPConfig()
client = saml2.client.Saml2Client(conf)

reqid, req = client.create_authn_request(destination="http://www.example.com/sso")
print(req)

<?xml version="1.0"?>
<ns0:AuthnRequest 
  xmlns:ns0="urn:oasis:names:tc:SAML:2.0:protocol" 
  xmlns:ns1="urn:oasis:names:tc:SAML:2.0:assertion" 
  Destination="http://www.example.com/sso" 
  ID="id-bXiUKMTZgcijXbyFl" 
  IssueInstant="2020-06-11T07:47:29Z" 
  ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" 
  Version="2.0"
>
  <ns1:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity"/>
  <ns0:NameIDPolicy Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</ns0:AuthnRequest>

I would consider this fixed.

[peterfarrell]

@c00kiemon5ter I tried in a clean environment and the fix in master works! The issue was installing from git in our requirements. A push to PyPI (maybe 5.1.1) would be much appreciated.

[c00kiemon5ter]

That's great news! I'm closing this, since this is now resolved.
I will try to get a release out asap.

[peterfarrell]

@c00kiemon5ter Many many thanks for working with us on this and readying a fix. Looking forward to the release.

[c00kiemon5ter]

This is now available in v5.2.0

[peterfarrell]

@c00kiemon5ter Many thanks! We're putting it through our QA process. Looks great!