FIDO Alliance Conformance Test Failures

[swoozey]

When running the FIDO Alliance conformance tests (v1.7.2) against the latest fido2-net-lib master several errors are seen that were not present before the MDS3 support was added. In one of the server "MakeCredential Request" tests we see the conformance test tool making reference to a json extension named "example.extension.bool" whereas in the fido2-net-lib demo project the property is referenced as "example.extension" which causes the test to fail. This issue is to address these current failings so that the fido2-net-lib demo project can pass all the conformance tests.

[aseigler]

These should be all fixed in #277 which is pending.

[andyatyeo]

I might be missing something then. I ran the FIDO conformance tests (now version 1.7.5) against the demo-update branch to get confidence that this branch when merged will address this issue. I selected all the core tests but none of the optional tests. I downloaded the test metadata (in a "Conformance" folder under the Demo folder) and set the appsettings.json to find it (fido2:MDSCacheDirPath="")
This resulted in 16 failures
(for reference, the current master branch for me currently gets 18 failures)
The actual webauthn registering and authenticating FIDO2 tokens is working great in practice (at least with the tokens I have tested), but without conformance test passes it is difficult to have confidence.
So there are 2 possibilities here:

• User Error on my part. Entirely possible!, maybe I have an environmental issue or am missing a setup step (I am investigate this currently)
• (or) Conformance tests do not pass/demo-update branch does not fully address this. (a while ago, before the MD3 updates we saw passing conformance tests on this library via the demo lib, but not since the MD3 updates (judging by exceptions I see at runtime I wonder if it is more the move to System.Json.Text rather then MD3 as such).

If anyone else has seen passing FIDO conformance tests against either the demo-update or the master recently it would be great to know (so I know whether to investigate code issues or environment/setup issues)

This is an example of an error I am getting against both demo-update and master branches (which I highlight this one because I think it relates to the "example.extension.bool" reported on this issue 318 originally (suggesting to me that demo-update alone does not address all conformance issues:

P-1 Get ServerPublicKeyCredentialCreationOptionsResponse, and check that: (a) response MUST contain "status" field, and it MUST be of type DOMString and set to "ok" (b) response MUST contain "errorMessage" field, and it MUST be of type DOMString and set to an empty string (c) response contains "user" field, of type Object and: (1) check that user.name is not missing, and is of type DOMString (2) check that user.displayName is not missing, and is of type DOMString (3) check that user.id is not missing, and is of type DOMString, and is not empty. It MUST be base64url encoded byte sequence, and is not longer than 64 bytes. (4) If user.icon is presented, check that it's is of type DOMString (d) response contains "rp" field, of type Object and: (1) check that rp.name is not missing, and is of type DOMString (2) check that rp.id is not missing, and is of type DOMString. (3) If rp.icon is presented, check that it's is of type DOMString (e) response contains "challenge" field, of type String, base64url encoded and not less than 16 bytes. (f) response contains "pubKeyCredParams" field, of type Array and: (1) each member MUST be of type Object (2) each member MUST contain "type" field of type DOMString (3) each member MUST contain "alg" field of type Number (4) MUST contain one member with type set to "public-key" and alg set to an algorithm that is supported by the authenticator (g) If response contains "timeout" field, check that it's of type Number and is bigger than 0 (h) response contains "extensions" field, with "example.extension" key presented‣ AssertionError: Response.extensions MUST be set to the requested extensions! Expected "{}" to equal "{"example.extension.bool":true}": expected {} to deeply equal { 'example.extension.bool': true } at n.eval (eval at compileCode (js/sandbox.js:25:26), <anonymous>:4670:16)"

[aseigler]

I just ran the test from that branch with all optional stuff turned on and got 161 pass 15 fails.

• Server-ServerPublicKeyCredentialCreationOptions-Req-1, P-1 Get ServerPublicKeyCredentialCreationOptionsResponse, looks like a string parsing thing, I'll look at it
• Server-ServerAuthenticatorAttestationResponse-Resp-1, F-3 Send ServerAuthenticatorAttestationResponse, we accept base64 or base64url for id, test wants to fail if it is base64 and allow base64url only. Makes sense from a conformance standpoint but I don't know. Fixed in Check return value from Base64.DecodeFromUtf8InPlace(), throw if error #331.
• Server-ServerAuthenticatorAttestationResponse-Resp-2, F-15, 16, 17 - these are all token binding related and removed from the spec, not sure why they are still testing for it
• Server-ServerAuthenticatorAttestationResponse-Resp-4, P-8 Send a valid ServerAuthenticatorAttestationResponse, wants us to accept packed with RS1, again makes sense from a conformance perspective but I don't know if I like it.
• Server-ServerAuthenticatorAttestationResponse-Resp-5, P-3 Send a valid ServerAuthenticatorAttestationResponse batch certificate, this is probably related to trust anchor vs trusted root, working on solution there
• F-10 Send ServerAuthenticatorAttestationResponse, probably same/similar as P-3 directly above
• Server-ServerAuthenticatorAttestationResponse-Resp-9, all TPM attestation tests are broken still due to TPM registration responses in v1.6.33 appear to have invalid cert path fido-alliance/conformance-test-tools-resources#618.
• Server-ServerPublicKeyCredentialGetOptionsResponse-Req-1, P-1 Get ServerPublicKeyCredentialGetOptionsResponse, this is same issue as very first failure, will look.
• Server-ServerAuthenticatorAssertionResponse-Resp-2, F-15, 16, 17 - token binding related

[houstonlin]

I just ran the test from that branch with all optional stuff turned on and got 161 pass 15 fails.
how can i to fix ? or Is there anything you can suggest on the program

[googyi]

FYI:
#456
I am working on these issues.

[abergs]

While we still have some tests that are failing, most of this has been adressed.