FIDO2 Server Conformance Testing with the Conformance Tool v1.6.46

[ThaminduDilshan]

Hi @ynojima,

By any chance were you able to run FIDO conformance testing with the tool version v1.6.46. Apparently FIDO alliance has added missing MakeCredential Response test cases back. I'm getting a no of test failures (almost half of the specs) with this tool update in our fido2 server implementation. Majority of the failures are due to data structure validation issues inside webauthn4j library (from fasterxml.jackson). Two frequent exceptions are,

1. InvalidFormatException: value is out of range
2. InvalidFormatException: value is not valid as JWS

Since these failures appear to be from the tool's metadata JSON files, I ran test tool v1.6.46 with the metadata JSONs from a previous version. I ended up in 5 test failues, two from the newly added MakeCredential Response test cases and rest with the previosuly known issues. Below are the two test failures from the MakeCredential Response test cases.

• Server-ServerAuthenticatorAttestationResponse-Resp-4 Test server support of the authentication algorithms: P12

  "com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id '1' as a subtype of `com.webauthn4j.data.attestation.authenticator.COSEKey`: known type ids = [2, 3]
  

• Server-ServerAuthenticatorAttestationResponse-Resp-5 Test server processing "packed" FULL attestation: P3

  java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
  

It would be greatly helpful if you could have a look and comment on this.

Cheers

[ThaminduDilshan]

Update

I'm getting a no of test failures (almost half of the specs) with this tool update in our fido2 server implementation. Majority of the failures are due to data structure validation issues inside webauthn4j library (from fasterxml.jackson). Two frequent exceptions are,

I was able to analyze almost all the stack traces and there's only one root cause for all the failures with the latest version of metadata json files. Total of 67 specs are failing due to this.

Caused by: com.fasterxml.jackson.databind.exc.InvalidFormatException: value is out of range (through reference chain: java.util.ArrayList[0])|

Click to expand the stack trace

at java.lang.Thread.run(Thread.java:750 undefined) [?:1.8.0_321]|Caused by: com.webauthn4j.converter.exception.DataConversionException: Input data does not match expected form|
   at com.webauthn4j.converter.util.JsonConverter.readValue(JsonConverter.java:66 undefined) |
   at com.webauthn4j.metadata.LocalFilesMetadataStatementsProvider.lambda$provide$0(LocalFilesMetadataStatementsProvider.java:46 undefined) |
   at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193 undefined) ~[?:1.8.0_321]|
   at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499 undefined) ~[?:1.8.0_321]|
   at com.webauthn4j.metadata.LocalFilesMetadataStatementsProvider.provide(LocalFilesMetadataStatementsProvider.java:50 undefined) |
   at com.webauthn4j.metadata.anchor.MetadataStatementsBasedTrustAnchorRepository.find(MetadataStatementsBasedTrustAnchorRepository.java:47 undefined) |
   at com.webauthn4j.metadata.anchor.AggregatingTrustAnchorRepository.lambda$find$0(AggregatingTrustAnchorRepository.java:39 undefined) |
   at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267 undefined) ~[?:1.8.0_321]|
   at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499 undefined) ~[?:1.8.0_321]|
   at com.webauthn4j.metadata.anchor.AggregatingTrustAnchorRepository.find(AggregatingTrustAnchorRepository.java:40 undefined) |
   at com.webauthn4j.validator.attestation.trustworthiness.certpath.DefaultCertPathTrustworthinessValidator.validate(DefaultCertPathTrustworthinessValidator.java:65 undefined) |
   at com.webauthn4j.validator.AttestationValidator.validate(AttestationValidator.java:121 undefined) |
   at com.webauthn4j.validator.RegistrationDataValidator.validate(RegistrationDataValidator.java:192 undefined) |
   at com.webauthn4j.WebAuthnRegistrationManager.validate(WebAuthnRegistrationManager.java:204 undefined) |
   at com.webauthn4j.WebAuthnManager.validate(WebAuthnManager.java:161) | ... 55 more|Caused by: com.fasterxml.jackson.databind.exc.InvalidFormatException: value is out of range (through reference chain: java.util.ArrayList[0])|
   at com.webauthn4j.converter.jacksonr.json.AuthenticationAlgorithmFromStringDeserializer.deserialize.deserialize(AuthenticationAlgorithmFromStringDeserializer.java:39 undefined) |
   at com.webauthn4j.converter.jacksonr.json.AuthenticationAlgorithmFromStringDeserializer.deserialize.deserialize(AuthenticationAlgorithmFromStringDeserializer.java:27 undefined) |
   at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:563 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:438 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:351 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:184 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.2.2]|
   at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.0]|
   at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666 undefined) ~[com.fasterxml.jackson.core.jackson-databind_2.13.2.2.jar:2.13.0]|
   at com.webauthn4j.converter.util.JsonConverter.readValue(JsonConverter.java:64 undefined) |
   at com.webauthn4j.metadata.LocalFilesMetadataStatementsProvider.lambda$provide$0(LocalFilesMetadataStatementsProvider.java:46 undefined) |
   at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193 undefined) ~[?:1.8.0_321]|
   at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499 undefined) ~[?:1.8.0_321]|
   at com.webauthn4j.metadata.LocalFilesMetadataStatementsProvider.provide(LocalFilesMetadataStatementsProvider.java:50 undefined) |
   at com.webauthn4j.metadata.anchor.MetadataStatementsBasedTrustAnchorRepository.find(MetadataStatementsBasedTrustAnchorRepository.java:47 undefined) |
   at com.webauthn4j.metadata.anchor.AggregatingTrustAnchorRepository.lambda$find$0(AggregatingTrustAnchorRepository.java:39 undefined) |
   at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267 undefined) ~[?:1.8.0_321]|
   at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708 undefined) ~[?:1.8.0_321]|
   at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234 undefined) ~[?:1.8.0_321]|
   at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499 undefined) ~[?:1.8.0_321]|
   at com.webauthn4j.metadata.anchor.AggregatingTrustAnchorRepository.find(AggregatingTrustAnchorRepository.java:40 undefined) |
   at com.webauthn4j.validator.attestation.trustworthiness.certpath.DefaultCertPathTrustworthinessValidator.validate(DefaultCertPathTrustworthinessValidator.java:65 undefined) |
   at com.webauthn4j.validator.AttestationValidator.validate(AttestationValidator.java:121 undefined) |
   at com.webauthn4j.validator.RegistrationDataValidator.validate(RegistrationDataValidator.java:192 undefined) |
   at com.webauthn4j.WebAuthnRegistrationManager.validate(WebAuthnRegistrationManager.java:204 undefined) |
   at com.webauthn4j.WebAuthnManager.validate(WebAuthnManager.java:161 undefined) |

Thanks

[ynojima]

Pull-request is sent: #657

[ynojima]

Now it passes at least MDS3 test cases: webauthn4j/webauthn4j-spring-security#899

[ThaminduDilshan]

Thanks for looking into this. Hope you'll be checking on rest of the failures as well

[ynojima]

I fixed following issue by adding this commit(webauthn4j/webauthn4j-spring-security@fa1e93e):

• P-5 Send a valid ServerAuthenticatorAttestationResponse with SELF "packed" attestation, for "ALG_SIGN_RSASSA_PKCSV15_SHA256_RAW" aka "RS256" algorithm, and check that server succeeds

Following issues are still under investigation:

• P-3 Send a valid ServerAuthenticatorAttestationResponse with FULL "packed" attestation that contains batch certificate, that is simply self referenced in the metadata, and check that server succeeds
• P-12 Send a valid ServerAuthenticatorAttestationResponse with SELF "packed" attestation, for "ALG_SIGN_ED25519_EDDSA_SHA512_RAW" aka "Ed25519" algorithm, and check that server succeeds

[ynojima]

P-12 Send a valid ServerAuthenticatorAttestationResponse with SELF "packed" attestation, for "ALG_SIGN_ED25519_EDDSA_SHA512_RAW" aka "Ed25519" algorithm, and check that server succeeds

This will be resolved by #662

[ThaminduDilshan]

Hi @ynojima

FIDO alliance has already released a new version of the conformance tool (v1.6.48).

Were you able to run the tests with pending webauthn4j library changes? It seems issues with the previous tool are fixed now.

[ynojima]

As far as I tested, TPM registration responses test case issues still persist: fido-alliance/conformance-test-tools-resources#618

[ThaminduDilshan]

Thanks for the update

[ThaminduDilshan]

Hi @ynojima

Noticed that there's a v1.7.0 release in the test tool. This has referred here as well. Have you run conformance tests against this version?
I couldn't check this locally as I'm currently not in a possession with the setup. But noticed they have taken this version out of the EXPERIMENTAL folder in FIDO resources.

[ynojima]

TPM registration responses test case issues still persists in 1.7.0.

[ThaminduDilshan]

Hi @ynojima,

I recently happened to run FIDO2 Server - MDS3 tests with conformance tool v1.7.6 and webauthn4j latest release. It looks like only the below test is failing now.

Server-ServerAuthenticatorAttestationResponse-Resp-5 Test server processing "packed" FULL attestation:
P-3 Send a valid ServerAuthenticatorAttestationResponse with FULL "packed" attestation that contains batch certificate, that is simply self referenced in the metadata, and check that server succeeds

The reason is,

NO_TRUST_ANCHOR
invalid cert path: Path does not chain with any of the trust anchors

Note
Btw I enabled the revocation check at MDS3 MetadataBLOBProvider by calling fidoMDS3MetadataBLOBProvider.setRevocationCheckEnabled(true) (code) as fido-alliance/conformance-test-tools-resources#633 is resolved now.

[ThaminduDilshan]

Moved the discussion to a new thread as a new minor tool version is available.
#819