Testcase enumeration

[woodruffw]

Not exhaustive, obviously.

This issue tracks a few "baseline" testcases we should include as part of an MVP.

They're categorized roughly below.

Path construction cases

• SUCCESS: Happy-path cases for various combinations of (root, intermediate..., EE)
• SUCCESS: Happy-path cases for pathlen constraints
• SUCCESS: Self-issued certificates are handled correctly in pathlen constraints (aren't counted against the constraint)
• FAILURE: Violating pathlen constraints (both exterior and interior to the certs)
• FAILURE: Unhandled critical extensions (e.g. SCT poison)
• SUCCESS: Successful SAN matching of various forms
  • Exact match (foo.example.com)
  • Left-most wildcard (foo.example.com matches *.example.com)
  • Wildcard in left-most position (foo.example.com matches f*o.example.com)
• FAILURE: Failed SAN matching of various forms (no exact, no wildcard)
• FAILURE: SAN matching error states
  • 6125: Wildcard not in left-most position (e.g. foo.*.example.com is invalid)
  • 6125: Wildcard should not match across the left-most label (e.g. foo.bar.example.com should not match *.example.com)
  • 6125: Wildcard in left-most position. but embedded inside an A-label or U-label of an internationalized domain label (e.g. xn--blah*.example.com is invalid)
  • 5280: SAN dNSNames must be ASCII only

Certificate state cases

• FAILURE: Ensure that various APIs are invariant preserving
  • 5280: CAs must contain an AuthorityKeyIdentifier unless self-signed
  • 5280: CAs must provide AuthorityKeyIdentifier.keyIdentifier unless self-signed
  • 5280: CAs must not assert AuthorityKeyIdentifier.critical
  • 5280: CAs must contain a SubjectKeyIdentifier
  • 5280: CAs must not assert SubjectKeyIdentifier.critical
  • 5280: CAs must contain a non-empty DN for their Issuer (Initial RFC 5280 test cases #7)
  • etc.
  • CA/B: EEs must have AuthorityInformationAccess and it must be well-formed

---

Reference material:

• RFC 5280: https://datatracker.ietf.org/doc/html/rfc5280
• RFC 6125: SAN and other identity matching in a PKIX/TLS context: https://www.rfc-editor.org/rfc/rfc6125.html
• CA/B Forum BRs: https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-v2.0.0.pdf

[woodruffw]

xrefs:

• crypto/x509: check the Key Usage extension golang/go#40100
• Deviations from RFC 5280 rustls/webpki#70

[woodruffw]

We may also be able to glean and adapt some testcases from https://github.com/google/x509test.

Edit: and from bettertls: https://github.com/Netflix/bettertls

https://netflixtechblog.com/revisiting-bettertls-certificate-path-building-4c978b79843f

Edit: Done for bettertls.

[woodruffw]

We probably also want a "chain of pain" testcase, i.e. where the server sends an expired intermediate but a valid path still exists (since chain building doesn't require us to consult the intermediates).

Refs:

• https://medium.com/@sleevi_/path-building-vs-path-verifying-the-chain-of-pain-9fbab861d7d6
• https://www.agwa.name/blog/post/fixing_the_addtrust_root_expiration

Edit: Done.

[woodruffw]

On the pathlen side, we also probably want a few testcases for self-issued (not self-signed) certificates: self-issued CA certificates aren't counted in path length constraints, meaning that the following is valid:

root -> ICA' (pathlen:1) -> ICA' (pathlen:1) -> ICA'' (pathlen:0) -> EE

(where ICA' is a single logical CA with two different CA certificates)

Edit: Done.

[woodruffw]

...and a testcase for "CAs" that aren't marked as such, e.g.

root -> ICA -> EE

...where ICA issued EE but is not marked as a CA.

Edit: Done.

[woodruffw]

Some more CA/B cases we'll want to cover:

• Root CAs MUST NOT contain an extKeyUsage
• Root CAs MAY have an AKI, but: (Add AKI testcases for root certs #48)
  • MUST NOT have authorityKeyIdentifier.authorityCertIssuer
  • MUST NOT have authorityKeyIdentifier.authorityCertSerialNumber
• All certs must have key types and margins explicitly permitted under the profile (e.g. no RSA under 2048)

[woodruffw]

Maybe use these as well: https://badssl.com/

https://github.com/chromium/badssl.com

Edit: Successor-ish project: https://github.com/wbond/badtls.io

Edit 2: wbond/badtls.io#1 is a blocker for integrating the badtls suite.

[woodruffw]

Potential weird FP case with OpenSSL here: #44 (comment)

[woodruffw]

From a quick look at pyca/cryptography#8873, some other cases we still need:

• Duplicate X.509v3 extensions (we may have to hack this together) (limbo: add duplicate_extensions test #65)
• Oversized serial number/empty serial number in certificate (RFC5280: add serial number cases #60)
• Completely nonsense extension contents for a well-known extension type (e.g. a subjectAltName that contains a raw domain name as ASCII bytes, rather than a DER-encoded GeneralName) (rfc5280: add testcase with malformed subjectAltName #73 (comment))

(cc @facutuesca)

[woodruffw]

Some non-MVP testcases:

• WebPKI validators should reject EC AlgorithmIdentifiers that aren't in namedCurve form (e.g. SECP256R1 with explicit parameters)

[woodruffw]

Some more "pedantic" WebPKI testcases:

• 7.1.4.3: If present, Subject.commonName MUST contain exactly one entry that is one of the values contained in the subjectAltName extension, and MUST be encoded as follows
  • For IPv4 addresses, must be an IPv4Address per RFC 3986 S. 3.2.2
  • For IPv6 addresses, must be be encoded in the text representation specified in RFC 5952 S. 4.
  • For FQDNs or wildcard domain names, must be a char-for-char copy of the dNSName entry from subjectAltName; P-labels must not be converted to their Unicode representation.
• 7.1.2.7.6 and 7.1.2.7.10: extKeyUsage is required in subscriber certificates, and MUST contain id-kp-serverAuth (MAY contain id-kp-clientAuth), and MUST NOT contain any other id-kp-*, anyExtendedKeyUsage, or the Precertificate Signing Certificate OID (1.3.6.1.4.1.11129.2.4.4)
• 7.1.2.10.6: CA EKUs are similar to subscriber cert EKUs

[woodruffw]

Another interesting testcase, similar to webpki::v1-cert: v1 certs cannot contain v3 extensions, so a certificate that's marked as v1 but has extensions should fail. This should look almost identical to the current v1-cert case, except without the no_extensions=True kwarg.

[woodruffw]

Pointed out in pyca/cryptography#8873 (comment): we don't have any IPv6 testcases yet.

Edit: Done: #95

[woodruffw]

Some more "pedantic" NC cases: both 5280 and CABF say that GeneralSubtree.minimum MUST be 0 and GeneralSubtree.maximum MUST be NULL.

[woodruffw]

Re: #1 (comment), some notes on integrating BetterTLS:

Export the entire test suite:

git clone https://github.com/Netflix/bettertls && cd bettertls
go build -o bettertls ./test-suite/cmd/bettertls
./bettertls export-tests > /tmp/bettertls.json

...which then needs to be mashed into Limbo testcase format.

Edit: Done.

[woodruffw]

Closed in favor of #174.