crypto/x509: return informative error if wrong type passed to MarshalPKIXPublicKey

[mathieudevos]

What version of Go are you using (go version)?

$ go version
go1.12.5

Does this issue reproduce with the latest release?

Not applicable

What did you do?

What did you expect to see?

Expected to see an error stating it requires a pointer, or for it to resolve instead

What did you see instead?

panic: x509: only RSA and ECDSA public keys supported

My suggestion, in https://golang.org/src/crypto/x509/x509.go?s=3023:3081#L68 , the function marshalPublicKey add 2 cases with simple recursion:

case rsa.PublicKey:
    return marshalPublicKey(&pub)
case ecdsa.PublicKey
    return marshalPublicKey(&pub)

Or same as above, but return error that states that it must be a pointer.

[FiloSottile]

It's already confusing enough without letting some functions take the non-pointer type, but we should return a better error.

[mathieudevos]

It's already confusing enough without letting some functions take the non-pointer type, but we should return a better error.

Actually, the more I read about this, the more I'm confused, there are already Marshaling functions for public keys from RSA (see: MarshalPKCS1PublicKey). But the ECDSA one is overloaded.

Can we just add a function called MarshalECPublicKey (there is MarshalECPrivateKey) which throws an error if you give in a non-pointer type.

It seems that MarshalPKIXPublicKey is overloaded with both RSA and ECDSA public keys, while there is already a func to marshal RSA public keys.

[FiloSottile]

That's because RSA keys have an RSA-specific format, PKCS#1, while PKIX is a format that can encode both RSA and ECDSA. That is also a mess, but it's its own mess that has little to do with pointer types, and it's due to what's specified, not what's implemented in Go.

Leaving this issue to be about the better errors, see #32541 (comment) if you want to open a proposal for an EC-only encoding function.

[mathieudevos]

Proposed to mark this as deprecated and rewrite it with the proposed new correctly-typed functions.

Proposal can be found here: #32657

Issue with increasing verbosity in this function is that we would need to use reflection to figure out what the user is actually sending us. That is way outside the scope of this function. The proposal aims to increase type-safety and usability.

Not sure which way we go with this, but if we go with the proposal, this can be closed.

[FiloSottile]

No need for reflection, just a type assertion will do.

[mathieudevos]

In that case to make this slightly more verbose, I would propose the following:

- return nil, pkix.AlgorithmIdentifier{}, errors.New("x509: only RSA and ECDSA public keys supported")
+ return nil, pkix.AlgorithmIdentifier{}, fmt.Errorf("x509: unsupported type: %T, only RSA and ECDSA public key supported", pub)

This does not solve the issue that non-pointer values will cause confusion, as such I would either write custom error functions for them or use recursion to handle them.