SSH Key Rejection

[harningt]

I have many SSH keys and the step of putting IdentitiesOnly in the .ssh/config is entirely do-able in most situations, but feels like it might help reduce friction if there were an option to have SSH keys coming in checked against the known set of keys and rejected if there were no users with a given key.
It doesn't look like it'd be that hard to implement (at the SSH server layer - don't allow through any key coming in, tell the server to go to the next auth method)... but I don't want to try implementing something if this is a design issue.