Refactor CacheCondition to remove multiple casts

[philwebb]

We can only use the condition on types so we can assert and cast just once.