Use pattern matching in Constraints to avoid unecessary casts

[stevenaw]

Sibling to #4065
Use pattern matching in everything under Constraints namespace to avoid unnecessary casts or comparisons to give a minor perf boost.