I'm wearing a hat my ex gave me
Do you think there is a good reason for wearing it still?
I like how it fits me
As long as it does not cause you any stress to wear something your ex gave you then it is fine. If you feel it causes you to focus attention on your ex, which stresses you out, then it may be a good idea to discard the hat.
My ex wants me to wear it
Well that doesn't seem like a good reason. For you to move on it may not be smart to be focused on what your ex wants.