I don't want to let my roommate wear my shirt
Why do you have a problem with your roommate wearing your shirt?
I'm trying to preserve it, so I don't want him touching it
Your roommates should be respectful of your property, unless you have explicitly told them that they are free to borrow clothes. If your roommate thought it was okay to borrow your shirt, you should forgive them, but let them know that you want to keep this one unworn.
It's my shirt and I want it for myself.
That's understandable. Next time make it absolutely clear that you do not want him wearing your shirts.