Can I go ahead and refuse to give my ex half of the security deposit?
Why don't you want to give him your half of the security deposit?
He was a very difficult tenant to live with.
Even though he was not a good tenant, it is the responsible thing to give him half the security deposit. That deposit is not excused if the person is a bad tenant and he can sue for it.
Whatever, I just want to cause him legal trouble.
It won't just be his legal troubles, it will be yours also. You should pay back others what you owe them. You want to have a good renter's history so you can get another place on your own.