I'm going to move this guys seat away from me.
Why would you move somebody's seat?
I want him off my back.
You should avoid trouble, and move yourself. Less to think about and easier to do so. Also you may upset them.
It's my damn business and I'll do what I want.
Except that it isn't just your business, its his as well. He might have placed his seat there because he likes the view. If you take that away from him there might be an altercation.