Can I ban a guy off a server?
Why do you want to ban this guy?
I don't want to listen to his bullshit anymore.
It's wrong to ban someone because you don't like listening to them. Other people may find them pleasant to interact with and may look on you badly if you ban this person.
It's my server, I can do whatever I want.
Some people really enjoy being a part of a server so you should not ban them unless you really must.