Is it selfish to want my son to not talk to his father when he's spending the night with me?
That sounds selfish. Why don't you want him to talk to his dad?
Because his dad's a fucker and I want time with my son.
Sharing custody with someone who isn't respectful towards you is a difficult situation, but you have to always think about your child's needs first. Even if you're frustrated with your child's father, it isn't in your child's best interest to cut contact with him, unless he's abusive. Try to get some mediation to improve your relationship with your child's father.
I just want his dad out of the picture.
I understand being frustrated trying to handle custody between you and your son's dad. If you try to keep his father out of his life though, your son is likely to resent you for it. Instead, just be the best parent you can and try to keep things peaceful between you and his father.