I'm not going to let my boyfriend take our son too far away from me to his parents or other places
Is there a reason why he has the child?
I can't take care of the child when I feel depressed for long periods.
Depression can be a tough think to deal with.   That should be all the more of a reason to let your boyfriend help with the child.  Creating conflict and not allowing him could make you more depressed and create and upsetting scenario for both him and the child.
I can't let him take our child. I don't want to fight with him over this.
I thought you guys were still together? He doesn't have full custody so you are still entitled to see your child. It is illegal for him to withhold the kid without a court order.