It sucks, but I really don't want my boyfriend to spend the day with his ex for their sons birthday.
That's pretty bad, do you have a problem with him spending time with his son?
Only because I can't be around his ex.
Despite him not being with the mother any longer, I imagine your boyfriend wants to see his son. I wouldn't worry about it, honestly.
I don't need you to tell me what to do.
While it could trigger feelings of jealousy and anger knowing that your boyfriend is with his ex, you need to remember that he is there for his sons benefit and not his ex's. Trying to keep them apart is unhealthy and controlling. It may be time to sit down with him and his ex and share your feelings.