I don't want to hang with my brother anymore
Is there something going on between you and your brother?
I can't handle the pain of him being late all the time
I get that being on time is very important, but it is equally hurtful to him for you to just cut him off like that. You two just need to compromise your differences and problems so you can continue hanging out.
Yes, I'm going tell him that it's okay to be late and then we can maintain the relationship.
Things happens all the time. We just can't be perfect and be on time for every plans we make. It's just life.