I think I might flirt with this woman instead of taking my brother for a walk.
Are you attracted to this lady?
Sure.  And I'd love to flirt with her.
I can understand why you would want to stay and flirt with this woman, since you might not see her again otherwise.  However, it is your responsibility to look after your brother, and you can't leave him alone.  Your family is more important to you right now than a chance at romance.
Fuck, you're right. I know my brother is very important to me.
Sometimes our priorities can get muddled, and we can lose sight of what's really important. I think you made a great decision.