I don't want to spend time with my youngest brother
Why's that? Do you have a problem with him?
Yes indeed, with his body odor
Why not tell him, then? If you think he stinks, other people probably do as well. Why not help him out?
He should just focus on improving himself and making sure he doesn't smell bad. There doesn't need to be a reason to spend time with family.
I still think you should tell him. I can't force you to spend time with him, but be considerate and let him know he stinks.