Can I take my daughter away from my brother when I move out?
Why do you want your daughter away from your brother?
Because he is a bad influence
If a family member is being toxic it is expected that you wouldn't want your kids to be around them. You don't want your brother to lead your daughter down the same path as him one day.
She will be fine, they're only related by blood after all.
I still think that if you believe that your brother is a bad influence on your daughter, then it is smart to keep him away. You have my full support in this.