I'm going to refuse to hold a baby
Why don't you want to hold a baby, do you not like them?
I just don't want to.
I'm sorry you feel that way. You don't have to hold other people's children if you don't want to. You can just politely explain that they are too young and you don't feel comfortable holding them.
I'm holding a cat instead.
Well that's nice! A cat can be nicer to hold for some people more than babies.