After I found this should I yell at my husband and leave him?
Why do you want to yell at him just for having a budget?
Because I want to let him have it for being so secretive about his money.
You husband shouldn't be hiding significant amounts of money from you, so I see why you're upset. Try talking to him first, and explaining how hurtful it is. Maybe he'll see why he needs to be more open with you about money.
I will yell at him then.
That's good, you should find out why he is finding money. That would be the smart thing to do.