I'm going to refuse to work with this client
Why don't you want to work with them?
I tried to work with him but I found that his attitude was very irritating.
Even if they are hard to work with you need more people to grow your business. I think you should try dealing when them and hope they get better.
I think it's better that I just refuse to work with them.
You may feel like that's the best choice, but you have an express duty to work with all clients no matter their idiosyncrasies. You could get fired if you refuse to deal with them.