I don't care about my client anymore
Why don't you care about your client anymore?
My client is not very nice, he had fun without me, and I am fed up with his actions.
You are not expected to stay with a client as a caregiver if it's not working for you.  You should start looking for a new client before you quit, one that you get along with better so that both of you will have a more pleasant experience.  You should keep caring about your clients for now until you get a new one, or it might effect your future as a caregiver.
Nah, I'm not going to see my client anymore.
It's good to set boundaries like that. If the client you are working with is mean to you, then you don't have to continue seeing him. It might be good to bring up how his actions affect you, but as long as there isn't a contract, you don't need to stay with them.