Am I wrong to refuse giving up a seat to the latecomer who had reserved it?
Does saving a seat not entitle them to it?
No, I want to sit down and there's no other seats. There's no discussing this.
You should respect that the person had already reserved the seat. There may have been mitigating circumstances that made them late, and your desire to sit doesn't override that.
I'm responsible for myself, not for others. I paid for that seat.
I don't think you really paid for that particular seat, since someone else had it reserved. Reserving a seat in a potentially crowed situation is a great idea, so you don't have these issues in the future. For now the proper thing to do is give the seat to the person it is reserved for.