I'm not going to give up my chair at a packed cafeteria table just to allow a group to sit together.
Does the reason have to do with the group of people wanting to sit at the table?
I don't think they are being fair asking me to move.
Maybe it's not fair to ask you to move but it would be fair of you to offer so the group can sit together.
By the logic that what I'm doing is wrong, why don't all the other people get up too? They just don't want to be asked.
You're not wrong to feel the way you do but imagine if you were with a group. Wouldn't you want someone to move if they could so your group could sit together?