My SO will never understand how I feel physically.
That's understandable. Do you want to explain that to him?
No, I don't want to spoil our relationship
You should tell your partner the truth about how you feel. They might want to understand and sympathize with you better to get closer.
Maybe one day I'll have the guts to tell him but not right now.
But if you tell him soon then he will be able to understand you and know how you really feel. It is important to be honest with your partner if you want the relationship to succeed.