I don't want to turn off my stream when my friends aren't comfortable talking due to me having an audience.
Why do you want to keep your stream on although your friends are uncomfortable with it?
I like putting on a show to my viewers
I understand wanting to establish yourself on a streaming platform, but its rude to stream others if they aren't comfortable with it. They may find you to be a rude host and could stop wanting to spend time with you.
Would they ditch me if I let them not play and just talk while I play?
You should encourage your friends to come over when you are not streaming. This will help you to maintain your friendships and your tendency to stream.