Equipped one on a test world and when I loaded my survival world the familiar still appeared floating around me. My Vis stone was still equipped, it didn't attack anything, and it was easily fixed by going back to my other world and removing it so It wasn't really that bad but F3+T didn't affect it so I brought the issue here. Though this does have interesting potential in cross world interactions even if it's accidental.