Primary IP of device lost if adding new IP to different interface

[herr-mhet]

Environment

Python version: 3.9.2
Nautobot version: 1.2.9

Steps to Reproduce

1. Add new device
2. Add multiple (virtual) interfaces
3. Add IP to one interface, set this as primary
4. Add IP to another interface
5. Observe that the device has lost its primary IP

Expected Behavior

Primary IP relationship is contained regardless of IP settings of other interfaces on the same device.

Observed Behavior

Device has lost the primary IP.

More environment information

I did migrate from netbox, so this might affect this problem (haven't checked on a clean install). Problem occurred on new devices, on new interfaces, on new IPs in new IP prefixes.

[glennmatthews]

I suspect this is related to #1262 in some way.