Ignore old custom field data

[jakubkrysl]

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.1

We used to have a custom field asw_vulnerabilityscan assigned to IP Addresses. It is no longer assigned to this model, but still exists and is assigned to other models. But our DB contains this field for 1036 of such IPs. Editing any of these is not possible as it crashes on the field being set.

AFAICT is it great to keep this data as long as the field exists since it is very easy to missclick and remove the field from some model by accident. But form validation should ignore these.

Steps to Reproduce

1. Create custom field
2. Assign it to some object
3. Create instance of such object
4. Fill in the custom field
5. Unassign the custom field from the object
6. Try to edit the created instance

Expected Behavior

No issues found.

Observed Behavior

Unable to edit, old custom field data breaks for validation.

[jakubkrysl]

I'm sorry, the reproducer might be incomplete. During process of fixing the data I found out assigning and unassigning the custom field starts RQ task nautobot.extras.tasks.provision_field, which takes quite some time. It timed out, which might be why there are some IPs with that custom field still set in the first place. Nonetheless form validation should not takes this data into account as user has no control over sending them.

[jathanism]

Hmmm... This is interesting. So you're saying that the worker was unable to complete the task and it resulted in this bizarre behavior?

[david-kn]

Hmmm... This is interesting. So you're saying that the worker was unable to complete the task and it resulted in this bizarre behavior?

Exactly (let me join this conversation as a @Thetacz colleague :-) ) - it is impossible to edit such an object (record) and this error is shown by nautobot.

[jathanism]

Okay thanks. Need to try to reproduce this and I'll follow up!

[lampwins]

So there are realistically two bugs here.

1. The custom field background tasks are not being run in an atomic transaction. This is an oversight on my part.
2. The job times out on larger data sets. This one will be easier to sort out in 1.1 with Replace RQ with Celery as queuing backend #223 which is already underway.

Additionally, we should probably create a command to normalize custom field data in the event one of these jobs fails. Muck like the trace paths command.