Traceback: diffsync.exceptions.ObjectAlreadyExists: Object None__<object>__ already present

[ggiesen]

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Expected Behavior

Import data from NetBox 2.10.7

Observed Behavior

...
14:00:00 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=virtualchassis "None__switch1__" model_id=None__switch1__
...
14:01:50 Data loading from Nautobot complete.
14:01:50 Beginning data synchronization...
14:02:15 Nautobot reported a data validation error - check your source data action=create model=<class 'nautobot.dcim.models.devices.Device'> model_data={'site': <Site: Site>, 'tenant': <Tenant: Tenant>, 'name': 'switch1b', 'rack': <Rack: B2.2 (B2.2)>, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': <DeviceType: N4032>, 'device_role': <DeviceRole: VPS Switch>, 'platform': <Platform: Dell NOS 6>, 'serial': '', 'asset_tag': None, 'cluster': None, 'custom_field_data': {'management_tool': 'Salt'}, 'status': 
<Status: Active>, 'local_context_data': None, 'virtual_chassis': None, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}
{'position': ['U36 is already occupied or does not have sufficient space to accommodate this device type: N4032 (1U)']}
14:02:15 device create did not return the model object. action=create diffs={'+': {'custom_field_data': {'management_tool': 'Salt'}, 'status': {'slug': 'active'}, 'local_context_data': None, 'virtual_chassis': {'master': {'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'switch1b', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}, 'device_role': {'name': 'VPS Switch'}, 'platform': {'name': 'Dell NOS 6'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 'switch1', 'domain': ''}, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> status=failure unique_id={'name': 'Site'}__{'name': 'Tenant'}__switch1b__{'group': {'site': 
{'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}__36__front__2__None__{'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}__{'name': 'VPS Switch'}__{'name': 'Dell NOS 6'}____None__None
14:02:15 No object resulted from sync, will not process child objects. action=create diffs={'+': {'custom_field_data': {'management_tool': 'Salt'}, 'status': {'slug': 'active'}, 'local_context_data': None, 'virtual_chassis': {'master': {'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'switch1b', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}, 'device_role': {'name': 'VPS Switch'}, 'platform': {'name': 'Dell NOS 6'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 
'switch1', 'domain': ''}, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> unique_id={'name': 'Site'}__{'name': 'Tenant'}__switch1b__{'group': {'site': 
{'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}__36__front__2__None__{'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}__{'name': 'VPS Switch'}__{'name': 'Dell NOS 6'}____None__None
Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/cli.py", line 63, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 92, in handle 
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 314, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 293, in sync_diff_element
    self.dst_diffsync.add(model)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 230, in add
    super().add(obj)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/__init__.py", line 623, in add
    raise ObjectAlreadyExists(f"Object {uid} already present")
diffsync.exceptions.ObjectAlreadyExists: Object None__switch1__ already present

Steps to Reproduce

1. $ python netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report > /tmp/netbox_data.json
2. $ nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.5

[ggiesen]

After dropping and recreating the nautobot DB, and re-running the import (with the exact same data file):

...
14:37:40 Data loading from Nautobot complete.
14:37:40 Beginning data synchronization...
14:41:53 Summary of changes             summary={'create': 21196, 'update': 14, 'delete': 0, 'no-change': 273}
14:41:53 First-pass synchronization complete, beginning second pass
Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/cli.py", line 63, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 95, in handle 
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 314, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 333, in sync_model
    model = model.update(attrs=attrs)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 290, in update
    record = self.update_nautobot_record(self.nautobot_model(), nautobot_ids, nautobot_attrs, multivalue_attrs)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 247, in update_nautobot_record
    record = nautobot_model.objects.get(**ids)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/lib64/python3.6/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/db/models/query.py", line 431, in get
    self.model._meta.object_name
nautobot.dcim.models.devices.DoesNotExist: Device matching query does not exist.

[glennmatthews]

Thanks for the report! The DoesNotExist error itself should be fixed (or at least handled gracefully without a crash) in the latest develop code base (see c170156) -- if you'd be able and willing to retest with the latest from GitHub, that would be a big help.

[ggiesen]

I can confirm with the latest develop (9f94637) that the traceback no longer occurs

[glennmatthews]

I believe this issue to have been fixed (most likely by #28) in the latest version of this plugin. If you're still encountering it, feel free to comment and I'll re-open this issue for further investigation. Thanks!