pydantic.error_wrappers.ValidationError crash on device import

[jathanism]

Environment

• Python version: 3.8.8
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Expected Behavior

Successful import from the same massive NetBox 2.10.3 dump. Related to #4.

Observed Behavior

Made it further, but crashed on a Device validation?

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 91, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 299, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 208, in create
    return super().create(diffsync, diffsync_ids, diffsync_attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 197, in create
    model = cls(**ids, diffsync=diffsync, **attrs)
  File "pydantic/main.py", line 400, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Device
status
  none is not an allowed value (type=type_error.none.not_allowed)

Steps to Reproduce

1.Receive my massive database dump "somehow"
2. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3 and then wait
3. Crash!

[ndom91]

Having this same error on a VRF for the RD field, although it can be empty / None in Netbox.

EDIT: For context

21:25:50 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=<class 'nautobot_netbox_importer.diffsync.models.ipam.VRF'> model_data={'id': UUID('1310d1a5-141b-4594-9ec6-d2819910dac4'), 'created': datetime.date(2021, 2, 28), 'last_updated': datetime.datetime(2021, 2, 28, 19, 46, 57, 326813, tzinfo=<UTC>), 'custom_field_data': {}, 'name': 'Telia-BGG', 'rd': None, 'enforce_unique': True, 'description': 'Telia Routing Instance', 'tags': [], 'tagged_items': <django.contrib.contenttypes.fields.create_generic_related_manager.<locals>.GenericRelatedObjectManager object at 0x7f9509d98be0>, 'pk': UUID('1310d1a5-141b-4594-9ec6-d2819910dac4')}
1 validation error for VRF
rd
  none is not an allowed value (type=type_error.none.not_allowed)

[glennmatthews]

Thanks for the report! I'll fix the VRF model validation as part of #6 as well.

[ndom91]

Oh I also just noticed, I had this similar error pop up many time with services and PowerOutletTemplates.

services example log message:

21:25:37 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=<class 'nautobot_netbox_importer.diffsync.models.ipam.Service'> model_data={'created': '2020-10-09', 'last_updated': '2020-10-09T12:48:45.670Z', 'custom_field_data': {}, 'device': {'site': {'name': 'Equinix WA1'}, 'tenant': {'name': 'MegaFon'}, 'name': 'Megafon_WA1_Perle', 'rack': {'group': None, 'name': '0206'}, 'position': None, 'face': '', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Perle'}, 'model': 'IOLAN SDS2'}, 'device_role': {'name': 'Console Server'}, 'platform': {'name': 'Linux'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'virtual_machine': None, 'name': 'Serial1', 'protocol': 'tcp', 'ports': '["10001"]', 'description': 'Console for re0', 'ipaddresses': [{'address': IPNetwork('123.456.789.123/32'), 'vrf': None, 'tenant': None, 'assigned_object_type': {'app_label': 'dcim', 'model': 'interface'}, 'assigned_object_id': {'device': {'site': {'name': 'Equinix WA1'}, 'tenant': {'name': 'MegaFon'}, 'name': 'Megafon_WA1_Perle', 'rack': {'group': None, 'name': '0206'}, 'position': None, 'face': '', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Perle'}, 'model': 'IOLAN SDS2'}, 'device_role': {'name': 'Console Server'}, 'platform': {'name': 'Linux'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 'eth0'}}], 'pk': 4}
1 validation error for Service
ports
  value is not a valid list (type=type_error.list)

Not sure which value is supposed to be a valid list here..

PowerOutletTemplate example log message:

21:25:17 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=<class 'nautobot_netbox_importer.diffsync.models.dcim.PowerOutletTemplate'> model_data={'device_type': {'manufacturer': {'name': 'Raritan'}, 'model': 'PX3-5497V'}, 'name': 'Outlet24', '_name': 'Outlet00000024', 'label': '', 'description': '', 'type': 'iec-60320-c19', 'power_port': {'device_type': {'manufacturer': {'name': 'Raritan'}, 'model': 'PX3-5497V'}, 'name': 'Inlet1'}, 'feed_leg': 'C', 'pk': 804}
1 validation error for PowerOutletTemplate
type
  instance of staticmethod expected (type=type_error.arbitrary_type; expected_arbitrary_type=staticmethod)

In this case I'm also not sure which value its complaining about unfortunately.

[glennmatthews]

Thank you Nico! I've opened #8 to separately track the latest issues you reported since they'll take me a bit of time to investigate.