!!Beta Testing Needed!! Please help us out!

[danner26]

Update: This beta has been officially released. Please see (release v2.0.0)[https://github.com/netbox-community/Device-Type-Library-Import/releases/tag/v2.0.0-2023-03-24] for more information. Also, thank you to everyone who assisted with this releases testing!

Hello all,

Tonight I have released the first iteration of the new codebase for this repository. I have completely rewritten the application, optimizing function calls and loops as much as possible, moved logging to its own callable section with verbosity options, abstracted the function calls into 3 separate classes, and more. This is a huge overhaul, and I have done my due diligence before releasing the beta. Now I need your help!

Please go ahead and clone the new branch pre-release-newbase which contains the current up to date copy of the new baseline. Please test it out, import devices and modules and assist me with finding any regressions or bugs.

I will keep this beta open for 2 weeks, so on March 23, 2023 we will be merging this to Master unless we have too many bugs/regressions to fix before. As of today, March 9th, 2023 we are on schedule for release.

Thank you for all who contribute!

[danner26]

Calling all past contributors, please if you can test out the new beta release before it is merged to master. This is a big change to the codebase so we need as many beta testers as possible. Thank you!

@dmcken
@ndom91
@mtinberg
@k0rventen
@dalrrard
@FloEisen
@gaby
@p-rintz
@rmundel
@suom1
@rsp2k
@eN0Rm
@lordwolf2004
@FlorianHeigl
@mayafox
@minitriga

[dmcken]

Glad to hear it, I'll test in the morning.

[danner26]

Glad to hear it, I'll test in the morning.

Thank you David!

[p-rintz]

Some notes:

/opt/Netbox-Device-Type-Library-Import/repo.py:78: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if folder.casefold() is not "testing":

You will need to handle the case where you run into the second except in the create functions in the netbox_api.py (ie. add a 'continue' after line 92 in netbox_api.py, but the same would likely need to happen elsewhere in the api, though I havent run into it). Right now it will run into the following issue where the api object was never created successfully:

{"weight":["A valid number is required."]}
Traceback (most recent call last):
  File "/opt/Netbox-Device-Type-Library-Import/nb-dt-import.py", line 53, in <module>
    main()
  File "/opt/Netbox-Device-Type-Library-Import/nb-dt-import.py", line 25, in main
    netbox.create_device_types(device_types)
  File "/opt/Netbox-Device-Type-Library-Import/netbox_api.py", line 95, in create_device_types
    self.device_types.create_interfaces(device_type["interfaces"], dt.id)
UnboundLocalError: local variable 'dt' referenced before assignment

That said, setting weight seems to be broken.

Netbox API seems to have changed:

weight | number($decimal)title: Weightx-nullable: true
-- | --
weight_unit | stringtitle: Weight unitEnum:

Instead of :

 - value: 31
   unit: kg

The capitalization of the Vendor 'neousys' will throw the following error (error is obvious, no idea how you have changed this case before, where the name changes, but the slug stays the same):
Error during manufacturer creation. - [{"slug":["manufacturer with this slug already exists."]},{"slug":["manufacturer with this slug already exists."]},{}]

(same for 'D-link')

Some others have changed their name, but also changed slug: ie. TrippLite (keeping the old manufacturers in the netbox db)

Similar issue with device types where name has changed, but slug hasnt:

{'manufacturer': {'name': 'Cisco', 'slug': 'cisco'}, 'model': 'Catalyst 9130AXI-B', 'slug': 'c9130axi-b', 'part_number': 'C9130AXI-B', 'u_height': 0, 'is_full_depth': False, 'interfaces': [{'name': 'WiFi', 'type': 'ieee802.11ax'}, {'name': 'GigabitEthernet0', 'type': '5gbase-t'}], 'console-ports': [{'name': 'console', 'type': 'rj-45'}, {'name': 'usb', 'type': 'usb-mini-b'}]}
The request failed with code 400 Bad Request: {'__all__': ['Device type with this Manufacturer and Slug already exists.']}

The following fails as well, as the parent subdevice_role was added after the device was created. And its not being updated.
Error '[{"__all__":["Subdevice role of device type (RS1619xs+) must be set to \"parent\" to allow device bays."]}]' creating Device Bay

This is better situated in the DeviceTypeLibrary itself, but the following wasnt correctly migrated to the new Name/Slug:
https://github.com/netbox-community/devicetype-library/blob/master/module-types/TrippLite/SNMPWEBCARD.yaml

[p-rintz]

I created a PR in the DL itself for the SNMPWEBCARD issue and here for the other two small issues we discussed.

[danner26]

Awesome, merged! I am waiting for the demo to be reset to continue the testing

[danner26]

Perfect that worked! I will start updating weight in the devicetype-library

[danner26]

@p-rintz take a look at this pr

It fixes the weight issue

[danner26]

Sweet, I just tested the changes and now all of the weight objects that were failing previously are working! Thank you for your assist on that

[dmcken]

I've done a complete import against a new instance of 3.4.5 and at least from a numbers and spot check it looks to have imported fine:

Package Installed https://github.com/netbox-community/devicetype-library.git
103 Vendors Found
2208 Device-Types Found
Modules Enabled. Creating Modules...
30 Module Vendors Found
287 Module-Types Found
Error '{"__all__":["Module type with this Manufacturer and Model already exists."]}' creating module type: {'manufacturer': {'name': 'Allied Telesis', 'slug': 'allied-telesis'}, 'model': 'SBx81GS24a', 'part_number': 'AT-SBx81GS24a', 'interfaces': [{'name': 'port1.{module}.1', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.2', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.3', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.4', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.5', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.6', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.7', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.8', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.9', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.10', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.11', 'type': '1000base-x-sfp'}, {'name': 'port1.{module}.12', 'type': '1000base-x-sfp'}]}
---
2208 devices created
56507 interfaces/ports updated
1 manufacturers created

real	52m2.124s
user	1m49.930s
sys	0m11.973s

You can ignore the error, I've already raised a PR 1209 to remove that module.

[danner26]

Awesome! Thank you for your assistance :)

[dmcken]

Redone locally on my home PC on NVME storage (behind virtualbox) on netbox 3.4.6. The PR was accepted, so error is gone.

Package devicetype-library is already installed, updating /home/dmcken/Device-Type-Library-Import/repo
103 Vendors Found
2210 Device-Types Found
Modules Enabled. Creating Modules...
30 Module Vendors Found
286 Module-Types Found
---
2210 devices created
56518 interfaces/ports updated
104 manufacturers created

real    10m42.798s
user    0m41.650s
sys     0m2.123s

I've looked through some of the device types I've added for Juniper and Mikrotik to the library and they are correct.

A 2nd run (the device-types have already been added), comes back with the following:

Package devicetype-library is already installed, updating /home/dmcken/Device-Type-Library-Import/repo
103 Vendors Found
2210 Device-Types Found
Modules Enabled. Creating Modules...
30 Module Vendors Found
286 Module-Types Found
---
0 devices created
0 interfaces/ports updated
0 manufacturers created

real    2m49.525s
user    0m27.376s
sys     0m0.818s

I guess if its easy to add I would put a line to display the number of module-types added (I know I noticed when I was performing my updates but I forget why I didn't add it to that pull request).

[danner26]

Sounds like a good addition that is probably actually a regression from the original.. I might have missed that part. If you could, please create an issue so that I can track the addition of the modules created line