Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we try to add Huawei support? #102

Closed
gabrielat opened this issue Nov 7, 2023 · 67 comments
Closed

Can we try to add Huawei support? #102

gabrielat opened this issue Nov 7, 2023 · 67 comments
Labels
enhancement New feature or request

Comments

@gabrielat
Copy link

Hello, I saw that there was an issue to add Huawei support, but it didn't follow through.
Is this all you need? How can I help?


To add a new supported device, we need, at least:

Originally posted by @dainok in #2 (comment)

@dainok
Copy link
Owner

dainok commented Nov 7, 2023

Best is if you are able to write discoverers, templates, ingestors. If not provide a lab and I'll do the best. Consider sponsoring :)

@dainok dainok added the enhancement New feature or request label Nov 7, 2023
@gabrielat
Copy link
Author

The lab is created only with text files? Or did you export this files from a software?

@dainok
Copy link
Owner

dainok commented Nov 8, 2023

You can use a simulator but at the end you need a set of text and YAML files. Text files contain the output of "display" commands, YAML files contain what you expect to be ingested into Netbox.
See https://github.com/dainok/netdoc/tree/master/netdoc/tests/netmiko_cisco_xr/lab1/logs

You can prepare a physical or a virtual lab then export logs, see https://github.com/dainok/netdoc/blob/master/scripts/lab_export.py

@dainok
Copy link
Owner

dainok commented Nov 8, 2023

Obviously lab files will be for public (testing purpose).

@dainok
Copy link
Owner

dainok commented Nov 19, 2023

Still interested? Can you provide the requested files?

@gabrielat
Copy link
Author

gabrielat commented Nov 19, 2023 via email

@dainok
Copy link
Owner

dainok commented Nov 19, 2023

Ok, thank you for the feedback.

@gabrielat
Copy link
Author

gabrielat commented Nov 22, 2023

Hi @dainok !
I think I managed to do the lab. Here are the files
netmiko_huawei_vrp.zip

@dainok
Copy link
Owner

dainok commented Nov 22, 2023

Thank you @gabrielat I'm working on that in the next weeks.

@dainok
Copy link
Owner

dainok commented Nov 22, 2023

@gabrielat one question: which netmiko tye should I use?

  • Huawei
  • Huawei OLT
  • Huawei SmartAX

@gabrielat
Copy link
Author

@dainok Huawei

@dainok
Copy link
Owner

dainok commented Nov 23, 2023

@gabrielat few questions:

  1. can you confirm the command "display arp all"? (you reported "display arp" only)
  2. can you provide the output of "display ip routing-table verbose" from node1?
  3. can you provide the output of "display vlan all" from node1?

@dainok
Copy link
Owner

dainok commented Nov 23, 2023

See #112

@gabrielat
Copy link
Author

gabrielat commented Nov 23, 2023 via email

@dainok
Copy link
Owner

dainok commented Nov 26, 2023

@gabrielat

I will go with the following commands, please confirm:

display current-configuration | include sysname
display current-configuration
display device manufacture-info
display lldp neighbor
display vlan all
display ip vpn-instance
display interface
display eth-trunk
display ip interface
display ip routing-table verbose
display lldp neighbor brief
display arp all
display mac-address

No commands are required on a per vrf bassis. ARP and routing are included for all VRFs in the above commands.
After you confirm, I'll start to write and adapt NTC templates (most of them are missing or not working as expected).

@dainok
Copy link
Owner

dainok commented Nov 26, 2023

I need from your side:

  1. the output of "display ip routing-table verbose" from node1?
  2. the output of "display vlan all" from node1?
  3. the output of "display interface" with a trunk with more than one aggregated interface (add one interface Eth-Trunk0, no matter if it's down). Please provide also "display eth-trunks"

@gabrielat
Copy link
Author

@gabrielat

I will go with the following commands, please confirm:

display current-configuration | include sysname display current-configuration display device manufacture-info display lldp neighbor display vlan all display ip vpn-instance display interface display eth-trunk display ip interface display ip routing-table verbose display lldp neighbor brief display arp all display mac-address

No commands are required on a per vrf bassis. ARP and routing are included for all VRFs in the above commands. After you confirm, I'll start to write and adapt NTC templates (most of them are missing or not working as expected).

Hi!
All commands listed above are ok, except "display vlan all". This is what Huawei displays on vlan:
display vlan ?
INTEGER<1-4094> VLAN ID
pool VLAN pool
summary Summary information of VLAN
vlan-name Specify VLAN name
| Matching output

@gabrielat
Copy link
Author

3. display interface

I've disassembled the lab I did before, can I sent you the results for a real switch?

display ip routing-table verbose:
Route Flags: R - relay, D - download to fib, T - to vpn-instance

Routing Tables: Public
Destinations : 363 Routes : 363

Destination: 0.0.0.0/0
Protocol: O_ASE Process ID: 1
Preference: 150 Cost: 1
NextHop: 10.10.0.1 Neighbour: 0.0.0.0
State: Active Adv Age: 30d22h50m33s
Tag: 0 Priority: high
Label: NULL QoSInfo: 0x0
IndirectID: 0x0
RelayNextHop: 0.0.0.0 Interface: Vlanif10
TunnelID: 0x0 Flags: D

Destination: 10.2.8.0/21
Protocol: OSPF Process ID: 19200
Preference: 10 Cost: 81
NextHop: 10.10.0.1 Neighbour: 0.0.0.0
State: Active Adv Age: 30d22h50m33s
Tag: 0 Priority: medium
Label: NULL QoSInfo: 0x0
IndirectID: 0x0
RelayNextHop: 0.0.0.0 Interface: Vlanif10
TunnelID: 0x0 Flags: D

 ...

display interfacve with eth-trunk:
Eth-Trunk2 current state : UP
Line protocol current state : UP
Description:link - core-saude
Switch Port, Link-type : trunk(configured),
PVID : 1, Hash arithmetic : According to SIP-XOR-DIP,Maximal BW: 40G, Current BW: 40G, The Maximum Frame Length is 9216
IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 6413-ab94-59a0
Current system time: 2023-11-27 08:18:42-03:00
Last 300 seconds input rate 380298448 bits/sec, 38240 packets/sec
Last 300 seconds output rate 64235528 bits/sec, 20093 packets/sec
Input: 48825888039 packets, 52742246838697 bytes
Unicast: 47966460072, Multicast: 795402201
Broadcast: 64025766, Jumbo: 21818227555
Discard: 0, Pause: 0
Frames: 0

Total Error: 0
CRC: 0, Giants: 0
Jabbers: 0, Fragments: 0
Runts: 0, DropEvents: 0
Alignments: 0, Symbols: 0
Ignoreds: 0

Output: 29386300826 packets, 21021916296143 bytes
Unicast: 29088080133, Multicast: 206571011
Broadcast: 91649683, Jumbo: 9291051991
Discard: 0, Pause: 0

Total Error: 0
Collisions: 0, ExcessiveCollisions: 0
Late Collisions: 0, Deferreds: 0
Buffers Purged: 0

Input bandwidth utilization  : 0.95%
Output bandwidth utilization : 0.16%

PortName Status Weight

40GE0/0/1 UP 1

The Number of Ports in Trunk : 1
The Number of UP Ports in Trunk : 1

LoopBack0 current state : UP
Line protocol current state : UP (spoofing)
Description:lo0-vrf default
Route Port,The Maximum Transmit Unit is 1500
Internet Address is 10.10.0.13/32
Physical is Loopback
Current system time: 2023-11-27 08:18:53-03:00
Last 300 seconds input rate 0 bits/sec, 0 packets/sec
Last 300 seconds output rate 0 bits/sec, 0 packets/sec
Realtime 0 seconds input rate 0 bits/sec, 0 packets/sec
Realtime 0 seconds output rate 0 bits/sec, 0 packets/sec
Input: 0 packets,0 bytes
0 unicast,0 broadcast,0 multicast
0 errors,0 unknownprotocol
Output:0 packets,0 bytes
0 unicast,0 broadcast,0 multicast
0 errors
Input bandwidth utilization : 0%
Output bandwidth utilization : 0%

@dainok
Copy link
Owner

dainok commented Nov 27, 2023

Sure, please send in txt/raw format as a separated file.
About VLANs: the command is "display vlan", and the output is the one you sent me in the past days?

The total number of VLANs is: 3
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports
--------------------------------------------------------------------------------
1    common  UT:40GE0/0/1(D)    40GE0/0/2(D)    40GE0/0/3(D)    40GE0/0/4(D)
                40GE0/0/5(D)    40GE0/0/6(D)    XGE0/0/1(D)     XGE0/0/3(D)

@gabrielat
Copy link
Author

Sure, please send in txt/raw format as a separated file. About VLANs: the command is "display vlan", and the output is the one you sent me in the past days?

The total number of VLANs is: 3
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports
--------------------------------------------------------------------------------
1    common  UT:40GE0/0/1(D)    40GE0/0/2(D)    40GE0/0/3(D)    40GE0/0/4(D)
                40GE0/0/5(D)    40GE0/0/6(D)    XGE0/0/1(D)     XGE0/0/3(D)

Yes, that's it

@gabrielat
Copy link
Author

I need from your side:

  1. the output of "display ip routing-table verbose" from node1?
  2. the output of "display vlan all" from node1?
  3. the output of "display interface" with a trunk with more than one aggregated interface (add one interface Eth-Trunk0, no matter if it's down). Please provide also "display eth-trunks"

Here are the files ypu've asked for :-)

display interface.txt
display ip routing-table verbose.txt

@dainok
Copy link
Owner

dainok commented Nov 29, 2023

@gabrielat parsing display vlan is very complex: I'm not able to parse tagged and untagged interfaces.
Any chance I can get which VLAN is passing and how on each interface? (display int or display vlan detail?

@gabrielat
Copy link
Author

@gabrielat parsing display vlan is very complex: I'm not able to parse tagged and untagged interfaces. Any chance I can get which VLAN is passing and how on each interface? (display int or display vlan detail?

Hi! I've looked the commands available but nothing seems to work properly..
Huawei doesn't have 'display vlan detail' and doesn't show the tagged and untagged vlans on 'display interface'.

From 'display vlan' I could see that tagged ports are preceeded with TG and untagged with UT.
For instance:

<rout> display vlan 234
--------------------------------------------------------------------------------
U: Up;         D: Down;         TG: Tagged;         UT: Untagged;
MP: Vlan-mapping;               ST: Vlan-stacking;
#: ProtocolTransparent-vlan;    *: Management-vlan;
--------------------------------------------------------------------------------

VID  Type    Ports                                                          
--------------------------------------------------------------------------------
234  common  UT:XGE1/0/4(U)     XGE1/0/5(U)                                     
             TG:XGE1/0/24(D)                                                    

VID  Status  Property      MAC-LRN Statistics Description      
--------------------------------------------------------------------------------
234  enable  default       enable  disable    vlan234           

Note that in section
VID Type Ports
There is

234  common  UT:XGE1/0/4(U)     XGE1/0/5(U)                                     
             TG:XGE1/0/24(D)                 

Saying that VLAN 234 is UNTAGGED on ports XGE1/0/4 and on XGE1/0/5, both in Up state, and also TAGGED on por XGE1/0/24 in Down state.

As we can see in:

<rout>display current-configuration interface XGigabitEthernet 1/0/4
#
interface XGigabitEthernet1/0/4
 port link-type access
 port default vlan 234
#
return
<rout>display current-configuration interface XGigabitEthernet 1/0/5
#
interface XGigabitEthernet1/0/5
 port link-type access
 port default vlan 234
#
return
<rout-cs-dc-saude-01>display current-configuration interface XGigabitEthernet 1/0/24
#
interface XGigabitEthernet1/0/24
 port link-type trunk
 port trunk allow-pass vlan 100 234 1823
#
return

I think it's not ideal, but this is the best way I could find..

@dainok
Copy link
Owner

dainok commented Nov 29, 2023 via email

@dainok
Copy link
Owner

dainok commented Dec 1, 2023

@gabrielat seems I successfully parsed VLANs.
Now VRFs (VPN in Huawei language).
Where I can see which VRF is associated to each interface?

@dainok
Copy link
Owner

dainok commented Dec 2, 2023

@gabrielat another question:
in display mac / arp / route, how are displayed 40GE and Trunk interfaces?
Can you post a short snippet?

@dainok
Copy link
Owner

dainok commented Dec 3, 2023

@gabrielat I completed most of the required tasks. I need:

  • a VRF example: where I can find VRF - interface association? I display interface / display ip route / display arp with a VRF
  • in display mac / arp / route, how are displayed 40GE and Trunk interfaces? Can you post a short snippet?

@dainok
Copy link
Owner

dainok commented Dec 4, 2023

@gabrielat based on templates, you should have this command: display port vlan
Please send me the output from ROUTER1.

@gabrielat
Copy link
Author

@gabrielat based on templates, you should have this command: display port vlan Please send me the output from ROUTER1.

here is the display port vlan return
display port vlan.txt

@dainok
Copy link
Owner

dainok commented Dec 5, 2023

@gabrielat ok go with tests.
You can experiment errors on discover phase, I couldn't test it on a lab.

@gabrielat
Copy link
Author

Hi @dainok ! Sorry, but I don't know how to add these on my installation. Can you tell me what I need to do? Thanks

@dainok
Copy link
Owner

dainok commented Dec 5, 2023

Follow https://github.com/dainok/netdoc/wiki/Developing-NetDoc
But on NTC template, use my url and my branch instead of the official one.

@gabrielat
Copy link
Author

gabrielat commented Dec 6, 2023

@dainok trying to run for the first time, got this error:

An exception occurred: `ValidationError: ['display current-configuration | include sysname', 'HOSTNAME'] is not of type 'string'

Failed validating 'type' in schema['properties']['command']: {'type': 'string'}

On instance['command']: ['display current-configuration | include sysname', 'HOSTNAME']`

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 264, in run
    output = discovery(
  File "/root/src/netdoc/netdoc/tasks.py", line 118, in discovery
    module.discovery(filtered_devices, filters=filters, filter_type=filter_type)
  File "/root/src/netdoc/netdoc/discoverers/netmiko_huawei_vrp.py", line 65, in discovery
    discoverylog.create(
  File "/root/src/netdoc/netdoc/schemas/discoverylog.py", line 93, in create
    validate(kwargs, get_schema_create(), format_checker=FormatChecker())
  File "/opt/netbox/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: ['display current-configuration | include sysname', 'HOSTNAME'] is not of type 'string'

Failed validating 'type' in schema['properties']['command']:
    {'type': 'string'}

On instance['command']:
    ['display current-configuration | include sysname', 'HOSTNAME']

@dainok
Copy link
Owner

dainok commented Dec 6, 2023

@gabrielat fixed (hopefully). can you update netdoc and test again?

@gabrielat
Copy link
Author

gabrielat commented Dec 6, 2023

@gabrielat fixed (hopefully). can you update netdoc and test again?

@dainok it's running now

@dainok
Copy link
Owner

dainok commented Dec 6, 2023

@gabrielat everything is working? any error?

@gabrielat
Copy link
Author

@dainok the discovery ran ok, the ingest returned an error due to IPv6:

An exception occurred: `ValidationError: '2804:1f20::20' is not a 'ipv4'

Failed validating 'format' in schema['properties']['address']: {'format': 'ipv4', 'type': 'string'}

On instance['address']: '2804:1f20::20'`

Traceback (most recent call last):
  File "/opt/netbox/netbox/extras/scripts.py", line 504, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox/netbox/scripts/netdoc_scripts.py", line 357, in run
    log_ingest(log)
  File "/root/src/netdoc/netdoc/utils.py", line 399, in log_ingest
    module.ingest(log)
  File "/root/src/netdoc/netdoc/ingestors/netmiko_huawei_vrp_display_lldp_neighbor.py", line 86, in ingest
    remote_discoverable_list += discoverable.get_list(
  File "/root/src/netdoc/netdoc/schemas/discoverable.py", line 91, in get_list
    validate(kwargs, get_schema(), format_checker=FormatChecker())
  File "/opt/netbox/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: '2804:1f20::20' is not a 'ipv4'

Failed validating 'format' in schema['properties']['address']:
    {'format': 'ipv4', 'type': 'string'}

On instance['address']:
    '2804:1f20::20'

@dainok
Copy link
Owner

dainok commented Dec 6, 2023

@gabrielat I see you are using IPv6. I'm not implemented it yet in NetDoc. IPv4 can be fine for now?

@gabrielat
Copy link
Author

@gabrielat I see you are using IPv6. I'm not implemented it yet in NetDoc. IPv4 can be fine for now?

Yes, but I ran netdoc on IPv4 addresses..

Running task 'multiple_tasks' with args {} on 7 hosts
Host '143.54.0.7': running task 'multiple_tasks'
Host '143.54.0.7': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.12': running task 'multiple_tasks'
Host '143.54.0.12': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.13': running task 'multiple_tasks'
Host '143.54.0.13': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.17': running task 'multiple_tasks'
Host '143.54.0.17': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.22': running task 'multiple_tasks'
Host '143.54.0.22': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.23': running task 'multiple_tasks'
Host '143.54.0.23': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.24': running task 'multiple_tasks'
Host '143.54.0.24': running task '{"command": "display current-configuration | include sysname", "template": "HOSTNAME", "enable": true, "order": 0}'
Host '143.54.0.7': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.12': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.17': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.13': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.23': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.22': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.24': running task '{"command": "display current-configuration", "template": "display current-configuration", "enable": true, "order": 1}'
Host '143.54.0.17': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.17': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.12': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.17': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.17': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.12': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.23': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.12': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.24': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.23': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.24': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.24': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.24': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.22': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.7': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.22': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.13': running task '{"command": "display device manufacture-info", "template": "display device manufacture-info", "enable": true, "order": 2}'
Host '143.54.0.7': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.22': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.23': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display lldp neighbor", "template": "display lldp neighbor", "enable": true, "order": 3}'
Host '143.54.0.7': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.12': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.17': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display ip vpn-instance", "template": "display ip vpn-instance", "enable": true, "order": 4}'
Host '143.54.0.22': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.12': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.7': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.23': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.17': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.23': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.17': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.24': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.17': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.24': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.13': running task '{"command": "display interface", "template": "display interface", "enable": true, "order": 5}'
Host '143.54.0.17': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.24': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.12': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.17': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.12': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.23': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.23': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.12': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.17': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.12': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.24': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.24': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.24': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.23': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.24': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.23': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.22': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.22': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.22': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.22': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.7': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.22': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.7': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.22': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.7': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.22': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.7': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.13': running task '{"command": "display ip vpn-instance interface", "template": "display ip vpn-instance interface", "enable": true, "order": 6}'
Host '143.54.0.13': running task '{"command": "display eth-trunk", "template": "display eth-trunk", "enable": true, "order": 7}'
Host '143.54.0.13': running task '{"command": "display ip interface", "template": "display ip interface", "enable": true, "order": 8}'
Host '143.54.0.7': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.7': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.7': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Host '143.54.0.13': running task '{"command": "display vlan all", "template": "display vlan all", "enable": true, "order": 9}'
Host '143.54.0.13': running task '{"command": "display lldp neighbor brief", "template": "display lldp neighbor brief", "enable": true, "order": 10}'
Host '143.54.0.13': running task '{"command": "display arp all", "template": "display arp all", "enable": true, "order": 11}'
Host '143.54.0.13': running task '{"command": "display mac-address", "template": "display mac-address", "enable": true, "order": 12}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.7': running task 'additional_tasks'
Host '143.54.0.7': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.7': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.12': running task 'additional_tasks'
Host '143.54.0.12': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.12': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.13': running task 'additional_tasks'
Host '143.54.0.13': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.13': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.17': running task 'additional_tasks'
Host '143.54.0.17': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.17': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.22': running task 'additional_tasks'
Host '143.54.0.22': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.22': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.23': running task 'additional_tasks'
Host '143.54.0.23': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.23': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Running task 'additional_tasks' with args {} on 1 hosts
Host '143.54.0.24': running task 'additional_tasks'
Host '143.54.0.24': running task '{"command": "display ip routing-table verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance cftv verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance met verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'
Host '143.54.0.24': running task '{"command": "display ip routing-table vpn-instance tel verbose", "template": "display ip routing-table verbose", "enable": true, "order": 100}'

@dainok
Copy link
Owner

dainok commented Dec 6, 2023

ok, let me think how I can skip IPv6 for now.
In the meanwhile you can ingest all logs from CLI using a netbox shell and https://github.com/dainok/netdoc/blob/master/scripts/log_ingest.py

@dainok
Copy link
Owner

dainok commented Dec 6, 2023

@gabrielat update netdoc and try again

@gabrielat
Copy link
Author

ok, let me think how I can skip IPv6 for now. In the meanwhile you can ingest all logs from CLI using a netbox shell and https://github.com/dainok/netdoc/blob/master/scripts/log_ingest.py

there were only 2 logs that weren't ingested due to ipv6 addresses (both from lldp).

@gabrielat
Copy link
Author

@gabrielat update netdoc and try again

I'll do that now

@gabrielat
Copy link
Author

gabrielat commented Dec 6, 2023

@gabrielat update netdoc and try again

1 ingest was ok, but then:

image

Increase timeout and run again?

@dainok
Copy link
Owner

dainok commented Dec 6, 2023 via email

@gabrielat
Copy link
Author

Seems that the VM cannot reach API to lookup vendor MAC address. That VM can reach the Internet? Does it have any restrictions? Sent from Proton Mail for iOS

On Wed, Dec 6, 2023 at 18:55, Gabriela Todeschini @.(mailto:On Wed, Dec 6, 2023 at 18:55, Gabriela Todeschini < wrote: > @.(https://github.com/gabrielat) update netdoc and try again 1 ingest was ok, but then: image.png (view on web) — Reply to this email directly, [view it on GitHub](#102 (comment)), or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Yes, it has access and no restrictions..

@dainok
Copy link
Owner

dainok commented Dec 6, 2023 via email

@dainok
Copy link
Owner

dainok commented Dec 7, 2023

Another question: how many mac addresses do you have in log 394?

@gabrielat
Copy link
Author

Another question: how many mac addresses do you have in log 394?

Hi, good morning!
This mac-addresses logs are big..

394.json

@gabrielat
Copy link
Author

Can you export the log and send it to me by email? Sent from Proton Mail for iOS

On Wed, Dec 6, 2023 at 19:15, Gabriela Todeschini @.(mailto:On Wed, Dec 6, 2023 at 19:15, Gabriela Todeschini < wrote: > Seems that the VM cannot reach API to lookup vendor MAC address. That VM can reach the Internet? Does it have any restrictions? Sent from Proton Mail for iOS > > On Wed, Dec 6, 2023 at 18:55, Gabriela Todeschini @.(mailto:On Wed, Dec 6, 2023 at 18:55, Gabriela Todeschini <wrote: > @.)(https://github.com/gabrielat) update netdoc and try again 1 ingest was ok, but then: image.png (view on web) — Reply to this email directly, [view it on GitHub]([#102 (comment)](#102 (comment))), or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> Yes, it has access and no restrictions.. — Reply to this email directly, [view it on GitHub](#102 (comment)), or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

do you still need it by email?

@dainok
Copy link
Owner

dainok commented Dec 11, 2023

No it's fine, thank you for the log.

@dainok
Copy link
Owner

dainok commented Dec 13, 2023

@gabrielat I tested on my lab and I'm ingesting 2k MAC addresses in a couple of minutes. So I cannot tell if you are triggering a bug or it is caused by a lack of resources.

Could you try to increase timeout?

And... could you skip/delete (for now) the log, so we can check if you are triggering other bugs?

@gabrielat
Copy link
Author

@gabrielat I tested on my lab and I'm ingesting 2k MAC addresses in a couple of minutes. So I cannot tell if you are triggering a bug or it is caused by a lack of resources.

Could you try to increase timeout?

And... could you skip/delete (for now) the log, so we can check if you are triggering other bugs?

Hi! Thank you for the reply.
I'll do that and I'll check the VM's resources and increase them also.

@dainok
Copy link
Owner

dainok commented Dec 18, 2023

@gabrielat can you confirm parsing is ok so I can send update to 3rd party repositories?

@gabrielat
Copy link
Author

hi @dainok still getting the timeout message, even with 3600s timeout config and 22gb of ram. this is the result messsage and log

image

830.json

@dainok
Copy link
Owner

dainok commented Dec 21, 2023

@gabrielat beside this, other logs are ingesting fine? Could you please delete this one and go with others?
I need to publish changes as soon as possible.

@gabrielat
Copy link
Author

gabrielat commented Dec 21, 2023 via email

@dainok
Copy link
Owner

dainok commented Dec 26, 2023

@gabrielat both ntc-templates and netdoc are merged to master.

Regarding your issue, can you try to manually ingest the huge log? (check https://github.com/dainok/netdoc/blob/master/scripts/log_ingest.py the script must be used in a NetBox/Django shell)

@dainok dainok closed this as completed Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants