Anyone using vjunos-switch with netlab?

[vsurresh]

Netlab version - 26.05

Hi, anyone using vjunos-switch with netlab? When I try it, netlab up fails after 400 seconds, but the box actually takes about 15 minutes to boot, and I can SSH into it fine once it's up.

I can then run netlab initial manually, but it fails on the "Wait for first interface (ge-0/0/0)" task

TASK [Wait for first interface (ge-0/0/0)] *******************************************************************************************************************************************************************************************************************
ok: [leaf2]
fatal: [leaf1]: FAILED! => {"changed": false, "failed_conditions": ["result[0] contains ge-0/0/0"], "msg": "One or more conditional statements have not been satisfied"}

When I SSH in and check manually, ge-0/0/0 isn't there at all. The generated clab.yml file shows eth1 as the first data-plane interface for both nodes, which matches what the containerlab vjunos-switch docs say (eth1 >> ge-0/0/0), so I'm not sure why the interface isn't showing up on the device itself.

Topology:

provider: clab
defaults.device: vjunos-switch
defaults.devices.vjunos-switch.clab.image: juniper_vjunos-switch:25.4R1.12
addressing:
    mgmt:
        ipv4: 192.168.202.0/24
    p2p:
        ipv4: 172.16.1.0/24
        prefix: 31
nodes: [leaf1, leaf2]
links:
    - leaf1-leaf2

Has anyone hit this? Is the 400s SSH-wait timeout just not bumped for vjunos-switch, and is there a known reason ge-0/0/0 doesn't show up?

admin@leaf1> show interfaces terse 
Interface               Admin Link Proto    Local                 Remote
cbp0                    up    up
demux0                  up    up
demux1                  up    up
dsc                     up    up
em1                     up    up
em1.0                   up    up   inet     10.0.0.4/8      
                                            128.0.0.1/2     
                                            128.0.0.4/2     
                                   inet6    fe80::5254:ff:fe12:bdfe/64
                                            fec0::a:0:0:4/64
                                   tnp      0x4             
esi                     up    up
fti0                    up    up
fxp0                    up    up
fxp0.0                  up    up   inet     10.0.0.15/24    
                                   inet6    2001:db8::2/64  
                                            fe80::e00:e3ff:fee7:2c00/64
gre                     up    up
ipip                    up    up
irb                     up    up
jsrv                    up    up
jsrv.1                  up    up   inet     128.0.0.127/2   
lo0                     up    up
lo0.16384               up    up   inet     127.0.0.1           --> 0/0
lo0.16385               up    up   inet    
lsi                     up    up
mif                     up    up
mtun                    up    up
pimd                    up    up
pime                    up    up
pip0                    up    up
pp0                     up    up
rbeb                    up    up
tap                     up    up
vtep                    up    up

Thanks

[ssasso]

Using it almost every day.

Are you using netlab in a VM or on a bare metal host?

vjunos has issues with (nested) virtualization, so you need to run it on bare metal machines.

[vsurresh]

@ssasso Thank you for the response. I'm using netlab on top of Proxmox and the VM is set to 'host' mode. I can run vjunos-router without any issues. Even with vjunos-switch it boots fine and I can SSH to the CLI. Which version of vjunos-switch are you using?

[ssasso]

@ssasso Thank you for the response. I'm using netlab on top of Proxmox and the VM is set to 'host' mode. I can run vjunos-router without any issues. Even with vjunos-switch it boots fine and I can SSH to the CLI. Which version of vjunos-switch are you using?

I'm using 25.2R1.9. The fact that it boots (in 15 min...) and you are able to SSH to the control plane does not mean it fully works. you stated that interfaces are not appearing, meaning that most probably something between the VCP and the VFP is not communicating correctly.

Ref:

• https://www.juniper.net/documentation/us/en/software/vjunos/vjunos-switch-kvm/topics/vjunos-switch-architecture-concept.html
• https://www.juniper.net/documentation/us/en/software/vjunos/vjunos-switch-kvm/topics/vjunos-switch-overview-understanding.html

the latter explicitly states Due to its nested architecture, the vJunos-switch cannot be used in any deployments that launch the instances from within a VM.

it would be great if you can try it on a bare metal host.

[vsurresh]

ah that makes sense, thank you. If you don't mind me asking, what type of CPU do you use?

[ipspace]

ah that makes sense, thank you. If you don't mind me asking, what type of CPU do you use?

I have two "bricks", one with Intel CPU (the Intel NUC, now long discontinued), the other with AMD CPU (Minisforum UM890 Pro). Both work fine with Junos.

[ipspace]

One other thing: I tried to run vJunos-something on a VM a long time ago, and it was completely useless due to the long boot times (what you're experiencing). One of the reasons was the storage access (over the network). It looks like Junos does a ton of disk read/writes, and if you're not using local SSDs (or NVMe), it is incredibly slow.

I can then run netlab initial manually, but it fails on the "Wait for first interface (ge-0/0/0)" task

Did you try to wait another 15 minutes? It would be nice to know whether the interface never appears (= something is truly broken) or it just takes a long, long time.

On a more serious note, that's useless. It's best to run Junos on a bare-metal server with SSDs.

[vsurresh]

I'm running netlab on Ubuntu on top of Proxmox with local SSDs. I didn't wait the full 15 minutes after I was able to SSH, maybe around 5, but I'll give it another go. From the docs and feedback from others, vjunos-switch needs to run on bare-metal, but everything I have runs on Proxmox at the moment. I might need to buy new hardware or repurpose one of my used machines to run bare-metal. On your Minisforum, how many vjunos-switch instances can you run?

[ipspace]

I'm running netlab on Ubuntu on top of Proxmox with local SSDs.

Ah, so it's really the nested virtualization (or maybe Proxmox?). My Proxmox VM was running on networked storage (IIRC).

repurpose one of my used machines to run bare-metal.

That's usually very error-prone and time-consuming task. I tried it a few times, and now I always opt for a new machine (after all, time is somewhat related to money)

On your Minisforum, how many vjunos-switch instances can you run?

Looks like the answer is 7 (vrnetlab VM, release 25.4R1.12), and it's memory-bound. CPU is also maxed out. It took the nodes 170 seconds to have the SSH server ready, versus ~90 seconds for one instance (which seems to burn two CPUs continuously, even after the device is idle).

Once you run out of memory (or swapping starts), some instances just die.

[vsurresh]

I did some more testing over the weekend. The VM running on Proxmox has an Intel i9, 8 CPUs, and 32GB RAM allocated to the VM. vjunos-switch eventually comes up with data plane interfaces but takes about 30 minutes for two nodes. With 4+ nodes, they don't come up even after 30+ minutes, so I gave up. It does work on the VM though.

I then tested the image on a Minisforum MS-A2 (Baremetal Ubuntu 22.04) with AMD Ryzen 9 9955HX and 32GB RAM. I can run 6 nodes with SSH coming up after about 6-7 minutes. The netlab ssh test fails at first, but once I wait for all nodes to have SSH and then wait a few more minutes for data plane interfaces, netlab initial works fine.

[INFO]    Checking SSH server(s) on spine1,spine2,leaf1,leaf2,leaf3,leaf4
[SSH]     SSH server on node leaf3 (device vjunos-switch) is ready after 404.5 seconds
[SSH]     SSH server on node leaf4 (device vjunos-switch) is ready after 406.3 seconds
[SSH]     SSH server on node leaf2 (device vjunos-switch) is ready after 407.9 seconds
[SSH]     SSH server on node spine2 (device vjunos-switch) is not ready after 409.4 seconds
[SSH]     SSH server on node leaf1 (device vjunos-switch) is not ready after 409.4 seconds
[SSH]     SSH server on node spine1 (device vjunos-switch) is not ready after 409.4 seconds
[FATAL]   initial: SSH server did not start in time on node(s) spine1,spine2,leaf1
[FATAL]   netlab up: netlab initial failed, aborting...

suresh@netlab-04:~/network-labs/v2$ docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED         STATUS                   PORTS                                                            NAMES
a5f52d8434d4   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-leaf1
0822a784066d   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-spine1
6c0b00bca664   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-leaf3
8c7c0eac34f4   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-spine2
a9a27a55c7bd   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-leaf2
392bd28dade4   vrnetlab/juniper_vjunos-switch:25.2R1.9   "uv run /launch.py -…"   7 minutes ago   Up 7 minutes (healthy)   22/tcp, 830/tcp, 5000/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-v2-leaf4
suresh@netlab-04:~/network-labs/v2$

Here is the usage for running 6 nodes. I think 6 nodes is the limit with 32 GB of RAM, if I upgrade to 64 GB (which costs a fortune now, thank you AI), I think I can push it to like 8-10 nodes.

[ipspace]

The VM running on Proxmox has an Intel i9, 8 CPUs, and 32GB RAM allocated to the VM. vjunos-switch eventually comes up with data plane interfaces but takes about 30 minutes for two nodes. It does work on the VM though.

It looks like something emulates the nested virtualization needed to run vJunos VMs inside the Proxmox VM, and that takes a lot of CPU cycles.

I then tested the image on a Minisforum MS-A2 (Baremetal Ubuntu 22.04) with AMD Ryzen 9 9955HX and 32GB RAM. I can run 6 nodes with SSH coming up after about 6-7 minutes.

I managed to run seven nodes in 64GB of RAM, so I'd say running 6 nodes within 32GB is definitely triggering swapping, and that's why it takes so long for devices to boot. I'd use free -h command to see how much RAM is free/available, and once that is close to zero, you're about to hit long start times.

The netlab ssh test fails at first, but once I wait for all nodes to have SSH and then wait a few more minutes for data plane interfaces, netlab initial works fine.

This is how you can extend the wait times: https://netlab.tools/netlab/initial/#wait-for-devices-to-become-ready

Hope this helps,
Ivan

[vsurresh]

Thank you. I was able to extend the wait times using the link; all good now, thanks. I will close the discussion.