Port trunk VLAN issue

[AurelienCapou]

Hello,
First of all I would like to point out that I am able to install an Ubuntu OS on my servers using an "PXE" workstation (DNSMASQ for DHCP + TFTP (for ipxe.efi file) and an NGINX server to transfer the kernel, the cloud init and the ISO ubuntu). I am writing here because I encountered a small problem during the installation process that I managed to get around but which does not suit me. Here is my problem:
My server manages to retrieve the IPXE firmware by TFTP and is assigned an IP address once the VLAN has been added via the "vcreate --tag 660 net0" command. Unfortunately I cannot connect my WS to continue the boot process ("autoboot net0-660" command). The ping doesn't even work anyway.

The network configuration is quite simple: my workstation is connected to a access port on the switch (VLAN 660), my DMZ server to a trunk port (VLAN 660 tagging frames).

I have to force the PVID 660 onto the switch port on the DMZ server side so that it can join the PXE server and finish installing correctly.
It is this modification that I would not like to make because I have specified the addition of the VLAN (vcreate) but it seems that it is not taken into account when making the requests.

I built the latest firmware from github and I only enabled the VCREATE and PING options in ipxe-master/src/config/general.h. I also enabled VLAN option in NIC interface from BIOS and my broadcom ethernet card have the latest driver.

Do you have any ideas to resolve this issue? Can I force my net0-660 interface into command ping and autoboot?

[NiKiZe]

So dhcp succeeds on the vlan without pvid, but all other traffic fails?

You get timeout issues on boot attempts as well? Hopefully the ipxe.org you see in the message should provide some guidence, but that might only be if dhcp fails.

What is the output of ifstat after any type of communication failure? (Looking for which driver is used as well as TX, RX, TXE, RXE)

If you have any management on this NIC the firmware might do weird things.

[AurelienCapou]

Yes DHCP succeeds because i set "Virtual LAN ID" on my NIC from the BIOS interface.
It works also after setting the VLAN in the firmware with vcreate but the boot attempt fail. I rebuild the ipxe.efi file with with the debug option DEBUG=tcp like mentionned in https://ipxe.org/err/4c0a60 :

[mcb30]

The network configuration is quite simple: my workstation is connected to a access port on the switch (VLAN 660)

If the workstation is connected to an access port, then the workstation should not be configured to set up any VLAN. The whole point of an access port is that it looks like a normal port with no VLAN tags: the tags are added (and removed) invisibly by the switch when packets are forwarded between that access port and any trunk ports with the same VLAN tag enabled.

[AurelienCapou]

I do not have configured any VLAN setting for my workstation because as you said, it is connected to an access port. I have a network problem on my server which are connected to a port trunk so I need to specify VLAN on my bios and on iPXE firmware but it's not working as explain in my previous comment.
My workstation have dnsmasq to send iPXE firmware to my servers.

[mcb30]

You are using words in a very confusing way: "server" would normally mean the thing serving up DHCP / DNS / TFTP / HTTP / etc. Having re-read what you wrote, you are attempting to boot your "server" from your "workstation" (which is actually acting as the boot server). 🙃

With that clarified: your problem turns out to be visible in the first screenshot, which shows:

iPXE> route
net0: 200.1.66.14/255.255.255.128 gw 200.1.66.10
...
net0-660: 200.1.66.14/255.255.255.128 gw 200.1.66.10

i.e. you are attempting to use the same IPv4 subnet on both the default (untagged) VLAN and on VLAN 660. This is not a valid configuration, since they are logically separate networks. When you attempt to ping the address 200.1.66.10, it is undefined whether that address is in net0: 200.1.66.14/255.255.255.128 or net0-660: 200.1.66.14/255.255.255.128, because they are the same IPv4 network!

You need to use a different IPv4 subnet on your VLAN 660.

[AurelienCapou]

I'm sorry I wasn't clear from the start. I want my boot server (WS) and my Dell PowerEdge servers to be in the same VLAN (660) and the same network (200.1.66.0/25) because they will talk together after the boot installation process.
The problem you point is that my net0 interface obtain also an IP address whereas i specify net0-660 in the dhcp command (first screenshot).
I already tried to delete net0 in order to have only net0-660 but the command ifclose net0 also delete net0-660.
Also, the command autoboot net0-660 is not successfull.
The vcreate command seemed easy to use but it's not...
Is it possible to create a fake VLAN interface, then update manually the ip address for net0 and after that create the good VLAN 660 ? In order to have different subnet like you said.
Thank you for your help.

[mcb30]

You should not be ending up with 200.1.66.0/25 assigned to your trunk (net0) interface. The IPv4 settings for net0 will be being inherited from the settings provided to the UEFI BIOS network stack that you are using to load iPXE.

Either you have something providing the same 200.1.66.0/25 subnet over the trunk interface, or something is causing iPXE to associate the UEFI BIOS network stack settings with the trunk device instead of the VLAN device.

Try building with DEBUG=cachedhcp,efi_cachedhcp and report what output you get from iPXE before you use the vcreate or dhcp commands.

[AurelienCapou]

Hello, i build with DEBUG=cachedhcp,efi_cachedhcp,tcp, here is the output for my PowerEdge with the option PVID 660 set to the port trunk interface of the switch (ping and autoboot work) :

Here are outputs for my PowerEdge without the option PVID in the port trunk interface of the switch (autoboot doesn't work) :

Is it normal to not have efi_cachedhcp output log ?

[mcb30]

Is it normal to not have efi_cachedhcp output log ?

Yes, efi_cachedhcp prints debug output only if an error occurs.

The debug output:

CACHEDHCP dhcp is for net0

shows that the DHCP packet obtained by the UEFI PXE stack is being incorrectly associated with net0 (the trunk interface) instead of being saved for later association with VLAN 660.

The VLAN is identified via the loaded image's device path. Could you please try building with DEBUG=efi_init, which will print out a line EFI image device path ... to show the path that iPXE will be parsing in order to locate the VLAN tag?

[AurelienCapou]

Here is the output with only efi_init DEBUG option :

Here is how I start up in PXE mode : F11 = Boot Manager > UEFI Boot Menu > Embedded NIC1 Port1

[mcb30]

Thank you. The device path is missing the VLAN component: i.e. your UEFI BIOS is incorrectly providing the device path for the trunk device rather than for VLAN 660. This is a bug in your UEFI BIOS.

You can work around this UEFI BIOS bug by explicitly clearing the IPv4 address on net0:

clear net0.dhcp/ip

That should delete the IPv4 routing through net0, which you can check using the route command.

[AurelienCapou]

Well done your command worked ! I can now access the menu generated by my boot.ipxe file. This file is quite simple and it works if i set PVID in the switch (like mentionned in my first post) :
#!ipxe
set ip 200.1.66.10
set url http://${ip}
set path ${url}/os-images/ubuntu-22.04
menu Select an OS to boot
item ubuntu-22.04 Installation automatique d'Ubuntu Server 22.04 LTS
choose --default exit --timeout 3000 option
:ubuntu-22.04
kernel ${path}/vmlinuz
initrd ${path}/initrd
imgargs vmlinuz initrd=initrd root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=${path}/ubuntu-live-server.iso autoinstall ds=nocloud-net;s=${url}/init/
boot
So the boot process starts well :

Unfortunately, it fails after because it can no longer receive a response from my DHCP server (DNSMASQ)...

Now I think the problem is no longer related to Ipxe but to network interface management, as the port is still tagged with VLAN 660. So the initial problem is solved, thank you again.
I'm still open to advice on how to solve the new one ;)

[mcb30]

You'll have to dig in to the Ubuntu documentation (or look at what Ubuntu builds into its initramfs images) to fix the next issue. The relevant code is all in shell script, so should be easily searchable, at least.