UEFI Aquantia 10Gb/s - Marvell AQtion 10Gbit - Driver aqnic650sys

[aarnaud]

Hi,

I have issue to use iPXE with 10Gb/s network adapter. The motherboard have 2 NIC 1Gb/s(work fine with iPXE) and 10Gb/s (that failed)

Aquantia 10Gb/s = Marvel AQtion 10Gbit = Driver aqnic650sys

I tried to update network adapter firmware, update BIOS too, I still have issue.
I tried ipxe.efi, snponly.efi and snp.efi still not working even with 1.21 and master build

It seem a similar issue than:
https://forums.fogproject.org/topic/14797/dell-precision-3930-rack-efi-pxe-issue/15

lspci -nn

02:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)

after typing ifconf

without netdevice debug:

I have a set-up to test and debug, so let me know how I can help.

[NiKiZe]

Try to simplify your setup by disabling all NICs except for the one with issues, make sure you can chain on that nic.

Also include full pciids for the device (lspci -nn)

[aarnaud]

Try to simplify your setup by disabling all NICs except for the one with issues, make sure you can chain on that nic.

already tested

Also include full pciids for the device (lspci -nn)

I don't have physical access so I will to find a way to boot on linux

[NiKiZe]

Try to simplify your setup by disabling all NICs except for the one with issues, make sure you can chain on that nic.

already tested

Also include full pciids for the device (lspci -nn)

I don't have physical access so I will to find a way to boot on linux

Your screenshots only show multiple NICs.

If you have it working via other NICs you should be able to run Linux and lspci that way? Not super important at this stage tho, just good to have.

[aarnaud]

@NiKiZe lspci -nn added in description, thanks.

[aarnaud]

Hi @mcb30 and @NiKiZe do you have any idea for this issue ?

[NiKiZe]

I was waiting for new pictures and missed that they where updated.
You could try to figure out which error is the main one, the iPXE.org error URLs are helpful. If the output is in text I could help with that.

At the same time you should try to reach out to the manufacturer, also make sure to update firmware if you haven't already. The reason being that it is not unlikely that the issue is with the NICs NII driver, rather than with iPXE.

[aarnaud]

All error that can I have on screen:

• http://ipxe.org/1a086194 => Error: Unknown from net/netdevice.c
• http://ipxe.org/28086090 => Error: Network unreachable from net/netdevice.c

after ifconf

• http://ipxe.org/2a086089 => Error: No buffer space available from net/netdevice.c
• http://ipxe.org/380f6093 => Error: The socket is not connected from net/udp.c
• http://ipxe.org/440e6083 => Error: Protocol not supported from net/tcpip.c
• http://ipxe.org/3c086083 => Error: Operation not supported from net/netdevice.c

Also I have answer with arping but not with ping

firmware is up to date for network card and bios.

Also, PXE boot is working with WDS pxe smsboot\x64\wdsmgfw.efi but not with iPXE (ipxe.efi or snp.efi or snponly.efi)

Is there some other test I can do ?

[aarnaud]

Answer from Marvell Support Mon, 22 Mar 2021

Hello Anthony,

Thank you for contacting Marvell Global Technical Support.

Support for Aquantia products is only available from whom the product was directly purchased.

If you are looking for Drivers or Firmware:
    https://www.marvell.com/support/downloads.html
If you are a Developer or Reseller and have a Signed NDA:  
    https://www.marvell.com/support/extranets.html
If you are looking for an NDA or to contact Marvell Sales:
    https://www.marvell.com/request-information.html
If you have any questions or comments for us, don't hesitate to ask.  We are always glad to help.

Best Regards,
Marvell Global Support Team

[aarnaud]

I wondering if it's a manufacturer driver issue or not because PXE boot is working with WDS (windows) pxe smsboot\x64\wdsmgfw.efi but not with iPXE (ipxe.efi or snp.uefi or snponly.efi)

[aarnaud]

Dell support:

Unfortunately, with PXE booting not being something supported, there's not a way to get anyone to test beyond confirming the hardware is working.
Since PXE is not something that is supported, there's not really a way to open an issue on it. We'd confirm the hardware is working as it should and that it boots fine through USB, both of which have been confirmed which shows the hardware is working as it should be.

Is there someone that can help me to debug the issue, and confirm or not is the issue is on iPXE or on Network card firmware ?

[NiKiZe]

Have you tried different speeds (connect to gig or 10 gig device and see if it makes any difference)
Also try to tcpdump or wireshark the traffic, try to use port mirroring on the switch unless you have a direct connection to another machine.

after the last chain, do an ifstat again to see what you have in packet counts

[aarnaud]

hi @NiKiZe
Thanks for your time,

• I tried with 1Gb/s port or 10Gb/s on switch, it's the same result
• I will work to get a pcap.
• I can see packet counts, see screenshot

[NiKiZe]

What settings for MTU do you have on the switch, server/router and in iPXE?

[aarnaud]

What settings for MTU do you have on the switch, server/router and in iPXE?

Default MTU, 1500 but Jumbo Frame is enabled on 10Gb/s switch. But when I tried to test on 1Gb switch without Jumbo Frame the result still same.

Also, ping failed but arping work

[aarnaud]

Hi,
today we reach the same issue with a Lenovo P620 that use the same network card Marvell AQtion 10Gbit Network Adapter
I don't known how I can debug that. I used ipxe 1.21.

[aarnaud]

I may have a clue:
If i try arping, I received 2 answer one the first time and nothing more
But if use ifclose, the arping receive response for all request.
It's like the BIOS didn't release the network stack after booting to ipxe.

[aarnaud]

it's seem the driver is here https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/aquantia/atlantic
I don't how It can be import in ipxe

[aarnaud]

I setup a lab to debug and capture packets. I also enable some others debug option :

DEBUG=snp,nii,netdevice,snponly,snpnet,efi_driver,efi_init,efi_pci

I detected in debug log that iPXE seem to detect twice the adpater, there is twice time CHAINED ... found SimpleNetwork for the same MAC. If compare when I plug the other nic (Intel 1G) without changing any bios config, there is only one line about SimpleNetwork

Marvel AQtion 10Gbit that failed:

Intel PCIe Ethernet that work:

[aarnaud]

This issue concern only EFI chainload with snp.efi, snponly.efi, ipxe.efi.
legacy boot work fine with undionly.kpxe, just tested

[aarnaud]

Update firmware to 3.1.121 did some fix, now DHCP work.

[aarnaud]

I discovered with tcpdump that iPXE resolved domain names with IPv6 and try to use an IPv6 address.
When I disable IPv6 on my router, it's work
When I disable IPv6 with #undef NET_PROTO_IPV6 it's work too.

Network card firmware to 3.1.121 and disable IPv6

It's seem we have a way to fix, I will test on different machines with this same network card to see if it's working.

[aarnaud]

I discovered with tcpdump that iPXE resolved domain names with IPv6 and try to use an IPv6 address.
When I disable IPv6 on my router, it's work
When I disable IPv6 with #undef NET_PROTO_IPV6 it's work too.

short version fix: Network card firmware to 3.1.121 and disable IPv6

It's seem we have a way to fix, I will test on different machines with this same network card to see if it's working.

[aarnaud]

quick update on Lenovo P620, firmware up to date with 4.2.44_dash and IPv6 disabled the issue is still here

[aarnaud]

After a dirty downgrade with DIAG.exe to version 3.1.121, it's working for Lenovo P620

I found this commit on a fork : 7cfd66c but was never merged.

maybe @pbasavaraju can help us ?

[gregjhogan]

I think the problem is just the missing driver, I rebased the Marvell Aquantia PR on ipxe master and my Lenovo P620 with a AQC107 ethernet controller went from not working to working.

https://github.com/commaai/ipxe/tree/aqc

[amigthea]

Update firmware to 3.1.121 did some fix, now DHCP work.

this solved for me too as I was on 3.1.98, thank you!

[maltris]

Tried to disable IPv6 and tried several commits with the added driver without success.

Firmware unknown for now as this is a fresh supermicro which does not show any hardware-information in its BMC.

Where would one get the latest firmware for the AQC113C? Marvell is not even mentioning this NIC on their website ...

[gregjhogan]

It seems like they develop their ipxe integration on the Aquantia/ipxe fork, and if you look through the branches you will find:
https://github.com/Aquantia/ipxe/commits/aqc113_ipxe

[aarnaud]

BTW this issue was created for AQC107

[gregjhogan]

BTW this issue was created for AQC107

The story of Marvell attempting to get support added for this hardware ...
#328
#370
#439
#444
#467

Since they seem to have upset the maintainers, I rebased and containerized the build process for portability, hopefully making it easier for others who need AQC107 support:
https://github.com/commaai/ipxe/blob/aqc/build.sh

[RCPost]

@gregjhogan - Thanks very much for the Aquantia/ipxe fork. I've used your script to compile iPXE again.

Appologies for the dumb question, but do you know how one would take the compiled iPXE and update a Fog Project server?

Thanks in advance!

[ThinkSalat]

How are you guys able to boot into this thing? My onboard ethernet is bootable, but I can't get this to show up on the boot menu. I'm using the qnap qna t310g1t, connected to an asrock pg-itx z790 board. I'm on the latest drivers and firmware available on marvell's website. I have booting through thunderbolt enabled as well.

[kentonv]

Tonight I managed to netboot Windows 10 over iSCSI via iPXE using an AQC113C:

• I compiled this branch: https://github.com/Aquantia/ipxe/commits/aqc1xx Note @gregjhogan linked this repo in an earlier comment, but linked the aqc113_ipxe branch, which does NOT compile. The aqc1xx branch, however, supports AQC113 (per the commit message). I did also try @gregjhogan's own fork (the repo under commaai) but this did not appear to support the AQC113.
• I also had to update my BIOS (on a Gigabyte Z790 Aorus Master motherboard). Before updating the BIOS, Windows would BSOD on boot with DRIVER IRQL NOT LESS OR EQUAL.

[maxpain]

I have the same problem

[srett]

The story of Marvell attempting to get support added for this hardware ... #328 #370 #439 #444 #467

Since they seem to have upset the maintainers

Well that's unfortunate, in the end the users suffer. Now we need to maintain yet another patch set. (This is not to say I find this pushy behavior acceptable)

[stappersg]

Time will tell how much #1171 solves of this issue.