-
Notifications
You must be signed in to change notification settings - Fork 298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plan for Mellanox ConnectX-4 support #706
Comments
Regarding |
@nnikolaev-virtualopensystems The normal operation of Failing all of that, it would also be possible for packetblaster to have a mode where it doesn't use the "DMA reset" trick and simply transmits packets. That should work for any I/O device but would likely become CPU bound if you use many NICs. (packetblaster does 100G - 10x10G - without breaking a sweat and that is because of the trick.) |
Added a link to Mellanox firmware release notes. Interesting reading. Gives some insight into the line between hardware/firmware on the cards i.e. which bugs are fixed and features added by firmware upgrades. |
I have pushed a major update of the ConnectX-4 driver in commit 7659eb6. I have been able to initialize the card and transmit and receive packets now. I am reformulating the code more cleanly now. The initialization side of this is pushed and next I am doing clean transmit and receive. I also need to provide a suitable API for multiprocess operation and for setting up the Flow Tables and hashing in useful ways. I squashed the history on that branch before I pushed to github. I will be fetching some draft code from the more complete history at I also started filing issues for things that I am following up with Mellanox support. See issues with tag 'mellanox'. |
I pushed a big update to the Mellanox driver with commit 21d0dc3. There is still work to do but most things are in place now. The driver is now designed for multiprocess operation for use with #1021. The design is to have one Example: -- App to setup the NIC
config.app(c, 'nic', ConnectX4, {pciaddress = '01:00.0',
queues = {'a', 'b', 'c'}})
-- Apps to perform I/O (can be in other processes)
config.app(c, 'io-a', IO, {pciaddress = '01:00.0', queue = 'a'})
config.app(c, 'io-b', IO, {pciaddress = '01:00.0', queue = 'b'})
config.app(c, 'io-c', IO, {pciaddress = '01:00.0', queue = 'c'}) Currently all queues are setup for hashing (RSS). There is more to do:
Current basic selftest output with sending/receiving packets between
|
Great progress! I will try to meet you half way and get the IO(Control) abstractions in order and think about nefarious selftests. |
Is this still progressing? It looks like things were close to done, and then the ticket stalled or something. It looks like Snabb is seeing a lot of development; is there just a ton of stuff going on around the I/O 2.0 that is holding up these other tickets? |
Hi @tsuraan. Yes, I am actually planning to loop back this month and try to get the driver branch ready for upstream. There is a quite complete driver here: connectx_4.lua. This basically works but seems to often exercise some bad cases in the firmware (sometimes the NIC gets wedged and requires a cold boot server power cycle to recover.) The more recent firmwares are also lacking some important information from their release notes (definitions of new error codes that are appearing sometimes.) It's a bit of a slow and frustrating process to resolve these issues but I have some new leads that I plan to follow up shortly. |
This issue spells out a plan for adding support for Mellanox ConnectX-4 NICs to Snabb Switch.
The idea is to develop "native" support for Mellanox cards. This requires understanding the hardware in depth and writing our own drivers. The necessary information is now public (as of June 2016): Mellanox Adapter Programmer's Reference Manual.
Logistics:
Driver:
Integration:
Related work:
Risks and unknowns:
Those risks should be taken care of when the PRM is released and we have a card to experiment with.
The text was updated successfully, but these errors were encountered: