From 30123fb248d30af24dac1d8e883b2908e09b0f95 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 28 May 2019 17:44:52 -0400 Subject: [PATCH 01/28] Update Release Notes Initial commit, still a lot to fill in --- docs/Releases.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/Releases.md b/docs/Releases.md index 8d2ea15fd..2f29e1677 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -13,6 +13,32 @@ use a date based versioning system. Now, a release version can look like `17.11` where the "major" number is the year and the "minor" number is the month. +## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, CI PR Review, Web Stats Updates, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. + +### Shared Core Mode: +This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. + +Usage: +Run NFs as usual but include the `-c` flag for the onvm_mgr to enable the shared cpu mode. + +### Major Architectureal/API/Initialization/Signal Handling Changes: +Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. +Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. +Instead of `onvm_nf_info` the NF will now pass the `onvm_nf_init_ctx` struct to onvm_mgr. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. + +### CI PR Review: +CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purpoces. The new CI system is able to approve/reject pull requsts. +CI currently performs these checks: + - Check the branch (for our discussed change of develop->master as main branch) + - Run performance check (speed tester currently with 35mil benchmark) + - Run linter (only on files from the PR diff) + +**v19.05 API Changes:** +FILL_IN + +**v19.05 API Additions:** +FILL_IN + ## v19.02 (2/19): Manager Assigned NF Cores, Global Launch Script, DPDK 18.11 Update, Web Stats Overhaul, Load Generator NF, CI (Internal repo only), minor improvements and bug fixes This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm From 38c36a0988be3a14253d9302955d1496116e2847 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 28 May 2019 18:08:51 -0400 Subject: [PATCH 02/28] Update Releases.md --- docs/Releases.md | 50 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 2f29e1677..0e54784db 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -18,14 +18,43 @@ is the month. ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. +The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. + + Usage: -Run NFs as usual but include the `-c` flag for the onvm_mgr to enable the shared cpu mode. + - To enable shared core mode pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they can share cores with other NFs + +Notes: + - All code for sharing CPUs is within `if (ONVM_ENABLE_SHARED_CPU)` blocks + - When enabled, you can run multiple NFs on the same CPU core with much less interference than if they are polling for packets + - This code does not provide any particular intelligence for how NFs are scheduled or when they wakeup/sleep + - Note that the manager threads all still use polling ### Major Architectureal/API/Initialization/Signal Handling Changes: -Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. -Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. +Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. +Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. Instead of `onvm_nf_info` the NF will now pass the `onvm_nf_init_ctx` struct to onvm_mgr. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. +The new NF launch/shutdown sequence looks as follows: +``` + struct onvm_nf_local_ctx *nf_local_ctx; + + nf_local_ctx = onvm_nflib_init_nf_local_ctx(); + onvm_nflib_start_signal_handler(nf_local_ctx, NULL); + + if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx)) < 0) + // error checks + + argc -= arg_offset; + argv += arg_offset; + + if (parse_app_args(argc, argv, progname) < 0) + // error checks + + onvm_nflib_run(nf_local_ctx, &packet_handler); + onvm_nflib_stop(nf_local_ctx); +``` + ### CI PR Review: CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purpoces. The new CI system is able to approve/reject pull requsts. CI currently performs these checks: @@ -33,6 +62,21 @@ CI currently performs these checks: - Run performance check (speed tester currently with 35mil benchmark) - Run linter (only on files from the PR diff) +### LPM Firewall NF: +FILL_IN + +### Payload Search NF: +FILL_IN + +### TTL Flags: +FILL_IN + +### Minor improvements +FILL_IN + +### Bug fixes: +FILL_IN + **v19.05 API Changes:** FILL_IN From c72cbad00336fb44b352e1b695647beac0c8788f Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 28 May 2019 18:10:12 -0400 Subject: [PATCH 03/28] Update Releases.md --- docs/Releases.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/Releases.md b/docs/Releases.md index 0e54784db..c07a49f0f 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -14,6 +14,9 @@ like `17.11` where the "major" number is the year and the "minor" number is the month. ## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, CI PR Review, Web Stats Updates, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. +This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm + +**This release features a lot of breaking API changes.** ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. From 46b1cda576d4f43e3faff03d45795a9a4782aba4 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Fri, 31 May 2019 23:34:41 -0400 Subject: [PATCH 04/28] Update Releases.md --- docs/Releases.md | 49 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index c07a49f0f..48cf3f03b 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -18,6 +18,8 @@ This release adds several new features and changes how the onvm_mgr and NFs star **This release features a lot of breaking API changes.** +Performance: This release should fix the major performance issues that were present in the last release. + ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. @@ -36,16 +38,21 @@ Notes: ### Major Architectureal/API/Initialization/Signal Handling Changes: Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. -Instead of `onvm_nf_info` the NF will now pass the `onvm_nf_init_ctx` struct to onvm_mgr. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. +Instead of `onvm_nf_info` the NF will now pass the `onvm_nf_init_ctx` struct to onvm_mgr. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. +Finally we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. The new NF launch/shutdown sequence looks as follows: ``` - struct onvm_nf_local_ctx *nf_local_ctx; + struct onvm_nf_local_ctx *nf_local_ctx; + struct onvm_nf_function_table *nf_function_table; nf_local_ctx = onvm_nflib_init_nf_local_ctx(); onvm_nflib_start_signal_handler(nf_local_ctx, NULL); - if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx)) < 0) + nf_function_table = onvm_nflib_init_nf_function_table(); + nf_function_table->pkt_handler = &packet_handler; + + if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) // error checks argc -= arg_offset; @@ -54,7 +61,7 @@ The new NF launch/shutdown sequence looks as follows: if (parse_app_args(argc, argv, progname) < 0) // error checks - onvm_nflib_run(nf_local_ctx, &packet_handler); + onvm_nflib_run(nf_local_ctx); onvm_nflib_stop(nf_local_ctx); ``` @@ -66,19 +73,43 @@ CI currently performs these checks: - Run linter (only on files from the PR diff) ### LPM Firewall NF: -FILL_IN +The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. ### Payload Search NF: -FILL_IN +The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. ### TTL Flags: -FILL_IN +Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets recieved. + +### NF to NF Messaging: +Adds the ability for NFs to send messages to other NFs. NFs need to define a message handler to receive messages and are responsible to +free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements -FILL_IN +Make Number of mbufs a Constant Value - Significant performance increase +Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated +Check if ONVM_HOME is Set Before Compiling ONVM +Update Broken Links in the Style Docs +Add Core Information to Web Stats +Add NF Core Number to Web Stats +Update Install Script Hugepage Setup & Kernel Driver Installation +Add Compatibility Changes to Run ONVM on Ubuntu 18.04.1 +Improve Installation Debugging Documentation +Change onvm-pktgen Submodule to Upstream Pktgen ### Bug fixes: -FILL_IN +Free Memory on ONVM_MGR Shutdown +Launch Script to Handle Multi-word String Arguments +NF Advanced Ring Thread Process NF Shutdown Messages +Adds NF Ring Cleanup Logic On Shutdown +Resolve Shutdown Memory Leaks +Add NF Tag Memory Allocation +Fix the Parse IP Helper Function +Fix Speed Tester NF Generated Packets Counter +Add Termination of Started but not yet Running NFs +Add ONVM mgr web mode memory cleanup on shutdown +Removes the Old Flow Tracker NF Launch Script +Fix Deprecated DPDK Function in Speed Tester NF **v19.05 API Changes:** FILL_IN From 22cb6c30827e31d650b610050667688b31512f91 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Fri, 31 May 2019 23:42:58 -0400 Subject: [PATCH 05/28] Update Releases.md --- docs/Releases.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 48cf3f03b..97f4d5a14 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -13,7 +13,7 @@ use a date based versioning system. Now, a release version can look like `17.11` where the "major" number is the year and the "minor" number is the month. -## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, CI PR Review, Web Stats Updates, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. +## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm **This release features a lot of breaking API changes.** @@ -63,7 +63,18 @@ The new NF launch/shutdown sequence looks as follows: onvm_nflib_run(nf_local_ctx); onvm_nflib_stop(nf_local_ctx); -``` +``` + +### Stats Updates: +New features provide new stats, thus we have updates onvm_mgr stats output. +Here is the default mode: +FILL_IN + +Here is the verbose mode: +FILL_IN + +The shared core mode adds another line in the end of the verbose stats output to showcase wakeup statistics. + ### CI PR Review: CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purpoces. The new CI system is able to approve/reject pull requsts. From 64539e6b95c6c7ced9ea921f47c4e52628fc48fd Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Fri, 31 May 2019 23:44:07 -0400 Subject: [PATCH 06/28] Update Releases.md --- docs/Releases.md | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 97f4d5a14..298ebadee 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -97,30 +97,30 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements -Make Number of mbufs a Constant Value - Significant performance increase -Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated -Check if ONVM_HOME is Set Before Compiling ONVM -Update Broken Links in the Style Docs -Add Core Information to Web Stats -Add NF Core Number to Web Stats -Update Install Script Hugepage Setup & Kernel Driver Installation -Add Compatibility Changes to Run ONVM on Ubuntu 18.04.1 -Improve Installation Debugging Documentation -Change onvm-pktgen Submodule to Upstream Pktgen +Make Number of mbufs a Constant Value - Significant performance increase +Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated +Check if ONVM_HOME is Set Before Compiling ONVM +Update Broken Links in the Style Docs +Add Core Information to Web Stats +Add NF Core Number to Web Stats +Update Install Script Hugepage Setup & Kernel Driver Installation +Add Compatibility Changes to Run ONVM on Ubuntu 18.04.1 +Improve Installation Debugging Documentation +Change onvm-pktgen Submodule to Upstream Pktgen ### Bug fixes: -Free Memory on ONVM_MGR Shutdown -Launch Script to Handle Multi-word String Arguments -NF Advanced Ring Thread Process NF Shutdown Messages -Adds NF Ring Cleanup Logic On Shutdown -Resolve Shutdown Memory Leaks -Add NF Tag Memory Allocation -Fix the Parse IP Helper Function -Fix Speed Tester NF Generated Packets Counter -Add Termination of Started but not yet Running NFs -Add ONVM mgr web mode memory cleanup on shutdown -Removes the Old Flow Tracker NF Launch Script -Fix Deprecated DPDK Function in Speed Tester NF +Free Memory on ONVM_MGR Shutdown +Launch Script to Handle Multi-word String Arguments +NF Advanced Ring Thread Process NF Shutdown Messages +Adds NF Ring Cleanup Logic On Shutdown +Resolve Shutdown Memory Leaks +Add NF Tag Memory Allocation +Fix the Parse IP Helper Function +Fix Speed Tester NF Generated Packets Counter +Add Termination of Started but not yet Running NFs +Add ONVM mgr web mode memory cleanup on shutdown +Removes the Old Flow Tracker NF Launch Script +Fix Deprecated DPDK Function in Speed Tester NF **v19.05 API Changes:** FILL_IN From d4a3198c238a6badf944b81d25632aecc822e25d Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sat, 1 Jun 2019 10:25:56 -0400 Subject: [PATCH 07/28] Update Releases.md --- docs/Releases.md | 70 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 23 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 298ebadee..85bb708a2 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -13,7 +13,7 @@ use a date based versioning system. Now, a release version can look like `17.11` where the "major" number is the year and the "minor" number is the month. -## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. +## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, Advanced Rings Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm **This release features a lot of breaking API changes.** @@ -65,6 +65,12 @@ The new NF launch/shutdown sequence looks as follows: onvm_nflib_stop(nf_local_ctx); ``` +### Advanced Rings Changes: +This release changes our approach to NFs using the advanced rings mode. Previously we were trying to provide APIs for advanced ring developers such as scaling, but this logic should be managed by the NFs themselves. Because of this we're reworking those APIs and letting the NF devs handle everything themselves. + - Speed Tester NF advanced rings mode is removed + - Scaling Example NF advanced rings mode has been reworked and cleaned up + - Extra APIs have been removed + ### Stats Updates: New features provide new stats, thus we have updates onvm_mgr stats output. Here is the default mode: @@ -86,41 +92,53 @@ CI currently performs these checks: ### LPM Firewall NF: The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. +The NF accepts a json config with these rules: +``` +"ruleName": { + "ip": "127.1.1.0", + "depth": 32, + "action": 0 +} +``` + + ### Payload Search NF: The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. ### TTL Flags: Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets recieved. +Usage: + - `-t TTL` will stop the NF after the NF tx exceeds `PKT_LIMIT * PKT_TTL_MULTIPLIER`, measured in seconds. + - `-l PACKET_LIMIT` will stop the NF after the NF was alive longer than TTL_TIME*TIME_TTL_MULTIPLIER, measured in millions of packets. + ### NF to NF Messaging: Adds the ability for NFs to send messages to other NFs. NFs need to define a message handler to receive messages and are responsible to free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements -Make Number of mbufs a Constant Value - Significant performance increase -Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated -Check if ONVM_HOME is Set Before Compiling ONVM -Update Broken Links in the Style Docs -Add Core Information to Web Stats -Add NF Core Number to Web Stats -Update Install Script Hugepage Setup & Kernel Driver Installation -Add Compatibility Changes to Run ONVM on Ubuntu 18.04.1 -Improve Installation Debugging Documentation -Change onvm-pktgen Submodule to Upstream Pktgen + - Make Number of mbufs a Constant Value - Significant performance increase + - Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated + - Check if ONVM_HOME is Set Before Compiling ONVM + - Add Core Information to Web Stats + - Update Install Script Hugepage Setup & Kernel Driver Installation + - Add Compatibility Changes to Run ONVM on Ubuntu 18.04.1 + - Various Documentation updates and fixes + - Change onvm-pktgen Submodule to Upstream Pktgen ### Bug fixes: -Free Memory on ONVM_MGR Shutdown -Launch Script to Handle Multi-word String Arguments -NF Advanced Ring Thread Process NF Shutdown Messages -Adds NF Ring Cleanup Logic On Shutdown -Resolve Shutdown Memory Leaks -Add NF Tag Memory Allocation -Fix the Parse IP Helper Function -Fix Speed Tester NF Generated Packets Counter -Add Termination of Started but not yet Running NFs -Add ONVM mgr web mode memory cleanup on shutdown -Removes the Old Flow Tracker NF Launch Script -Fix Deprecated DPDK Function in Speed Tester NF + - Free Memory on ONVM_MGR Shutdown + - Launch Script to Handle Multi-word String Arguments + - NF Advanced Ring Thread Process NF Shutdown Messages + - Adds NF Ring Cleanup Logic On Shutdown + - Resolve Shutdown Memory Leaks + - Add NF Tag Memory Allocation + - Fix the Parse IP Helper Function + - Fix Speed Tester NF Generated Packets Counter + - Add Termination of Started but not yet Running NFs + - Add ONVM mgr web mode memory cleanup on shutdown + - Removes the Old Flow Tracker NF Launch Script + - Fix Deprecated DPDK Function in Speed Tester NF **v19.05 API Changes:** FILL_IN @@ -128,6 +146,12 @@ FILL_IN **v19.05 API Additions:** FILL_IN +**v19.05 API Additions:** +FILL_IN + +**v19.05 Removed APIs:** +FILL_IN + ## v19.02 (2/19): Manager Assigned NF Cores, Global Launch Script, DPDK 18.11 Update, Web Stats Overhaul, Load Generator NF, CI (Internal repo only), minor improvements and bug fixes This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm From 970393ff616a0ace61de4d5bd7e135eba8addae0 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sat, 1 Jun 2019 10:37:26 -0400 Subject: [PATCH 08/28] Update Releases.md --- docs/Releases.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 85bb708a2..4f3c47c10 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -117,8 +117,8 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements - - Make Number of mbufs a Constant Value - Significant performance increase - - Reuse NF Instance IDs - Reuse instance IDs of old NFs that have terminated + - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This lead toi performance degradation as the requested number of mbufs was too high, changing this to a constatnt has significanly improved performance. + - **Reuse NF Instance IDs** - Reuse instance IDs of old NFs that have terminated. The instance IDs are still continiously incremented up to the `MAX_NFS` constant, but when that number is reached the next NF instance ID will be wrapped back to the starting value and find the first unoccupied instance ID. - Check if ONVM_HOME is Set Before Compiling ONVM - Add Core Information to Web Stats - Update Install Script Hugepage Setup & Kernel Driver Installation From d19d770cfce62e5ad6dabf58a7c70091f82f4219 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sat, 1 Jun 2019 10:45:17 -0400 Subject: [PATCH 09/28] Update Releases.md --- docs/Releases.md | 60 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 6 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 4f3c47c10..694c172cf 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -72,14 +72,62 @@ This release changes our approach to NFs using the advanced rings mode. Previous - Extra APIs have been removed ### Stats Updates: -New features provide new stats, thus we have updates onvm_mgr stats output. -Here is the default mode: -FILL_IN +This release updates both console and web stats. +The new default mode now displays NF tag and core ID: +``` +PORTS +----- +Port 0: '90:e2:ba:b3:bc:6c' -Here is the verbose mode: -FILL_IN +Port 0 - rx: 4 ( 0 pps) tx: 0 ( 0 pps) + +NF TAG IID / SID / CORE rx_pps / tx_pps rx_drop / tx_drop out / tonf / drop +---------------------------------------------------------------------------------------------------------------------- +speed_tester 1 / 1 / 4 1693920 / 1693920 0 / 0 0 / 40346970 / 0 +``` + +Verbose mode also adds PNT(Parent ID), S|W(NF state, sleeping or working), CHLD(Children count): +``` +PORTS +----- +Port 0: '90:e2:ba:b3:bc:6c' + +Port 0 - rx: 4 ( 0 pps) tx: 0 ( 0 pps) + +NF TAG IID / SID / CORE rx_pps / tx_pps rx_drop / tx_drop out / tonf / drop + PNT / S|W / CHLD drop_pps / drop_pps rx_drop / tx_drop next / buf / ret +---------------------------------------------------------------------------------------------------------------------- +speed_tester 1 / 1 / 4 9661664 / 9661664 94494528 / 94494528 0 / 94494487 / 0 + 0 / W / 0 0 / 0 0 / 0 0 / 0 / 128 +``` + +The shared core mode adds wakeup information stats: +``` +PORTS +----- +Port 0: '90:e2:ba:b3:bc:6c' + +Port 0 - rx: 5 ( 0 pps) tx: 0 ( 0 pps) + +NF TAG IID / SID / CORE rx_pps / tx_pps rx_drop / tx_drop out / tonf / drop + PNT / S|W / CHLD drop_pps / drop_pps rx_drop / tx_drop next / buf / ret + wakeups / wakeup_rt +---------------------------------------------------------------------------------------------------------------------- +simple_forward 2 / 2 / 4 27719 / 27719 764439 / 764439 0 / 764439 / 0 + 0 / S / 0 0 / 0 0 / 0 0 / 0 / 0 + 730557 / 25344 + +speed_tester 3 / 1 / 5 27719 / 27719 764440 / 764439 0 / 764440 / 0 + 0 / W / 0 0 / 0 0 / 0 0 / 0 / 1 + 730560 / 25347 + + + +Shared CPU stats +---------------- +Total wakeups = 1461122, Wakeup rate = 50696 +``` -The shared core mode adds another line in the end of the verbose stats output to showcase wakeup statistics. ### CI PR Review: From b544bc9f646688d0a753cf4a9fd98ce0eca0b611 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sat, 1 Jun 2019 10:58:56 -0400 Subject: [PATCH 10/28] Update Releases.md --- docs/Releases.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 694c172cf..25c60744c 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -18,16 +18,17 @@ This release adds several new features and changes how the onvm_mgr and NFs star **This release features a lot of breaking API changes.** -Performance: This release should fix the major performance issues that were present in the last release. +**Performance**: This release should fix the major performance issues that were present in the last release. + +**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now catalloged by tags. ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. - Usage: - - To enable shared core mode pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they can share cores with other NFs + - To enable shared core mode pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they can share cores with other NFs Notes: - All code for sharing CPUs is within `if (ONVM_ENABLE_SHARED_CPU)` blocks From c04c48224530c4cf52d0c71c30847cfe71531215 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sat, 1 Jun 2019 22:49:39 -0400 Subject: [PATCH 11/28] Update Releases.md --- docs/Releases.md | 123 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 116 insertions(+), 7 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 25c60744c..5ab40c4c0 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -37,13 +37,122 @@ Notes: - Note that the manager threads all still use polling ### Major Architectureal/API/Initialization/Signal Handling Changes: -Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. -Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. -Instead of `onvm_nf_info` the NF will now pass the `onvm_nf_init_ctx` struct to onvm_mgr. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. -Finally we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. +- Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm + + Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. + ```c + struct onvm_nf_local_ctx { + struct onvm_nf *nf; + rte_atomic16_t nf_init_finished; + rte_atomic16_t keep_running; + }; + ``` -The new NF launch/shutdown sequence looks as follows: -``` +- Reworking the `onvm_nf` struct + + Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. + + ```c + struct onvm_nf { + struct rte_ring *rx_q; + struct rte_ring *tx_q; + struct rte_ring *msg_q; + /* Struct for NF to NF communication (NF tx) */ + struct queue_mgr *nf_tx_mgr; + uint16_t instance_id; + uint16_t service_id; + uint8_t status; + char *tag; + /* Pointer to NF defined state data */ + void *data; + + struct { + uint16_t core; + /* Instance ID of parent NF or 0 */ + uint16_t parent; + rte_atomic16_t children_cnt; + } thread_info; + + struct { + uint16_t init_options; + /* If set NF will stop after time reaches time_to_live */ + uint16_t time_to_live; + /* If set NF will stop after pkts TX reach pkt_limit */ + uint16_t pkt_limit; + } flags; + + /* NF specific functions */ + struct onvm_nf_function_table *function_table; + + /* + * Define a structure with stats from the NFs. + * + * These stats hold how many packets the NF will actually receive, send, + * and how many packets were dropped because the NF's queue was full. + * The port-info stats, in contrast, record how many packets were received + * or transmitted on an actual NIC port. + */ + struct { + volatile uint64_t rx; + volatile uint64_t rx_drop; + volatile uint64_t tx; + volatile uint64_t tx_drop; + volatile uint64_t tx_buffer; + volatile uint64_t tx_returned; + volatile uint64_t act_out; + volatile uint64_t act_tonf; + volatile uint64_t act_drop; + volatile uint64_t act_next; + volatile uint64_t act_buffer; + } stats; + + struct { + /* + * Sleep state (shared mem variable) to track state of NF and trigger wakeups + * sleep_state = 1 => NF sleeping (waiting on semaphore) + * sleep_state = 0 => NF running (not waiting on semaphore) + */ + rte_atomic16_t *sleep_state; + /* Mutex for NF sem_wait */ + sem_t *nf_mutex; + } shared_core; + }; + ``` + + - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for intializtion + + This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. + + ```c + struct onvm_nf_init_cfg { + uint16_t instance_id; + uint16_t service_id; + uint16_t core; + uint16_t init_options; + uint8_t status; + char *tag; + /* If set NF will stop after time reaches time_to_live */ + uint16_t time_to_live; + /* If set NF will stop after pkts TX reach pkt_limit */ + uint16_t pkt_limit; + }; + ``` + + - Adding a function table struct `onvm_nf_function_table` + + Finally we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. + + ```c + struct onvm_nf_function_table { + nf_setup_fn setup; + nf_msg_handler_fn msg_handler; + nf_user_actions_fn user_actions; + nf_pkt_handler_fn pkt_handler; + }; + ``` + +**Overall, the new NF launch/shutdown sequence looks as follows:** +```c struct onvm_nf_local_ctx *nf_local_ctx; struct onvm_nf_function_table *nf_function_table; @@ -142,7 +251,7 @@ CI currently performs these checks: The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. The NF accepts a json config with these rules: -``` +```json "ruleName": { "ip": "127.1.1.0", "depth": 32, From f3cf9894fe607250397fc4ebc1f38141536451ab Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Sun, 2 Jun 2019 14:02:09 -0400 Subject: [PATCH 12/28] Fix grammar errors, expand some sections --- docs/Releases.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 5ab40c4c0..53aca3da1 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -20,10 +20,10 @@ This release adds several new features and changes how the onvm_mgr and NFs star **Performance**: This release should fix the major performance issues that were present in the last release. -**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now catalloged by tags. +**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. ### Shared Core Mode: -This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct. +This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared core variables for mgr are in the `nf_wakeup_info` structs, the NF shared core vars were moved to the `onvm_nf` struct. The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. @@ -50,7 +50,7 @@ Notes: - Reworking the `onvm_nf` struct - Which leads us to the `onvm_nf` struct rework. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info` struct and it was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. + Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info`, which was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. ```c struct onvm_nf { @@ -140,7 +140,7 @@ Notes: - Adding a function table struct `onvm_nf_function_table` - Finally we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. + Finally, we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. ```c struct onvm_nf_function_table { @@ -183,6 +183,10 @@ This release changes our approach to NFs using the advanced rings mode. Previous ### Stats Updates: This release updates both console and web stats. + + - For web stats this adds the Core Mappings page with the information about core mappings for both onvm_mgr and NFs. + - For console stats this overhauls the displayed stats and adds new information, see more bellow. + The new default mode now displays NF tag and core ID: ``` PORTS From f7ee08269d89fb911857a364b68883ee7e7ae6f1 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 00:28:09 -0400 Subject: [PATCH 13/28] Update Releases.md --- docs/Releases.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 53aca3da1..b19a875a4 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -13,7 +13,7 @@ use a date based versioning system. Now, a release version can look like `17.11` where the "major" number is the year and the "minor" number is the month. -## v19.05 (5/19): Shared Core Mode, Major Architectureal/API/Initialization/Signal Handling Changes, Advanced Rings Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. +## v19.05 (5/19): Shared Core Mode, Major Architectural/API/Initialization/Signal Handling Changes, Advanced Rings Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm **This release features a lot of breaking API changes.** @@ -31,12 +31,12 @@ Usage: - To enable shared core mode pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they can share cores with other NFs Notes: - - All code for sharing CPUs is within `if (ONVM_ENABLE_SHARED_CPU)` blocks + - All code for sharing CPUs is within `if (ONVM_NF_SHARE_CORE)` blocks - When enabled, you can run multiple NFs on the same CPU core with much less interference than if they are polling for packets - This code does not provide any particular intelligence for how NFs are scheduled or when they wakeup/sleep - Note that the manager threads all still use polling -### Major Architectureal/API/Initialization/Signal Handling Changes: +### Major Architectural/API/Initialization/Signal Handling Changes: - Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. @@ -119,7 +119,7 @@ Notes: }; ``` - - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for intializtion + - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for initialization This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. @@ -237,15 +237,15 @@ speed_tester 3 / 1 / 5 27719 / 27719 764440 / 764439 -Shared CPU stats ----------------- +Shared core stats +----------------- Total wakeups = 1461122, Wakeup rate = 50696 ``` ### CI PR Review: -CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purpoces. The new CI system is able to approve/reject pull requsts. +CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purposes. The new CI system is able to approve/reject pull requests. CI currently performs these checks: - Check the branch (for our discussed change of develop->master as main branch) - Run performance check (speed tester currently with 35mil benchmark) @@ -268,7 +268,7 @@ The NF accepts a json config with these rules: The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. ### TTL Flags: -Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets recieved. +Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets received. Usage: - `-t TTL` will stop the NF after the NF tx exceeds `PKT_LIMIT * PKT_TTL_MULTIPLIER`, measured in seconds. @@ -279,7 +279,7 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements - - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This lead toi performance degradation as the requested number of mbufs was too high, changing this to a constatnt has significanly improved performance. + - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This lead to performance degradation as the requested number of mbufs was too high, changing this to a constant has significantly improved performance. - **Reuse NF Instance IDs** - Reuse instance IDs of old NFs that have terminated. The instance IDs are still continiously incremented up to the `MAX_NFS` constant, but when that number is reached the next NF instance ID will be wrapped back to the starting value and find the first unoccupied instance ID. - Check if ONVM_HOME is Set Before Compiling ONVM - Add Core Information to Web Stats From 96518946d590f23225458fb3c52218b7ff0cdc4d Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 16:22:32 -0400 Subject: [PATCH 14/28] Update Releases.md --- docs/Releases.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index b19a875a4..55a6834eb 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -13,14 +13,14 @@ use a date based versioning system. Now, a release version can look like `17.11` where the "major" number is the year and the "minor" number is the month. -## v19.05 (5/19): Shared Core Mode, Major Architectural/API/Initialization/Signal Handling Changes, Advanced Rings Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. -This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm +## v19.05 (5/19): Shared Core Mode, Major Architectural Changes, Advanced Rings Changes, Stats Updates, CI PR Review, LPM Firewall NF, Payload Search NF, TTL Flags, minor improvements and bug fixes. +A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm **This release features a lot of breaking API changes.** **Performance**: This release should fix the major performance issues that were present in the last release. -**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. +**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. We're also starting to merge releases into master by pull requests, thus developers should branch of the develop branch and submit PRs against the develop branch. ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared core variables for mgr are in the `nf_wakeup_info` structs, the NF shared core vars were moved to the `onvm_nf` struct. @@ -36,7 +36,7 @@ Notes: - This code does not provide any particular intelligence for how NFs are scheduled or when they wakeup/sleep - Note that the manager threads all still use polling -### Major Architectural/API/Initialization/Signal Handling Changes: +### Major Architectural Changes: - Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. From 2230b1fedbd0c5cc66053bfd892c312f4914480f Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 16:43:31 -0400 Subject: [PATCH 15/28] Add API changes --- docs/Releases.md | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 55a6834eb..2bfb66047 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -303,16 +303,43 @@ free the custom message data. If the message is sent to a NF that doesn't have a - Fix Deprecated DPDK Function in Speed Tester NF **v19.05 API Changes:** -FILL_IN + - `int onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_info **nf_info_p)` -> `int onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_function_table *nf_function_table)` + - `int onvm_nflib_run(struct onvm_nf_info* info, pkt_handler_func pkt_handler)` -> `int onvm_nflib_run(struct onvm_nf_local_ctx *nf_local_ctx)` + - `int onvm_nflib_return_pkt(struct onvm_nf_info *nf_info, struct rte_mbuf* pkt)` -> `int onvm_nflib_return_pkt(struct onvm_nf *nf, struct rte_mbuf *pkt)` + - `int onvm_nflib_return_pkt_bulk(struct onvm_nf_info *nf_info, struct rte_mbuf** pkts, uint16_t count)` -> `onvm_nflib_return_pkt_bulk(struct onvm_nf *nf, struct rte_mbuf **pkts, uint16_t count)` + - `int onvm_nflib_nf_ready(struct onvm_nf_info *info)` -> `int onvm_nflib_nf_ready(struct onvm_nf *nf)` + - `int onvm_nflib_handle_msg(struct onvm_nf_msg *msg, __attribute__((unused)) struct onvm_nf_info *nf_info)` -> + `int onvm_nflib_handle_msg(struct onvm_nf_msg *msg, struct onvm_nf_local_ctx *nf_local_ctx)` + - `void onvm_nflib_stop(struct onvm_nf_info *nf_info)` -> `void onvm_nflib_stop(struct onvm_nf_local_ctx *nf_local_ctx)` + - `struct onvm_nf_scale_info *onvm_nflib_get_empty_scaling_config(struct onvm_nf_info *parent_info)` -> `struct onvm_nf_scale_info *onvm_nflib_get_empty_scaling_config(struct onvm_nf *nf)` + - `struct onvm_nf_scale_info *onvm_nflib_inherit_parent_config(struct onvm_nf_info *parent_info, void *data)` -> +`struct onvm_nf_scale_info *onvm_nflib_inherit_parent_config(struct onvm_nf *nf, void *data)` + -**v19.05 API Additions:** -FILL_IN **v19.05 API Additions:** -FILL_IN + - `struct onvm_nf_local_ctx *onvm_nflib_init_nf_local_ctx(void)` + - `struct onvm_nf_function_table *onvm_nflib_init_nf_function_table(void)` + - `int onvm_nflib_start_signal_handler(struct onvm_nf_local_ctx *nf_local_ctx, handle_signal_func signal_hanlder)` + - `int onvm_nflib_send_msg_to_nf(uint16_t dest_nf, void *msg_data)` + - `int onvm_nflib_request_lpm(struct lpm_request *req)` + - `struct onvm_configuration *onvm_nflib_get_onvm_config(void)` **v19.05 Removed APIs:** -FILL_IN + - `int onvm_nflib_run_callback(struct onvm_nf_info* info, pkt_handler_func pkt_handler, callback_handler_func callback_handler)` + - `struct rte_ring *onvm_nflib_get_tx_ring(struct onvm_nf_info* info)` + - `struct rte_ring *onvm_nflib_get_rx_ring(struct onvm_nf_info* info)` + - `struct onvm_nf *onvm_nflib_get_nf(uint16_t id)` + - `void onvm_nflib_set_setup_function(struct onvm_nf_info* info, setup_func setup)` + + + + //TODO find a place for these? + exposed apis-> + `int onvm_nflib_start_nf(struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_init_cfg *nf_init_cfg)` + `struct onvm_nf_init_cfg *onvm_nflib_init_nf_init_cfg(const char *tag)` + `struct onvm_nf_init_cfg *onvm_nflib_inherit_parent_init_cfg(struct onvm_nf *parent)` + ## v19.02 (2/19): Manager Assigned NF Cores, Global Launch Script, DPDK 18.11 Update, Web Stats Overhaul, Load Generator NF, CI (Internal repo only), minor improvements and bug fixes This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm From 1e585cb3cb253f37fc906d611d3fcfe492d68247 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 17:02:15 -0400 Subject: [PATCH 16/28] Add links to docs --- docs/Releases.md | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 2bfb66047..52a904a6d 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -27,14 +27,7 @@ This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. -Usage: - - To enable shared core mode pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they can share cores with other NFs - -Notes: - - All code for sharing CPUs is within `if (ONVM_NF_SHARE_CORE)` blocks - - When enabled, you can run multiple NFs on the same CPU core with much less interference than if they are polling for packets - - This code does not provide any particular intelligence for how NFs are scheduled or when they wakeup/sleep - - Note that the manager threads all still use polling +Usage and implementation details can be found [here][shared_core_docs]. ### Major Architectural Changes: - Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm @@ -254,19 +247,14 @@ CI currently performs these checks: ### LPM Firewall NF: The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. -The NF accepts a json config with these rules: -```json -"ruleName": { - "ip": "127.1.1.0", - "depth": 32, - "action": 0 -} -``` +Documentation for this NF can be found [here][firewall_nf_readme]. ### Payload Search NF: The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. +Documentation for this NF can be found [here][payload_scan_nf_readme]. + ### TTL Flags: Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets received. @@ -618,3 +606,9 @@ Each module comes with a header file with commented prototypes. And each c and h ## 4/24/16: Initial Release Initial source code release. + + + +[firewall_nf_readme]: ../examples/firewall/README.md +[payload_scan_nf_readme]: ../examples/payload_scan/README.md +[shared_core_docs]: ./NF_Dev.md#shared-cpu-mode From cc8da12f24d120c2649cee6ba856235a4aa41301 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 17:13:03 -0400 Subject: [PATCH 17/28] Update Releases.md --- docs/Releases.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 52a904a6d..77f9e5d03 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -170,9 +170,13 @@ Usage and implementation details can be found [here][shared_core_docs]. ### Advanced Rings Changes: This release changes our approach to NFs using the advanced rings mode. Previously we were trying to provide APIs for advanced ring developers such as scaling, but this logic should be managed by the NFs themselves. Because of this we're reworking those APIs and letting the NF devs handle everything themselves. - - Speed Tester NF advanced rings mode is removed - - Scaling Example NF advanced rings mode has been reworked and cleaned up - - Extra APIs have been removed + - Speed Tester NF advanced rings mode is removed + - Extra APIs have been removed + - Removes support for advanced rings scaling APIs + - Scaling Example NF advanced rings mode has been reworked, the new implementation now does its own pthread creation instead of relying on the onvm scaling APIs. Also makes a clear separation between default and advanced ring mode. + - Because of these changes some internal nflib APIs were exposed to the NF (`onvm_nflib_start_nf`, `onvm_nflib_init_nf_init_cfg`, `onvm_nflib_inherit_parent_init_cfg`) + + ### Stats Updates: This release updates both console and web stats. From 18448b0dc4c28aa09d1d775b58e9782a7d5f6133 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 19:29:29 -0400 Subject: [PATCH 18/28] Move API changes down --- docs/Releases.md | 208 ++++++++++++++++++++++++----------------------- 1 file changed, 105 insertions(+), 103 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 77f9e5d03..57a68db32 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -33,116 +33,20 @@ Usage and implementation details can be found [here][shared_core_docs]. - Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. - ```c - struct onvm_nf_local_ctx { - struct onvm_nf *nf; - rte_atomic16_t nf_init_finished; - rte_atomic16_t keep_running; - }; - ``` - Reworking the `onvm_nf` struct Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info`, which was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. - ```c - struct onvm_nf { - struct rte_ring *rx_q; - struct rte_ring *tx_q; - struct rte_ring *msg_q; - /* Struct for NF to NF communication (NF tx) */ - struct queue_mgr *nf_tx_mgr; - uint16_t instance_id; - uint16_t service_id; - uint8_t status; - char *tag; - /* Pointer to NF defined state data */ - void *data; - - struct { - uint16_t core; - /* Instance ID of parent NF or 0 */ - uint16_t parent; - rte_atomic16_t children_cnt; - } thread_info; - - struct { - uint16_t init_options; - /* If set NF will stop after time reaches time_to_live */ - uint16_t time_to_live; - /* If set NF will stop after pkts TX reach pkt_limit */ - uint16_t pkt_limit; - } flags; - - /* NF specific functions */ - struct onvm_nf_function_table *function_table; - - /* - * Define a structure with stats from the NFs. - * - * These stats hold how many packets the NF will actually receive, send, - * and how many packets were dropped because the NF's queue was full. - * The port-info stats, in contrast, record how many packets were received - * or transmitted on an actual NIC port. - */ - struct { - volatile uint64_t rx; - volatile uint64_t rx_drop; - volatile uint64_t tx; - volatile uint64_t tx_drop; - volatile uint64_t tx_buffer; - volatile uint64_t tx_returned; - volatile uint64_t act_out; - volatile uint64_t act_tonf; - volatile uint64_t act_drop; - volatile uint64_t act_next; - volatile uint64_t act_buffer; - } stats; - - struct { - /* - * Sleep state (shared mem variable) to track state of NF and trigger wakeups - * sleep_state = 1 => NF sleeping (waiting on semaphore) - * sleep_state = 0 => NF running (not waiting on semaphore) - */ - rte_atomic16_t *sleep_state; - /* Mutex for NF sem_wait */ - sem_t *nf_mutex; - } shared_core; - }; - ``` - - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for initialization This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. - ```c - struct onvm_nf_init_cfg { - uint16_t instance_id; - uint16_t service_id; - uint16_t core; - uint16_t init_options; - uint8_t status; - char *tag; - /* If set NF will stop after time reaches time_to_live */ - uint16_t time_to_live; - /* If set NF will stop after pkts TX reach pkt_limit */ - uint16_t pkt_limit; - }; - ``` - Adding a function table struct `onvm_nf_function_table` Finally, we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. - ```c - struct onvm_nf_function_table { - nf_setup_fn setup; - nf_msg_handler_fn msg_handler; - nf_user_actions_fn user_actions; - nf_pkt_handler_fn pkt_handler; - }; - ``` **Overall, the new NF launch/shutdown sequence looks as follows:** ```c @@ -246,7 +150,7 @@ CI is now available on the public branch. Only a specific list of whitelisted us CI currently performs these checks: - Check the branch (for our discussed change of develop->master as main branch) - Run performance check (speed tester currently with 35mil benchmark) - - Run linter (only on files from the PR diff) + - Run linter (only on the PR diff) ### LPM Firewall NF: The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. @@ -262,10 +166,6 @@ Documentation for this NF can be found [here][payload_scan_nf_readme]. ### TTL Flags: Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets received. -Usage: - - `-t TTL` will stop the NF after the NF tx exceeds `PKT_LIMIT * PKT_TTL_MULTIPLIER`, measured in seconds. - - `-l PACKET_LIMIT` will stop the NF after the NF was alive longer than TTL_TIME*TIME_TTL_MULTIPLIER, measured in millions of packets. - ### NF to NF Messaging: Adds the ability for NFs to send messages to other NFs. NFs need to define a message handler to receive messages and are responsible to free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. @@ -294,6 +194,110 @@ free the custom message data. If the message is sent to a NF that doesn't have a - Removes the Old Flow Tracker NF Launch Script - Fix Deprecated DPDK Function in Speed Tester NF +**v19.05 API Struct changes:** + +* Adding `onvm_nf_local_ctx` which is malloced and passed into `onvm_nflib_init` + ```c + struct onvm_nf_local_ctx { + struct onvm_nf *nf; + rte_atomic16_t nf_init_finished; + rte_atomic16_t keep_running; + }; + ``` +* Adding a function table for eaiser callback managing + ```c + struct onvm_nf_function_table { + nf_setup_fn setup; + nf_msg_handler_fn msg_handler; + nf_user_actions_fn user_actions; + nf_pkt_handler_fn pkt_handler; + }; + ``` + +* Renaming the old `onvm_nf_info` -> `onvm_nf_init_cfg` + ```c + struct onvm_nf_init_cfg { + uint16_t instance_id; + uint16_t service_id; + uint16_t core; + uint16_t init_options; + uint8_t status; + char *tag; + /* If set NF will stop after time reaches time_to_live */ + uint16_t time_to_live; + /* If set NF will stop after pkts TX reach pkt_limit */ + uint16_t pkt_limit; + }; + ``` + +* Consolidating previous `onvm_nf_info` and `onvm_nf` into a singular `onvm_nf` struct + ```c + struct onvm_nf { + struct rte_ring *rx_q; + struct rte_ring *tx_q; + struct rte_ring *msg_q; + /* Struct for NF to NF communication (NF tx) */ + struct queue_mgr *nf_tx_mgr; + uint16_t instance_id; + uint16_t service_id; + uint8_t status; + char *tag; + /* Pointer to NF defined state data */ + void *data; + + struct { + uint16_t core; + /* Instance ID of parent NF or 0 */ + uint16_t parent; + rte_atomic16_t children_cnt; + } thread_info; + + struct { + uint16_t init_options; + /* If set NF will stop after time reaches time_to_live */ + uint16_t time_to_live; + /* If set NF will stop after pkts TX reach pkt_limit */ + uint16_t pkt_limit; + } flags; + + /* NF specific functions */ + struct onvm_nf_function_table *function_table; + + /* + * Define a structure with stats from the NFs. + * + * These stats hold how many packets the NF will actually receive, send, + * and how many packets were dropped because the NF's queue was full. + * The port-info stats, in contrast, record how many packets were received + * or transmitted on an actual NIC port. + */ + struct { + volatile uint64_t rx; + volatile uint64_t rx_drop; + volatile uint64_t tx; + volatile uint64_t tx_drop; + volatile uint64_t tx_buffer; + volatile uint64_t tx_returned; + volatile uint64_t act_out; + volatile uint64_t act_tonf; + volatile uint64_t act_drop; + volatile uint64_t act_next; + volatile uint64_t act_buffer; + } stats; + + struct { + /* + * Sleep state (shared mem variable) to track state of NF and trigger wakeups + * sleep_state = 1 => NF sleeping (waiting on semaphore) + * sleep_state = 0 => NF running (not waiting on semaphore) + */ + rte_atomic16_t *sleep_state; + /* Mutex for NF sem_wait */ + sem_t *nf_mutex; + } shared_core; + }; + ``` + **v19.05 API Changes:** - `int onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_info **nf_info_p)` -> `int onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_function_table *nf_function_table)` - `int onvm_nflib_run(struct onvm_nf_info* info, pkt_handler_func pkt_handler)` -> `int onvm_nflib_run(struct onvm_nf_local_ctx *nf_local_ctx)` @@ -307,8 +311,6 @@ free the custom message data. If the message is sent to a NF that doesn't have a - `struct onvm_nf_scale_info *onvm_nflib_inherit_parent_config(struct onvm_nf_info *parent_info, void *data)` -> `struct onvm_nf_scale_info *onvm_nflib_inherit_parent_config(struct onvm_nf *nf, void *data)` - - **v19.05 API Additions:** - `struct onvm_nf_local_ctx *onvm_nflib_init_nf_local_ctx(void)` - `struct onvm_nf_function_table *onvm_nflib_init_nf_function_table(void)` From 37d479a3213fa5584bfe22a344ce1c13d5b5f7e7 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Mon, 3 Jun 2019 20:31:32 -0400 Subject: [PATCH 19/28] Update Releases.md --- docs/Releases.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 57a68db32..f30178f2a 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -18,7 +18,7 @@ A CloudLab template is available with the latest release here: https://www.cloud **This release features a lot of breaking API changes.** -**Performance**: This release should fix the major performance issues that were present in the last release. +**Performance**: This release increases Pktgen benchmark performance from 7Mpps to 13.1 Mpps (measured by Pktgen sending packets to the ONVM Basic Monitor), thus fixing the major performance issue that was present in the last release. **Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. We're also starting to merge releases into master by pull requests, thus developers should branch of the develop branch and submit PRs against the develop branch. @@ -32,7 +32,7 @@ Usage and implementation details can be found [here][shared_core_docs]. ### Major Architectural Changes: - Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm - Previously the initialization sequence for NFs was tied to the `onvm_nf_info` struct which was used to initialize with the onvm_mgr. This was fine until we encountered the issue with Signal Handling, using the initialization sequence, the signal handling only started when initialization (dpdk init + onvm nflib init) has completely finished. This is not a good practice as proper cleanup might need to occur when handling signals during the initialization sequence. Therefore a decision was made to introduce a new NF context struct(`onvm_nf_local_ctx`) which would be malloced in the heap instead of being rte_malloced like the `onvm_nf_info`. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. + Previously the initialization sequence for NFs wasn't able to properly cleanup if a signal was recieved. Because of this we have introduced a new NF context struct(`onvm_nf_local_ctx`) which would be malloced before intilization begins and would help handle cleanup. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. - Reworking the `onvm_nf` struct @@ -45,7 +45,7 @@ Usage and implementation details can be found [here][shared_core_docs]. - Adding a function table struct `onvm_nf_function_table` - Finally, we introduced the `onvm_nf_function_table` struct that allows NF developers to fill in specific callback functions for their NFs. + Finally, we introduced the `onvm_nf_function_table` struct that groups all NF callback functions that can be set by developers. **Overall, the new NF launch/shutdown sequence looks as follows:** @@ -80,8 +80,6 @@ This release changes our approach to NFs using the advanced rings mode. Previous - Scaling Example NF advanced rings mode has been reworked, the new implementation now does its own pthread creation instead of relying on the onvm scaling APIs. Also makes a clear separation between default and advanced ring mode. - Because of these changes some internal nflib APIs were exposed to the NF (`onvm_nflib_start_nf`, `onvm_nflib_init_nf_init_cfg`, `onvm_nflib_inherit_parent_init_cfg`) - - ### Stats Updates: This release updates both console and web stats. From 6b909cf49e188cda63a9cc2ab02e1677d3d1b1a6 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 09:56:04 -0400 Subject: [PATCH 20/28] Update Releases.md --- docs/Releases.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/Releases.md b/docs/Releases.md index f30178f2a..6fff2aca2 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -99,7 +99,7 @@ NF TAG IID / SID / CORE rx_pps / tx_pps rx_drop / tx_drop speed_tester 1 / 1 / 4 1693920 / 1693920 0 / 0 0 / 40346970 / 0 ``` -Verbose mode also adds PNT(Parent ID), S|W(NF state, sleeping or working), CHLD(Children count): +Verbose mode also adds `PNT`(Parent ID), `S|W`(NF state, sleeping or working), `CHLD`(Children count): ``` PORTS ----- @@ -141,6 +141,18 @@ Shared core stats Total wakeups = 1461122, Wakeup rate = 50696 ``` +The super verbose stats output dumps all stats in a coma separated list for easy script parsing: +``` +#YYYY-MM-DD HH:MM:SS,nic_rx_pkts,nic_rx_pps,nic_tx_pkts,nic_tx_pps +#YYYY-MM-DD HH:MM:SS,nf_tag,instance_id,service_id,core,parent,state,children_cnt,rx,tx,rx_pps,tx_pps,rx_drop,tx_drop,rx_drop_rate,tx_drop_rate,act_out,act_tonf,act_drop,act_next,act_buffer,act_returned,num_wakeups,wakeup_rate +2019-06-04 08:54:52,0,4,4,0,0 +2019-06-04 08:54:53,0,4,0,0,0 +2019-06-04 08:54:54,simple_forward,1,2,4,0,W,0,29058,29058,29058,29058,0,0,0,0,0,29058,0,0,0,0,28951,28951 +2019-06-04 08:54:54,speed_tester,2,1,5,0,S,0,29058,29058,29058,29058,0,0,0,0,0,29059,0,0,0,1,28952,28952 +2019-06-04 08:54:55,0,4,0,0,0 +2019-06-04 08:54:55,simple_forward,1,2,4,0,W,0,101844,101843,72785,72785,0,0,0,0,0,101843,0,0,0,0,101660,101660 +2019-06-04 08:54:55,speed_tester,2,1,5,0,W,0,101844,101843,72785,72785,0,0,0,0,0,101844,0,0,0,1,101660,101660 +``` ### CI PR Review: From 54ab0ca05cb6fbbc0c2968b14e79af8a853d2322 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 10:04:34 -0400 Subject: [PATCH 21/28] Fix suggested grammar/whitespace issues Spelling is hard --- docs/Releases.md | 39 ++++++++++++++++----------------------- 1 file changed, 16 insertions(+), 23 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 6fff2aca2..952e18890 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -20,7 +20,7 @@ A CloudLab template is available with the latest release here: https://www.cloud **Performance**: This release increases Pktgen benchmark performance from 7Mpps to 13.1 Mpps (measured by Pktgen sending packets to the ONVM Basic Monitor), thus fixing the major performance issue that was present in the last release. -**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. We're also starting to merge releases into master by pull requests, thus developers should branch of the develop branch and submit PRs against the develop branch. +**Repo changes**: Default branch has been changed to `master`, active development can still be seen in `develop`. Most of the development is now done on the public repo to improve visibility, planned projects and improvements can be seen in this [pinned issue](https://github.com/sdnfv/openNetVM/issues/91), additionally pull requests and issues are now cataloged by tags. We're also starting to merge releases into master by pull requests, thus developers should branch off the develop branch and submit PRs against the develop branch. ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared core variables for mgr are in the `nf_wakeup_info` structs, the NF shared core vars were moved to the `onvm_nf` struct. @@ -30,24 +30,23 @@ The code is based on the hybrid-polling model proposed in [_Flurries: Countless Usage and implementation details can be found [here][shared_core_docs]. ### Major Architectural Changes: -- Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm +- Introduce a local `onvm_nf_init_ctx` struct allocated from the heap before starting onvm. - Previously the initialization sequence for NFs wasn't able to properly cleanup if a signal was recieved. Because of this we have introduced a new NF context struct(`onvm_nf_local_ctx`) which would be malloced before intilization begins and would help handle cleanup. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. + Previously the initialization sequence for NFs wasn't able to properly cleanup if a signal was received. Because of this we have introduced a new NF context struct(`onvm_nf_local_ctx`) which would be malloced before initialization begins and would help handle cleanup. This struct contains relevant information about the status of the initialization sequence and holds a reference to the `onvm_nf` struct which has all the information about the NF. -- Reworking the `onvm_nf` struct +- Reworking the `onvm_nf` struct. Previously the `onvm_nf` struct contained a pointer to the `onvm_nf_info`, which was used during processing. It's better to have one main struct that represents the NF, thus the contents of the `onvm_nf_info` were merged into the `onvm_nf` struct. This allows us to maintain a cleaner API where all information about the NF is stored in the `onvm_nf` struct. - - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for initialization + - Replace the old `onvm_nf_info` with a new `onvm_nf_init_ctx` struct that is passed to onvm_mgr for initialization. This struct contains all relevant information to spawn a new NF (service/instance IDs, flags, core, etc). When the NF is spawned this struct will be released back to the mempool. - - Adding a function table struct `onvm_nf_function_table` + - Adding a function table struct `onvm_nf_function_table`. Finally, we introduced the `onvm_nf_function_table` struct that groups all NF callback functions that can be set by developers. - **Overall, the new NF launch/shutdown sequence looks as follows:** ```c struct onvm_nf_local_ctx *nf_local_ctx; @@ -84,7 +83,7 @@ This release changes our approach to NFs using the advanced rings mode. Previous This release updates both console and web stats. - For web stats this adds the Core Mappings page with the information about core mappings for both onvm_mgr and NFs. - - For console stats this overhauls the displayed stats and adds new information, see more bellow. + - For console stats this overhauls the displayed stats and adds new information, see more below. The new default mode now displays NF tag and core ID: ``` @@ -154,34 +153,28 @@ The super verbose stats output dumps all stats in a coma separated list for easy 2019-06-04 08:54:55,speed_tester,2,1,5,0,W,0,101844,101843,72785,72785,0,0,0,0,0,101844,0,0,0,1,101660,101660 ``` - ### CI PR Review: -CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI due to security purposes. The new CI system is able to approve/reject pull requests. +CI is now available on the public branch. Only a specific list of whitelisted users can currently run CI for security purposes. The new CI system is able to approve/reject pull requests. CI currently performs these checks: - Check the branch (for our discussed change of develop->master as main branch) - Run performance check (speed tester currently with 35mil benchmark) - Run linter (only on the PR diff) ### LPM Firewall NF: -The firewall NF drops or forwards packets based on rules provided in the json file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. - -Documentation for this NF can be found [here][firewall_nf_readme]. +The firewall NF drops or forwards packets based on rules provided in a JSON config file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. Documentation for this NF can be found [here][firewall_nf_readme]. ### Payload Search NF: -The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. - -Documentation for this NF can be found [here][payload_scan_nf_readme]. +The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. The NF also has an inverse mode to drop on match and forward otherwise. Documentation for this NF can be found [here][payload_scan_nf_readme]. ### TTL Flags: Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in seconds and in millions of packets received. ### NF to NF Messaging: -Adds the ability for NFs to send messages to other NFs. NFs need to define a message handler to receive messages and are responsible to -free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. +Adds the ability for NFs to send messages to other NFs. NFs need to define a message handler to receive messages and are responsible to free the custom message data. If the message is sent to a NF that doesn't have a message handler the message is ignored. ### Minor improvements - - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This lead to performance degradation as the requested number of mbufs was too high, changing this to a constant has significantly improved performance. + - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This led to performance degradation as the requested number of mbufs was too high, changing this to a constant has significantly improved performance. - **Reuse NF Instance IDs** - Reuse instance IDs of old NFs that have terminated. The instance IDs are still continiously incremented up to the `MAX_NFS` constant, but when that number is reached the next NF instance ID will be wrapped back to the starting value and find the first unoccupied instance ID. - Check if ONVM_HOME is Set Before Compiling ONVM - Add Core Information to Web Stats @@ -206,7 +199,7 @@ free the custom message data. If the message is sent to a NF that doesn't have a **v19.05 API Struct changes:** -* Adding `onvm_nf_local_ctx` which is malloced and passed into `onvm_nflib_init` +* Adding `onvm_nf_local_ctx` which is malloced and passed into `onvm_nflib_init`: ```c struct onvm_nf_local_ctx { struct onvm_nf *nf; @@ -214,7 +207,7 @@ free the custom message data. If the message is sent to a NF that doesn't have a rte_atomic16_t keep_running; }; ``` -* Adding a function table for eaiser callback managing +* Adding a function table for eaiser callback managing: ```c struct onvm_nf_function_table { nf_setup_fn setup; @@ -224,7 +217,7 @@ free the custom message data. If the message is sent to a NF that doesn't have a }; ``` -* Renaming the old `onvm_nf_info` -> `onvm_nf_init_cfg` +* Renaming the old `onvm_nf_info` -> `onvm_nf_init_cfg`: ```c struct onvm_nf_init_cfg { uint16_t instance_id; @@ -240,7 +233,7 @@ free the custom message data. If the message is sent to a NF that doesn't have a }; ``` -* Consolidating previous `onvm_nf_info` and `onvm_nf` into a singular `onvm_nf` struct +* Consolidating previous `onvm_nf_info` and `onvm_nf` into a singular `onvm_nf` struct: ```c struct onvm_nf { struct rte_ring *rx_q; From 0471f3748ca2834d85c1da85f6ef3ff7de866747 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 10:38:52 -0400 Subject: [PATCH 22/28] Update API section --- docs/Releases.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 952e18890..1c7956902 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -163,7 +163,6 @@ CI currently performs these checks: ### LPM Firewall NF: The firewall NF drops or forwards packets based on rules provided in a JSON config file. This is achieved using DPDK's LPM (longest prefix matching) library. Default behavior is to drop a packet unless the packet matches a rule. The NF also has a debug mode to print decisions for every packet and an inverse match mode where default behavior is to forward a packet if it is not found in the table. Documentation for this NF can be found [here][firewall_nf_readme]. - ### Payload Search NF: The Payload Scan NF provides the functionality to search for a string within a given UDP or TCP packet payload. Packet is forwarded to its destination NF on a match, dropped otherwise. The NF also has an inverse mode to drop on match and forward otherwise. Documentation for this NF can be found [here][payload_scan_nf_readme]. @@ -320,7 +319,11 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes - `int onvm_nflib_start_signal_handler(struct onvm_nf_local_ctx *nf_local_ctx, handle_signal_func signal_hanlder)` - `int onvm_nflib_send_msg_to_nf(uint16_t dest_nf, void *msg_data)` - `int onvm_nflib_request_lpm(struct lpm_request *req)` - - `struct onvm_configuration *onvm_nflib_get_onvm_config(void)` + - `struct onvm_configuration *onvm_nflib_get_onvm_config(void)` + These APIs were previously internal but are now exposed for advanced ring NFs: + - `int onvm_nflib_start_nf(struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_init_cfg *nf_init_cfg)` + - `struct onvm_nf_init_cfg *onvm_nflib_init_nf_init_cfg(const char *tag)` + - `struct onvm_nf_init_cfg *onvm_nflib_inherit_parent_init_cfg(struct onvm_nf *parent)` **v19.05 Removed APIs:** - `int onvm_nflib_run_callback(struct onvm_nf_info* info, pkt_handler_func pkt_handler, callback_handler_func callback_handler)` @@ -328,14 +331,6 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes - `struct rte_ring *onvm_nflib_get_rx_ring(struct onvm_nf_info* info)` - `struct onvm_nf *onvm_nflib_get_nf(uint16_t id)` - `void onvm_nflib_set_setup_function(struct onvm_nf_info* info, setup_func setup)` - - - - //TODO find a place for these? - exposed apis-> - `int onvm_nflib_start_nf(struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_init_cfg *nf_init_cfg)` - `struct onvm_nf_init_cfg *onvm_nflib_init_nf_init_cfg(const char *tag)` - `struct onvm_nf_init_cfg *onvm_nflib_inherit_parent_init_cfg(struct onvm_nf *parent)` ## v19.02 (2/19): Manager Assigned NF Cores, Global Launch Script, DPDK 18.11 Update, Web Stats Overhaul, Load Generator NF, CI (Internal repo only), minor improvements and bug fixes From bc4ce53e7694d5e3dd7b9bd42944776e19daae94 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 10:43:35 -0400 Subject: [PATCH 23/28] Update Releases.md --- docs/Releases.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/Releases.md b/docs/Releases.md index 1c7956902..3fa87d410 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -49,26 +49,26 @@ Usage and implementation details can be found [here][shared_core_docs]. **Overall, the new NF launch/shutdown sequence looks as follows:** ```c - struct onvm_nf_local_ctx *nf_local_ctx; - struct onvm_nf_function_table *nf_function_table; +struct onvm_nf_local_ctx *nf_local_ctx; +struct onvm_nf_function_table *nf_function_table; - nf_local_ctx = onvm_nflib_init_nf_local_ctx(); - onvm_nflib_start_signal_handler(nf_local_ctx, NULL); +nf_local_ctx = onvm_nflib_init_nf_local_ctx(); +onvm_nflib_start_signal_handler(nf_local_ctx, NULL); - nf_function_table = onvm_nflib_init_nf_function_table(); - nf_function_table->pkt_handler = &packet_handler; +nf_function_table = onvm_nflib_init_nf_function_table(); +nf_function_table->pkt_handler = &packet_handler; - if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) - // error checks +if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) + // error checks + +argc -= arg_offset; +argv += arg_offset; - argc -= arg_offset; - argv += arg_offset; +if (parse_app_args(argc, argv, progname) < 0) + // error checks - if (parse_app_args(argc, argv, progname) < 0) - // error checks - - onvm_nflib_run(nf_local_ctx); - onvm_nflib_stop(nf_local_ctx); +onvm_nflib_run(nf_local_ctx); +onvm_nflib_stop(nf_local_ctx); ``` ### Advanced Rings Changes: @@ -140,7 +140,7 @@ Shared core stats Total wakeups = 1461122, Wakeup rate = 50696 ``` -The super verbose stats output dumps all stats in a coma separated list for easy script parsing: +The super verbose stats mode has also been updated to include new stats: ``` #YYYY-MM-DD HH:MM:SS,nic_rx_pkts,nic_rx_pps,nic_tx_pkts,nic_tx_pps #YYYY-MM-DD HH:MM:SS,nf_tag,instance_id,service_id,core,parent,state,children_cnt,rx,tx,rx_pps,tx_pps,rx_drop,tx_drop,rx_drop_rate,tx_drop_rate,act_out,act_tonf,act_drop,act_next,act_buffer,act_returned,num_wakeups,wakeup_rate @@ -206,6 +206,7 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes rte_atomic16_t keep_running; }; ``` + * Adding a function table for eaiser callback managing: ```c struct onvm_nf_function_table { @@ -332,7 +333,6 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes - `struct onvm_nf *onvm_nflib_get_nf(uint16_t id)` - `void onvm_nflib_set_setup_function(struct onvm_nf_info* info, setup_func setup)` - ## v19.02 (2/19): Manager Assigned NF Cores, Global Launch Script, DPDK 18.11 Update, Web Stats Overhaul, Load Generator NF, CI (Internal repo only), minor improvements and bug fixes This release adds several new features and changes how the onvm_mgr and NFs start. A CloudLab template is available with the latest release here: https://www.cloudlab.us/p/GWCloudLab/onvm From 480685af40f652e9007fc998cf583daaddace69c Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 10:47:39 -0400 Subject: [PATCH 24/28] Update Releases.md --- docs/Releases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Releases.md b/docs/Releases.md index 3fa87d410..d7bf7bfc0 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -82,7 +82,7 @@ This release changes our approach to NFs using the advanced rings mode. Previous ### Stats Updates: This release updates both console and web stats. - - For web stats this adds the Core Mappings page with the information about core mappings for both onvm_mgr and NFs. + - For web stats this adds the Core Mappings page with the core layout for both onvm_mgr and NFs. - For console stats this overhauls the displayed stats and adds new information, see more below. The new default mode now displays NF tag and core ID: From 092b4a02a82a1b48e7ea2bd49a3fbae754be2721 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 11:14:23 -0400 Subject: [PATCH 25/28] Update Releases.md --- docs/Releases.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Releases.md b/docs/Releases.md index d7bf7bfc0..7aff221e3 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -174,7 +174,8 @@ Adds the ability for NFs to send messages to other NFs. NFs need to define a mes ### Minor improvements - **Make Number of mbufs a Constant Value** - Previously the number of mbufs was calculated based on the `MAX_NFS` constant. This led to performance degradation as the requested number of mbufs was too high, changing this to a constant has significantly improved performance. - - **Reuse NF Instance IDs** - Reuse instance IDs of old NFs that have terminated. The instance IDs are still continiously incremented up to the `MAX_NFS` constant, but when that number is reached the next NF instance ID will be wrapped back to the starting value and find the first unoccupied instance ID. + - **Reuse NF Instance IDs** - Reuse instance IDs of old NFs that have terminated. The instance IDs are still continiously incremented up to the `MAX_NFS` constant, but when that number is reached the next NF instance ID will be wrapped back to the starting value and find the first unoccupied instance ID. + - Fix all major style errors - Check if ONVM_HOME is Set Before Compiling ONVM - Add Core Information to Web Stats - Update Install Script Hugepage Setup & Kernel Driver Installation From 6ce49f479cf653857cc5d2555d3e18002de05766 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 13:50:37 -0400 Subject: [PATCH 26/28] Update Releases.md --- docs/Releases.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Releases.md b/docs/Releases.md index 7aff221e3..d5c0098fd 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -25,7 +25,7 @@ A CloudLab template is available with the latest release here: https://www.cloud ### Shared Core Mode: This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores. NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep. Shared core variables for mgr are in the `nf_wakeup_info` structs, the NF shared core vars were moved to the `onvm_nf` struct. -The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. +The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-Core NFs. However, we have recently released a full version of the NFVNice system as an experimental branch, which can be found [here][nfvnice_branch]. Usage and implementation details can be found [here][shared_core_docs]. @@ -617,3 +617,4 @@ Initial source code release. [firewall_nf_readme]: ../examples/firewall/README.md [payload_scan_nf_readme]: ../examples/payload_scan/README.md [shared_core_docs]: ./NF_Dev.md#shared-cpu-mode +[nfvnice_branch]: https://github.com/sdnfv/openNetVM/tree/experimental/nfvnice-reinforce From 88a969edcd088c0272335a9af8c5dc6a2352affd Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 13:53:07 -0400 Subject: [PATCH 27/28] Update NF_Dev.md --- docs/NF_Dev.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/NF_Dev.md b/docs/NF_Dev.md index c22095d52..389080997 100644 --- a/docs/NF_Dev.md +++ b/docs/NF_Dev.md @@ -47,7 +47,7 @@ This is an **EXPERIMENTAL** mode for OpenNetVM. It allows multiple NFs to run on This code allows you to evaluate resource management techniques for NFs that share cores, however it has not been fully tested with complex NFs, therefore if you encounter any bugs please create an issue or a pull request with a proposed fix. -The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-CPU NFs. +The code is based on the hybrid-polling model proposed in [_Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization_ by Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, and Timothy Wood, published at _Co-NEXT 16_][flurries_paper] and extended in [_NFVnice: Dynamic Backpressure and Scheduling for NFV Service Chains_ by Sameer G. Kulkarni, Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K. K. Ramakrishnan, Timothy Wood, Mayutan Arumaithurai and Xiaoming Fu, published at _SIGCOMM '17_][nfvnice_paper]. Note that this code does not contain the full Flurries or NFVnice systems, only the basic support for shared-Core NFs. However, we have recently released a full version of the NFVNice system as an experimental branch, which can be found [here][nfvnice_branch]. Usage / Known Limitations: - To enable pass a `-c` flag to the onvm_mgr, and use a `-s` flag when starting a NF to specify that they want to share cores @@ -119,3 +119,4 @@ values. [msg_passing]: ../onvm/onvm_nflib/onvm_msg_common.h [flurries_paper]: https://dl.acm.org/citation.cfm?id=2999602 [nfvnice_paper]: https://dl.acm.org/citation.cfm?id=3098828 +[nfvnice_branch]: https://github.com/sdnfv/openNetVM/tree/experimental/nfvnice-reinforce From 7db7976b229b02556147f5bdfd72a4e34035ba00 Mon Sep 17 00:00:00 2001 From: Mykola Yurchenko Date: Tue, 4 Jun 2019 13:53:35 -0400 Subject: [PATCH 28/28] Update Releases.md --- docs/Releases.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Releases.md b/docs/Releases.md index d5c0098fd..e69161747 100644 --- a/docs/Releases.md +++ b/docs/Releases.md @@ -618,3 +618,5 @@ Initial source code release. [payload_scan_nf_readme]: ../examples/payload_scan/README.md [shared_core_docs]: ./NF_Dev.md#shared-cpu-mode [nfvnice_branch]: https://github.com/sdnfv/openNetVM/tree/experimental/nfvnice-reinforce +[flurries_paper]: https://dl.acm.org/citation.cfm?id=2999602 +[nfvnice_paper]: https://dl.acm.org/citation.cfm?id=3098828