Skip to content

OpenNetVM v19.05#134

Merged
koolzz merged 48 commits intomasterfrom
develop
Jun 4, 2019
Merged

OpenNetVM v19.05#134
koolzz merged 48 commits intomasterfrom
develop

Conversation

@koolzz
Copy link
Copy Markdown
Member

@koolzz koolzz commented Jun 4, 2019

This PR is used to merge all features for OpenNetVM v19.05 into master

kevindweb and others added 30 commits March 17, 2019 14:27
Fixes onvm web mode memory leak by properly freeing the cJSON objects upon exiting. cJSON objects are destroyed with helper functions from `/onvm/lib/cJSON.c`. 

Commit log:

* Free memory from JSON objects from web stats

* Fixed logic of JSON object memory free'ing

* Delete extra line

* Spaces > tabs

* Fixed style issues
In the `Install.md` installing libnuma-dev might fail if your system is not up-to-date. Added a note about running `sudo apt-get libnuma-dev` to fix this.

Commit log:

* Update Install.md
Adds debugging symbols setup troubleshooting to `Debug.md`.

Commit log:

* Update Debug.md
Modifies necessary files to make ONVM compatible with Ubuntu 18.04.1 LTS.

With the new Ubuntu version comes an upgraded version of gcc, which has stricter checks in place with regards to strict aliasing, formatting, etc. This pull request modifies code where these errors occurred.

Commit log:

* Updating makefiles

* Updating makefiles for gcc 7.3.0

* Update Makefile

* Updating Makefiles for Ubuntu 18.04.1 LTS compatibility

* Updating Makefiles

* Update Makefile

* Update onvm_common.h

* Update onvm_common.h

* Update onvm_common.h

* Update onvm_common.h

* Update cJSON.c

* Update onvm_nflib.c

* Update onvm_nflib.c

* testing

* Update cJSON.c

* tests

* testing

* Update Makefile

* Update onvm_init.c

* Update onvm_init.c

* Update onvm_init.c

* Update onvm_init.c

* Update onvm_init.c

* Update onvm_init.c

* Remove no changes

* Removing modified pr file

* Removing modified pr file

* Removing modified pr file

* Removing modified pr file

* Update pktgen-dpdk

* Revert "Update pktgen-dpdk"

This reverts commit aae8018.

* Update pktgen-dpdk

* Revert "Update pktgen-dpdk"

This reverts commit 1099ed8.

* Update pktgen-dpdk

* Revert "Update pktgen-dpdk"

This reverts commit 356de57.

* Update cJSON.c

* Fallthrough comments

* Update fallthrough comments
Fixed all style issues by running clang-format with a slightly modified Google format style. Added the `.clang-format` file for possible future usage.

Commit log:

* Adding clang format file

* Format onvm_mgr code

* Format onvm_nflib code

* Format examples code

* Remove extra semicolon

* Fix style on if/else

* Fix .h file ordering issue

* Changed sprintf to snprintf
Adds TTL and packet limit flags to stop the NF or the onvm_mgr based on time since startup or based on packets received. Default measurements for these flags are in second and in millions of packets. 

Commit log:

* Implemented ttl for pkts/ms

* Clean up for ttl

* Add manager ttl, fix time ttl for NFs

 - Adds a ttl flag to the manager (-t)

* Added examples/docs for mgr ttl

* Clarify flags

* Change TTL to seconds

* Fix ttl example in go script

* Minor fixes

* Fix if check

* Add mgr packet limit flag

* Added -l flag to go.sh

* Fix style spacing issues

* Update onvm_nflib.c

* Added ttl to adv rings, cleanup

* Clarify that ttl is disabled by default

* Cleanup, optimization, docs

* Fix pkt limit bug in onvm_mgr
Before this commit when the NF failed startup (for example launched with invalid arguments) it wouldn't properly cleanup as it wasn't considered RUNNING. But the STARTED NFs already had an allocated core, thus this would corrupt memory and onvm_mgr would consider cores to be busy when a NF actually didn't start. This fixes NF termination and does different levels of cleanup based on NF state.

Commit log:

* Fix bad cleanup on NF termination

* Merge branch 'develop' of https://github.com/sdnfv/openNetVM into 
bugfix_free_core_on_termination
Updates install script hugepage setup and kernel driver installation with functions from the `dpdk-setup.sh` script.

Summary:
 - Fixes huge page setup to the install.sh and setup_environment.sh scripts. 
 - Sets memory for all NUMA nodes, instead of just node0 
 - Removes the old igb_uio module on install.sh

Commit log:

* Update install script (hp/igb_uio)

* Fixed typo in remove_mnt_huge
Adds the ability for NFs to send messages to other NFs. NFs need to
define a message handler to receive messages and are responsible to
free the custom message data. If the message is sent to a NF that
doesn't have a message handler the message is ignored.

Commit log:

* Implemented NF to NF msg passing

* Fix style nit

* Fix ring enqueue to multi-prod

* Fix white space issue
Add a note about using the `dpdk/usertools/dpdk-setup.sh`  script into
the install documentation as an alternative to the provided
`scripts/install.sh` script for installing DPDK.

Commit log:

* Document dpdk tool for cleaning up huge pages

* Suggest different onvm script

* Rework troubleshooting to suggest manual DPDK configuration.

* Update Number of Hugepages
This PR adds the NF core ID value into the web stats page, the core ID
field is displayed next to the NF service and instance IDs.

Commit log:

* Add core number to web stats

* Reverted back markdown files

* Rebuilt and minified content

* Updated minification process

* Testing minify

* Removed husky

* Updated README with build information
Change the `./tools/dpdk_nic_bind.py` to `./usertools/dpdk-devbind.py`,
which reflects the latest DPDK version.

Commit log:

* Update dpdk script name

* Other minor Install doc updates
This change integrates the CI pipeline introduced in e2220e5 to the public repo.

To achieve this, the CI rig in the aforementioned commit has been updated with the following changes:

    * Use GitHub secret token to validate all requests coming to CI rig through webhook
    * Encrypt GitHub secret token at rest.  Decrypt secret token when initializing server to use for request validation
    * Requests must come from GitHub IPs
    * If request comes from OpenNetVM repo, then the user must be in a whitelist
    * Adds audit logs to identify unauthorized requests

Commit log:

* Add security checks, extend CI to public repo

* Added authorized users

* Authorized user log

* Add secret encryption

* Add logging

* Better logging, optimized encryption

* Fix unused var
This PR fixes packet generation the Speed Tester NF, with this change
if we can't generate the X requested packets and we only generate Z
packets we will enqueue Z packets. Before we enqueued X packets,
causing a SEGFAULT because some packets would be NULL.

Commit log:

* Packet number bug fix

* Suggested fixes for review

* Had to fix both conditionals

* More code style fixes

* Fixed placement of counter initialization

* Check for uninitialized packets

* Fix style, add space

* Style nit, I need to train my eyes
- Change line limit to 120 chars
- Add linting of .cpp and .h files
- Fix minor style issues from the linter adjustment

Commit log:

* Enfore 120 line length in linter

* Change default linelen

* Fix some line length changes

* Style fix

* Add lint for cpp .h files
Previously, we have been converting IP's in string form to IPv4 format
in reverse order, this PR modifies this to parse in proper format
`IPv4(ip[0], ip[1], ip[2], ip[3]`. And adds changes to the NFs that
were using this function.

Commit log:

* Modifying pkt helper file and nf_router

* Modifying pkt helper file

* Style fix

* Style fix

* Reverse ip fix (all Nf's)

* load generator fix

* Remove pcap file

* Small fix to load balancer
Before the tag was stored on the stack, which wasn't readable from the
onvm_mgr. This PR changes this and allocates memory for the NF tag,
thus it can now be used by the onvm_mgr.

Commit log:

* Fix nf tag by mallocing memry

* Change to non const char * and fix mem cleanup

* Fix speed tester tag
Fix stile for header guards, newline braces and whitespace.

Commit log:

* Add some linter fixes

* Fix a few more header guard styles

* Fixes to linter errors
The Payload Scan NF provides the functionality to search for a string
within a given UDP or TCP packet payload. Packet is forwarded to its
destination NF on a match, dropped otherwise.

Commit log:

* Payload scanner

* Update payload_nf.c

* Payload update

* Updating makefile, payload name

* Editing tags

* Removing duplicate nf

* Editing the gitignore file

* Linter stuff

* Initialize struct members to 0

* Style nit- spaces

* Adding inverse option (-i)

* Inverse option fix, style fixes

* Adding the payload_scan into the Makefile

* Style edit

* Read me

* Read me

* Putting struct in nf_info->data

* Fixes to global

* Read me fix

* Update load_balancer.c

* Oops, wrong branch

* Style fixes

* Style fixes (pkt_total)

* Zmalloc, style, changed back to counter (till I figure out a better solution)
This change frees structs used for NF tx that were allocated on NF
startup but not properly cleaned on nf shutdown. Also includes some
code cleanup and error checking.

Commit log:

* Proper nf struct cleanup
Add a "Core Mappings" page to openNetVM web stats.This view shows both
onvm_mgr and NF core usage,  which is displayed in a card view format.
The cards have the name and id of the running NFs but also provide a
link to the full page.

Commit log:

* Initial commit to react core page

* Fixed nf termination core issue

* Fixed husky git hook

* Get manager cores from tx, rx, and master

* Fixed stats event handler

* Fixed comment issue

* Tab nit

* Updated event type and logic

* Fix nf tag by mallocing memry

* Change to non const char * and fix mem cleanup

* Fixed onvm manager stats bugs

* Propogated fixes from changing manager stats

* Style nits and termination fixes

* Fixed core page NF teardown

* Don't overwrite develop comment

* Style nits and memory fix
Reuse instance IDs of old NFs that have terminated. Reusing will only
happen when the next instance id counter hits the set MAX_NFS limit.
Allows to infinitely run NFs and makes MAX_NFS a value of concurrent
running NFs instead.

Commit log:

* Reuse NF instance IDs

* Minor cleanup
A few links in the style docs were outdated and didn't properly open,
this PR fixes them.

Commit log:

* Fixing the broken style guide link

Updating the link in the style guide

* Update broken style guide
Previously if ONVM_HOME was not set, running make would fail without
properly warning the user about the error. With this PR the Makefile
will first check if the environment variable is set before attempting
to compile ONVM.

Commit log:

* Check if ONVM_HOME is set
Before, NFs' DPDK rings were not properly cleaned up during the shutdown process.  Since the rings were not being freed, ONVM manager could not receive traffic if NFs were stopped before network traffic was fully processed.

This change updates the NF shutdown logic to free data on each DPDK ring that the NF uses.

Commit log:

* Add ring cleanup on shutdown
Before, when running the advanced rings mode in speed tester and scaling, closing the manager would not kill the NF running.

This adds a message handler in the advanced rings loop that listens for the MSG_STOP macro sent by the manager.

Closes #105 

Commit log:

* Listen for stop message

* Removing extra line

* Style fixes

* Avoiding memleak

* Style updates

* Style fixes

* Style nits

* Style nits

* Style nits
The firewall NF drops or forwards packets based on rules provided in
the json file. This is achieved using DPDK's LPM (longest prefix
matching) library. Default behavior is to drop a packet unless the
packet matches a rule. The NF also has a debug mode to print decisions
for every packet and an inverse match mode where default behavior is to
forward a packet if it is not found in the table.

Commit log:

* Syncing with current

* Updates for ubuntu 18

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Updates

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update onvm_config_common.c

* Update onvm_config_common.c

* Updates

* Update onvm_config_common.c

* Updates

* updates

* Update onvm_config_common.c

* Update onvm_config_common.c

* Update firewall.c

* Update firewall.c

* test

* Update firewall.c

* test

* Update firewall.c

* Update firewall.c

* test

* Update firewall.c

* Update firewall.c

* Update onvm_config_common.c

* updates

* Update firewall.c

* Update firewall.c

* Update rules.json

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* test

* updates

* updates

* Update go.sh

* Updates

* Revert "Updates"

This reverts commit 120b624.

* Update firewall.c

* updates

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* updates

* Update onvm_config_common.c

* Update onvm_config_common.c

* Update firewall.c

* Delete rules.json

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update README.md

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update onvm_nflib.c

* Update onvm_nflib.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Update onvm_nf.c

* modifying go script, removing unnecessary code

* Update firewall.c

* Update firewall.c

* Update firewall.c

* Style fixes

* more style fixes

* More style fixes

* Moving rules file, logic for if running from examples directory

* Moving rules.json file

* Delete rules.json from examples direc

* Nuking custom go.sh script

* Update README.md

adding -f to the readme

* Adding pkt total, drop, accepted stats

* fixing readme

* documentation and style fixes

* bug fix: logic of dropping/accepting packets

* bug fix: logic of dropping/accepting packets

* style: change to 8 space indents

* Multiple run support

* style updates

* added not ipv4 field

* specify directory to rules.json file instead of specific name

* IP's are now entered in proper format in rules.json

* printing IP's in debug mode

* Style fixes

* Style fixes

* Style fixes/double free/corruption error

* Style fixes/double free/corruption error

* style

* Revert "style"

This reverts commit b2753a6.

* Style fixes

* Style fixes/whitespace

* spacing fix

* spacing fix

* Space fix

* Double free/corruption fix

* Fixed depth issue

* Style fixes

* Read me updates

* Parsing ip as cpu

* Compatibility with onvm_pkt_parse_ip PR merge

* Small fix in load_balancer

* Remove unused fw accept tag

* Creating function for char to ip

* Adding a function to parse string to ip

* Style nit

* Style nit

* Style nits and TODO

* TODO

* ip_disp to ip_string

* Restoring lpm struct (accidentally deleted)

* Restoring lpm status macro

* Style fixes, naming, etc

* Style fixes, naming

* Style fixes

* Style nit
Previously the launch script would butcher the arguments with spaces in
them, this PR fixes this issue for NF specific arguments (onvm/dpdk
don't have any string args with spaces)

Commit log:

* Fixed string arguments for go script

* new line

* Fixed variable name and config file launch

* Remove extra spacing
Adds a list of checks that CI runs through:
 - Check the branch (will reject if not develop)
 - Run performance check (will reject if < 97% of current avg)
 - Run linter, only on modified files (will reject if linter complains)

Commit log:

* Enfore 120 line length in linter

* Change default linelen

* Fix some line length changes

* Style fix

* Update CI to post review to PRs

* Fix post_msg typo

* Add dpdk base-virtaddr arg
This code introduces **EXPERIMENTAL** support to allow NFs to efficiently run on **shared** CPU cores.  NFs wait on semaphores when idle and are signaled by the manager when new packets arrive. Once the NF is in wake state, no additional notifications will be sent until it goes back to sleep.  Shared cpu variables for mgr are in the `nf_wakeup_info` structs, the NF shared cpu vars were moved to the `onvm_nf` struct.

 - The scaling/pthread model required modifications to the signal handling process. To accommodate proper signal handling and fix other issues, children will block all signals and only the parent will handle signals. When terminating the parent NF will tell all children to stop running, which in turn will turn their children to stop running, which in turn will tell... this ensures that the parent thread blocks until absolutely all descendants have exited (this is required because even though a pthread child is detached if the main process exits it will be abruptly killed)
 - Creating an `onvm_nf_context` to be malloced before nflib starts and has info about that specific NF (shutdown information/nf and nf_info structs). This struct replaces `onvm_nf_info` in most API calls. _In the next pr `onvm_nf_info` will be reworked into a nf initialization structure, more on that later_
 - Running the `onvm_nflib_start_signal_handler(nf_context, NULL);` call before `onvm_nflib_init` to ensure proper signal handling even on NF startup (pretty sure that resolves #93)
 - `onvm_nflib_stop` is now executed from the main(), instead of inside `onvm_nflib_run`. Also, the `onvm_nflib_stop` will be automatically called for the children spawned by the scaling API.
  
Other related changes:

 - Stats `-v` mode gets an update with the shared cpu information if such is available
 - onvm gets a config structure which contains onvm configuration flags for now but might be expanded later. This will be stored in a memzone and shared with the NFs.

**API additions:**

 - `onvm_nflib_start_signal_handler(struct onvm_nf_context *nf_context, handle_signal_func signal_hanlder);`

**API changes:**

 - `onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_info **nf_info_p);` -> `onvm_nflib_init(int argc, char *argv[], const char *nf_tag, struct onvm_nf_context *nf_context);`
 - `onvm_nflib_run_callback(struct onvm_nf_info *info, pkt_handler_func pkt_handler, callback_handler_func callback_handler);` -> `onvm_nflib_run_callback(struct onvm_nf_context *nf_context, pkt_handler_func pkt_handler, callback_handler_func callback_handler);`
 - `onvm_nflib_run(struct onvm_nf_info *info, pkt_handler_func pkt_handler);` -> `onvm_nflib_run(struct onvm_nf_context *nf_context, pkt_handler_func pkt_handler);`
 - `onvm_nflib_stop(struct onvm_nf_info *nf_info);` -> `onvm_nflib_stop(struct onvm_nf_context *nf_context);`

All code changes are featurized using INTERRUPT_SEM macro.

See the ONVM README.md file for more information and warnings.

Commit log:

* Shared CPU NFs (#150)

* merges rebased to master with #147

* reverting num_clients update.

* replace tabs with spaces

* Readme describing experimental shared CPU support

* Add usage and known limitations for shared CPU code

* Bug Fix: Fixes instance ID assignment race condition (#183) (#185)

Fixes a race condition in NF instance ID assignment.  Before, if two NFs were started at the same time, they would be assigned the same instance ID.  This change modifies logic in `onvm_nf_next_instance_id` to avoid assigning the same ID to two starting NFs.

* Bug Fix: Fixes instance ID assignment race condition
  - Fixes #178 
  - Changed incrementation of next_instance_id in onvm_mgr/onvm_nf.c
such that the ID can't be reused before an NF becomes active.

* partial refactoring of api w accordance of nf_info

* onvm mgr compiles

* flow table compiles -- all examples compile

* changed onvm_get_rx and onvm_get_tx to get rx_q and tx_q associated with onvm_nf struct

* Did requested changes. Also updated newer NFs (such as arp_response and flow_tracker). Also did some small style changes in  some other documentation in onvm_nflib.h

* Corrected some incorrect references to nflib functions in NF_Dev.md

* removed typing error

* Minor cleanup

* Stats cleanup

 - Some stats were unused and broken(?) nuked them
 - added stats that are possible usefull to stdout (slightly ugly)

* Add functionality to scale same NF instances

* Fixed a few global vars for multithreading to work

* Code clean up, comments and style fixes

* Added scale return value

* Fixed nf_mode for running advanced rings

* Refactored the onvm_nf/onvm_nf_info passing

- Keeping onvm_nf_info small, moved vars to onvm_nf
- Added a callback function for the scaling child thread
- Cleaned up code

* Adv rings NF scaling, setup func for spawned NFS

- Adds ability to scale NFs with advanced rings
- Adds a setup function for NFs
- More docs and cleanup

* Fixed build errors, changed NF typedef func names

* Launching NFs with different state/pkt processing

 -  Moving nf_info as passed arg to packet_handler -> allows for spawned NF to access their own nf_info
 -  Launching new NFs now accepts arguments for packet_handler function, setup function, callback function, advanced rings function
 -  Add a data pointer to onvm_nflib_info to maintain state
 -  Working on a scaling example (currently testing code, not in usable state) will showcase different features of spawning NFs

* Major changes, init breakdown, argv for spawned nf

- This is a major change, a lot of nflib initlization is refactored to allow spawned NFs to launch with their own service id
- There is a bug when the spawn a lot of new NFs we can no longer retrieve the nf_info from nf_info_mp, this needs to be fixed as it breaks stuff

* Fixing bugs, cleaning up code, updating example NF

- Fixed the nf_info bug where the mempool would get messed up as children weren't cleared properly.
- Updated the example scaling NF
- Removed most of the debugging code

* Clean up of shared cpu branch

Cleaned up messy bits, replacing macro branches with if statements, encapsulating code in functions, and getting rid of unnecessary code.

* The code cleaning commit has arrived

 - Style fixes
 - Removing debug code
 - Making code prettier
 - Adding better docs
 - Implementing requested changes

* Minor cleanup/whitespace fixes

* Fix Large ServieNum port crash bug

* Implementing requested changes

* Update scaling  docs

* Doc typos fixed

* Doc fixes

* Performace fixes

 - Changed MAX_NFS back to 128, improves performance
 - Changed the order inside of the onvm_nf struct, due to struct packing this reduced the overall size of onvm_nf down to 168 from 176

* Fixed memzone init size issue

* Update Releases.md

* Fix space

* initial pthread testing

* Small fixes

* small fixes

* More minor fixes

* More tiny fixes

* Scaling nf fixes

* Small fixes

* Logging, README fixes

* Added Manual/MGR core assignment mode, cleanup

* Added dedicated core option, error codes

* Combining shared cpu & pthreads

 - Dedicated pthread to watch for signals, this ensures that when user sends a SIGINT the threads are properly woken up and cleaned up. Not possible with the previous signal handler as it didn't have NF infromation

* Implemented shared cpu mutex logic for adv rings

* Add hacky shared cpu fix

* Fix some unused/old code

* Minor fixes, cleanup

* Remove testing print statement

* Remove unused func

* Fix shutdown thread join and cleanup

* Shared CPU mode as an argument

 - Add shared cpu '-c' flag passed to onvm_mgr
 - Add custom flags memzone
 - Surround all shared cpu code with if(shared_cpu)
 - Overall cleanup

* Update README.md

* Update NF_Dev.md

* Fix onvm_manager README

* Scaling nf share cpu flag

* Style fixes

* Cleanup

* Code cleanup

* Cleanup

* Onvm config struct for flags

* Refactoring and things

* Remove unused vars, rename var

* Fix stats & remove unrelated code

* Style fixes

* Fix imports in speed tester

* Fix race by properly killing grandchildren

* Rename vars, cleanup, comments

* Final cleanup

* Minore style fixes

* Minore style fixes

* Only cleanup shm when in shared cpu mode

* Fix unused macro

* Fix flag_p name to sleep_state

* Fix comment about signal thread

* Fix some style issues

* Fix scaling adv ring children termination

* Fix sleep issue with shared cpu scaling

* Fix cleanup, logical redesign

* The bug is strong with this one

* Fix type

* Pre-init signaling thread

* Better pre-init termination

* Working on combining signal handlers

* Remove unused dpdk_init_finished var

* Rewrote some termination logic

* Add defines in header file

* Fix signal handling, add nf context (api changes)

 - For proper termination of both advanced rings and normal NFs create
   nf context which is malloced as opposed to rte_malloc and is created
   prior to nflib init.
 - To properly handle signals and simplify the process instead of using
   a dedicated pthread, use signal() + global termination context
 - Involves a bunch of API changes to handle the new context variable,
   some API changes are still missing

* Cleanup comments, remove unused vars

* Further cleanup

* Cleanup

* More cleanup/updates

* Resolve needed signaling issues

* Update NFs to new API

* Fix fake getopts hack

* Add custom nf specific sig handling

* Update docs with paper description

* Update NF_Dev.md

* Update NF_Dev.md

* update references to research papers

* Remove debug pring

* Make children_cnt atomic

* Fix docs and comments

* Fix style issues

* Redo termination cleanup

* Removed ugly flag, proposed by the one and only @nks5295

* Fixed API calls for all example NFs

* Remove extra onvm_nflib_stop call in SpeedTester

* Add time based stop

* Fix Basic Monitor NF

* Fixed iter cnt for shutdown

* Fix comment

* Make keep_running atomic
koolzz and others added 13 commits May 31, 2019 01:46
This PR merges together onvm_nf_info and onvm_nf structs, into a single onvm_nf struct with internal variable groupings for readability, also renames key structs.

Summary:

The old `onvm_nf_info` is re purposed as `onvm_nf_init_cfg` a struct used to initialize the NF with the manager. This struct is passed to onvm_mgr and is then freed when initialization is complete.
In attempt to merge these structs the usage of `onvm_nf_info` has been removed from all internal and external API calls. Excluding the startup sequence.
The struct `onvm_nf_context` was renamed to `onvm_nf_local_ctx`
Added a `nf_function_table` struct that will hold all the NF defined callback functions.
Changed all NF facing API calls to pass `onvm_nf_local_ctx` instead of onvm_nf as NFs might need access to the keep_running variable.

Commit log:

* Remove bootstrap struct from context

* Nuke some extra code

* Fix firewall API

* Remove unused globals in example NFs

* Fix style nit

* Fix double define

* Global rename/regrouping of onvm_nf struct

* Restructure scaling logic

* Rename nf_context->nf_local_ctx

* Further cleanup of scaling API

* Update method name

* Rename callbackhandlerfunc -> callback_func

* Fix linter

* Remove trailing whitespace

* Fix docs

* Fix more docs

* Free fix, code cleanup

* Set pointer to NULL on cleanup

* Testing function table struct

* Massive rename, API fixes, remove circular refr

* Don't mind me fixin whitespace

* Fix speed tester double func_table init bug

* Enforce 120 char limit

* Rem trailing whitespace

* Rem extra api call in speed_tester

* Fix docs

* Huuuuge linter fix, well tested
This PR adds memory cleanup for onvm_mgr allocated structs.

Commit log:

* Free structs calloced in main

* Free another struct
This PR adds functions that will generate tcp/udp packets with the
provided protocol headers. Also includes helper functions to swap the
values in ether/ip/tcp/udp headers.

Commit log:

* facilitate packet creation with udp packet helper functions

* Fix: return packet instead of NULL

* Add tcp functions from nick's server

* Use linter properly

* Fix compilation error due to redefinition

* format header file

* Add docs, fix checksum calculation
This PR features additions and fixes to the onvm_mgr console stats:
 - Adds NF tag, parent id, core id, children count
 - Shortens the NF state to a single upper case
   letter - S (sleeping) or W (working)

Commit log:

* Added nf tag and core to stats

* Made constants for the stats in the header file

* Fixed verbose stat line-up

* Fixed service id totals stats

* Fixed style of output

* Fixed shared cpu verbose

* Moved print formatters to onvm_stats.h

* Fixed up stat lines

* Fixed macro styling
Due to an accident when merging there was a double decrement of the
NF parent's children count on NF shutdown. This pr resolves the bug.

Commit log:

* Fixed double decrement for children count
This PR changes CI lint check to report errors only for the lines
changed in the PR as opposed to reporting errors for the entire files.

Commit log:

* Fixed run_linter function in ci helper

* Fixed variable initialization

* Commented spaghetti mess and added loop break

* Fixed styling and comments
This PR updates an API call in a place long forgotten, the land of the
Speed Tester NF PCAP replaying.

Commit log:

* Fix pcap send in speed_tester
Add a call to `onvm_nflib_stop` before doing an `rte_exit` to properly
cleanup NF information and signal onvm_mgr that the NF is stopping.

Commit log:

* Fix pcap send in speed_tester

* Fixed nf cleanup on errors

* I know ci will complain
This PR properly clamps the tag to the max char limit and fixes an ASCII char print count to properly align console stats.

Commit log:

* Also tiny style fix for stats

* Fix stats linup error

* Fix tag clamping
The previous name doesn't fit well because NFs can share the same CPU
but not necessarily share cores as CPUs often have 2 or more cores. A
better fitting name is shared core mode. 

Affected macro name changes: 
- ONVM_ENABLE_SHARED_CPU -> ONVM_NF_SHARE_CORES

Commit log:

* Rename shared cpu to shared core

* Fix Readme

* Fix naming in stats

* Fix Stats Printing for shared core
This PR swaps the parent id and children count attribute in stats so
that it matches the description header. 

Commit log:

* Change print order
Currently the advanced ring support in onvm is not clean, this pr attempts to resolve this by doing the following:

* Removes unused APIs
* Removes support for APIs written for advanced rings (see advanced rings scaling) that would never actually be used by the developers, because if developers are using advanced rings the assumption is they will also want to manage their own threads etc.
* Removes Advanced Rings mode from the Speed tester NF
* Cleans up and redo the adv ring mode in the Scaling NF; the new implementation now does its own pthread creation instead of relying on onvm scaling APIs. Also make a clear separation between default and advanced ring mode.
* Because of the adv rings changes some internal nflib APIs were exposed to the NF (onvm_nflib_start_nf, onvm_nflib_init_nf_init_cfg, onvm_nflib_inherit_parent_init_cfg)

Commit log:

* WIP advanced rings rework

* Remove deprecated adv rings vars

* Add proper cleanup for adv rings example

* Ignore this one

* Add sig handling global for adv rings

* Refactor, remove data pointer

* Cleanup, doc fixes

* Refactor scaling into 2 modes

* Cleanup some scaling code

* Sneaking in quick doc fix in here

* Fix mgr termination
This PR adds an atomic flag to `nf_local_ctx` to prevent double
shutdown cleanup.

Commit log:

* Add atomic shutdown flag to context
@onvm
Copy link
Copy Markdown

onvm commented Jun 4, 2019

In response to PR creation

CI Message

Your results will arrive shortly

Copy link
Copy Markdown

@onvm onvm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In response to PR creation

CI Message

Run successful see results:
❌ FIX PR submitted to master branch instead of develop
✔️ Speed tester performance check passed
✔️ Linter passed

[Results from nimbnode30]

  • Median TX pps for Speed Tester: 35142446
  • Performance rating - 100.41% (compared to 35000000 average)

@koolzz
Copy link
Copy Markdown
Member Author

koolzz commented Jun 4, 2019

DON"T SQUASH, NORMAL MERGE ONLY

@koolzz koolzz changed the title OpenNetVM v19.05 Release OpenNetVM v19.05 Jun 4, 2019
koolzz added 5 commits June 4, 2019 00:30
This PR fixes all licences so that they actually end on 2019

Commit log:

* Bump all licences by hand

* Hand picked license changes @nks5295, no cheating
Fix a few header inconsistencies and add stats documentation.

Commit log:

* Fix extra verbose stats header

* Fix comas, clean up defs

* Update README.md
This PR changes our init sequence to actually return error codes
instead of calling rte_exit, when something goes wrong.

Commit log:

* Fix shutdown when onvm_mgr isn't alive

* Sleep after the print statement
This PR adds release notes for upcoming v19.05 release

Commit log:

* Update Release Notes

Initial commit, still a lot to fill in

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Fix grammar errors, expand some sections

* Update Releases.md

* Update Releases.md

* Add API changes

* Add links to docs

* Update Releases.md

* Move API changes down

* Update Releases.md

* Update Releases.md

* Fix suggested grammar/whitespace issues

Spelling is hard

* Update API section

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update Releases.md

* Update NF_Dev.md

* Update Releases.md
This PR explains how to fix the dpdk's `Cannot mmap memory for
rte_config at [0x7ffff7ff3000], got [0x7ffff7ff2000] - please use
'--base-virtaddr'` error.

Commit log:

* Add small troubleshooting note

* Update Releases.md
@koolzz koolzz merged commit 46bbc96 into master Jun 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants