Snabb 2023.10 “Enigma”

This release pulls in recent improvements and bug fixes from LuaJIT upstream, refines engine pacing, adds a snabb config shutdown command, and adds support for ConnectX-7 NICs.

What's Changed

• apps.rss, apps.ipfix: add timeline events by @eugeneia in #1508
• Fix engine.Hz mode by @eugeneia in #1509
• apps.mellanox: add support for ConnectX-7 by @alexandergall in #1511
• apps.mellanox.connectx: add support for PCAM register by @alexandergall in #1512
• Collected changes for October 2023 release “Enigma” by @eugeneia in #1515
• Implement shutdown command for the config program by @alexandergall in #1517
• Timed pace breathing by @eugeneia in #1510
• Sync with raptorjit August-2023 by @eugeneia in #1514
• lib.hardware.pci: fix pcall in shutdown() by @alexandergall in #1516

Full Changelog: v2023.06...v2023.10

Snabb 2023.06 “Davion”

This small summer release contains various improvements for Snabbflow (snabb ipfix) and a bug fix for Snabb lwAFTR.

What's Changed

• rss: fix missing local declaration in BPF filter construction by @alexandergall in #1504
• Collected changes for June 2023 release: Davion by @eugeneia in #1507
• lib.ipv4.reassemble: fix missing argument in call to reassembly_error by @eugeneia in #1500
• apps.ipfix.maps: re-create map ctable if source file has changed by @alexandergall in #1501
• rss.metadata: store the metadate in the packet headroom area by @alexandergall in #1502
• ipfix: add acquire-cpu option to YANG schema by @alexandergall in #1503
• ipfix: add HTTPS templates to extract the TLS SNI by @alexandergall in #1506

Full Changelog: v2023.04...v2023.06

Snabb 2023.04 “Caspar”

This release finally merges @lukego's timeline: a probabilistic high-frequency event log that is useful to understand and debug runtime latencies. It can be enabled via the environment variable SNABB_TIMELINE_ENABLE. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Snabbflow integration (timeline, group freelist and interlink sizes) by @eugeneia in #1497
• Timeline 2022 by @eugeneia in #1486
• core.group_freelist: fix a deadlock by @eugeneia in #1494
• lib.poptrie: fix bug where address bits where interpreted in reverse order by @eugeneia in #1495
• apps.rss: remove 59 (no-next-header) from list of IPv6 extension headers by @alexandergall in #1496
• lib.yang.list: fix bug in remove_obsolete_nodes by @eugeneia in #1498
• Collected changes for April 2023 release: Caspar by @eugeneia in #1499
• [sketch] Timeline redux by @lukego in #1098

Full Changelog: v2022.12...v2023.04

Contributors

Alexander Gall (1):

• 8e7be8e: apps.rss.metadata: remove 59 (no-next-header) from list of IPv6 extension headers

Luke Gorrie (10):

• 3e4c447: dasm_x86.lua: Add support for RDTSCP instruction
• 95872b1: core.timeline: New module for "timeline" logs
• 930f294: Makefile: Embed "*.events" timeline specs
• aa9980a: engine: Add engine.events, app.events, link.events
• 8530942: core.link: Add dropped bytes counter
• 9b252f0: engine: Instrument with timeline events
• 40ceef6: core.timeline: Switch to double-float on disk
• 563ae7a: engine: update timeline level probabilities
• ffb7379: engine: Randomize timeline log level with math
• 16a6bb2: engine: Remove timeline packet payload sampling

Max Rottenkolber (51):

• 1bb4108: core.app: remove remains from 16a6bb2 (timeline packet sampling)
• 0e52b24: core.timeline: make timeline log available to other core modules
• e834845: core.packet: record packet allocation events to timeline
• a6def7b: Revert "core.timeline: make timeline log available to other core modules"
• 8201d1d: lib.ptree.worker: reformulate to use engine.main with done func
• debd26c: engine.main: fix memory leak due to latency histogram creation
• edaad6c: core.app: set timeline log level at the very end of breathe loop
• 9c4c804: core.packet: reorder group_fl released/reclaimed/unlocked events
• 004f5bb: core.timeline: decouple log level from event rate
• 8e68109: core.app: initialize timeline log rate to 7 (no logging)
• 8f0c6bf: core.timeline: add switch to disable timeline logging
• 30cafdc: core.app: add timeline events for app network configuration
• e5faba0: Fix some wrong assumptions about the event rate
• 872eaab: app.events: reduce tick/ticked event rate
• 0252254: engine.events: fix breath_ticked description
• 8080451: core.app: log breath_ticked only when tick()=true
• cd43d69: engine.events: increase rate of commited_counters
• 8091531: Revert "app.events: reduce tick/ticked event rate"
• 3e0fee7: core.app: fix bug introduced in merge with master
• cb11a26: app.events: reduce tick/ticked event rate
• bfa3aa6: lib.ptree: fix bugs introduced by merge with master
• 8c34646: timeline: cleaup merge
• 840e1e5: timeline: cleanup
• db3c5cb: engine: add timeline_enabled toggle (on by default)
• 7b00bcf: Revert "engine: add timeline_enabled toggle (on by default)"
• 9896b4b: timeline: disable by default, add lib.scheduling toggle
• 21e2017: timeline: rework group freelist events
• b7a8af9: timeline: remove unused arguments to tick events
• 068b4d5: timeline: remove linkstats for push events
• f41f7a1: timeline: fixup rework group freelist events
• bcca06e: core.sync: add load primitive
• 45d3ac7: core.group_freelist: fix a deadlock
• b916131: group_freelist: revert panic
• f8c977c: group_freelist: add deadlock detection
• 2fbc94b: lib.poptrie: fix bug where address bits where interpreted in reverse order
• 67b3d9c: lib.poptrie: optimize x86 lookup
• a382b1d: core.timeline: fix selftest
• 00287bb: engine: allow timely timeline initialization
• a6fe6dc: snabb ipfix probe: add --timeline option to CLI
• 66f0eb3: lib.scheduling: add group_freelist_size setting
• f906776: lib.interlink: fix shm printing bug
• 50cad00: ipfix probe: make group freelist and interlink sizes configurable via schema
• 81a9e19: timeline: enable via environment variable
• f805e81: Revert "snabb ipfix probe: add --timeline option to CLI"
• 0916c24: Revert "engine: allow timely timeline initialization"
• 559c80d: core.sync: load64, cas64
• f47228b: core.group_freelist: use 64 bit sequence
• b6c130d: core.group_freelist: ensure finish is atomic
• e8eb4fd: Revert "group_freelist: add deadlock detection"
• f3905b3: core.group_freelist: cleanup
• 7472b83: lib.yang.list: fix bug in remove_obsolete_nodes

Snabb 2022.12 “Balaclava”

This release introduces a new data structure to represent YANG lists, allowing us to expose lists with the attribute ordered-by: user as a configuration interface. Additionally we have made previously hardcoded freelist and interlink buffer sizes configurable at runtime to enable tuning of those buffers to fit application demands. Finally, ipfix probe has been optimized to operate without software RSS when the configuration allows it. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Configurable interlink queue, freelist, and group freelist sizes by @eugeneia in #1489
• Add lib.yang.list hash trie to store YANG lists by @eugeneia in #1490
• ipfix probe: cleanup code, optimize operation without software rss by @eugeneia in #1492
• Collected changes for December 2022 release: Balaclava by @eugeneia in #1493

Full Changelog: v2022.10...v2022.12

Contributors

Max Rottenkolber (100):

• 4b58a3c: heap 1
• 536c929: heap 2
• 9ea357e: list 1
• 1afa4cd: heap no overflow, simplify, fix bugs
• c325cd9: heap collisions
• 62d4ae4: remove, iterator
• e451f5e: enable LUA52COMPAT
• 00a6169: fix bugs
• 0f8580e: ListMeta
• 2075954: add_or_update
• 2313cc0: fix heap _ref bug
• 559faf3: refactor
• b3692f5: refactor
• b60b918: ip bench
• 9c6a29e: refactor
• a6492c0: funny
• d8e53fe: ListMeta fixes
• cc7b972: Make keys/members specs extendable
• 89a67ac: todo: deterministic struct field order
• ddabaa3: lib.yang.list: support optional members
• 64331e4: lib.yang.list: sort fields in build_type
• 62696b7: lib.yang.list: more builtin types
• 95a9d31: lib.yang.list: add lvalue type (for arbitrary Lua values)
• abc0f72: lib.yang.list: add ipv4/ipv6-address types
• 147557f: lib.yang.list: single key meta, object() getter
• 8e1da16: lib.yang.list: load/save
• e36d84e: lib.yang.list: object() functions a is-list? predicate
• 4c90e38: lib.yang.list: add from_table constructor
• 4eca2e7: core.shm: pass through extra sizeof args in map
• f948fa6: lib.interlink: make queue size configurable
• f1577c9: core.shm: allow mapping of partial objects
• c7bb164: core.group_freelist: configurable group freelist size
• 7b6ee58: core.packet: configurable freelist size
• 1ac0362: core.packet: fix freelist shm display bugs
• 07dc242: lib.interlink: revert from typeof to cdef for struct interlink
• 9400c62: lib.scheduling: add max_packets parameter
• 690bd3e: lib.yang: integrate list into data
• 5d20d5a: lib.yang.list: clean up lvalues, load, save
• d728c32: lib.yang: integrate list into binary
• 86a8a32: lib.yang.list: fix bug wrt lvalues/optional
• 96ce235: lib.yang.binary: test list lvalues and fix bug
• dbc6f5e: lib.yang.list: add optional lvalue test case
• c5609e9: lib.yang.data: cleanup
• d5e508a: lib.yang: update README
• 1db6430: lib.yang.path_data: wip
• a66ee30: lib.yang.path_data: wip
• de2acbc: lib.yang.path: cleanup
• c3423ec: libya.ng.path_data: wip
• 63dde84: lib.yang.path: fix bugs
• d267f22: lib.yang.list: add identityref and leafref types
• 275da2a: lib.yang.path_data: wip
• 97a5899: lib.yang.path: add unparse_path
• ee4af97: lib.yang.path_data: wip passing
• adb40ef: lib.yang.path_data: fix bugs, extend tests
• e8c9725: lib.yang.list: integrate state, ptree support
• 6ff9f33: lib.yang.list: add struct support
• ab96b5d: list lwaftr wip
• 00ba121: lib.lua.StackTracePlus: pull upstream
• 5eebd6f: core.main: re-enable StackTracePlus
• 4bf08f8: lib.yang: factor out lib.yang.ctype
• bb9eca2: lib.yang.list: use lib.yang.ctype
• bc571d3: apps.lwaftr.binding_table: fixup
• f3c8e18: lib.yang.binary: serialize struct ts for list
• 8c8c25c: program.lwaftr.tests.probased.genyang: integrate
• a3c0e39: core.main: fallback to debug.traceback when STP fails
• 27a5ec9: integrate lwaftr wip fixup
• 990ab59: lib.yang.list: fix empty iterator bug
• 3cd57c2: lib.yang.state: fixup
• c764412: lib.yang.path/path_data: handle rpc schema
• 231381d: lib.yang: only ever handle list objects for list productions
• 34c4324: lib.yang.path_data: add grammar_for_schema_by_name
• 4fc54c2: integrate lwaftr wip
• 7ae62e7: lib.yang.list: use ctype if available
• 1145275: integrate lwaftr fix bugs
• 264bd6a: lib.equal: do not compare tables with metatables
• ad5c3a5: lwaftr integrate fix bugs
• c2261f2: lib.yang.data: test ordered-by
• 3858999: lib.yang.list: fix ordering bugs
• 308e56c: lib.yang.path_data: fix setter and remover bugs
• b04a261: lib.yang.list: fix bug in field_order
• 04c79b6: lib.yang.data: avoid redundantly optional list members
• 0059c0b: lib.yang.state: muffle warnings
• 4e56332: lwaftr: remove legacy schema and migrations
• b0ea121: lwaftr: supply mandatory padding value in native_binding_table_from_ietf
• 5bd3087: Integrate snabbflow
• 1896a2c: snabb ipfix: remove class order leaf
• 78f1a11: lib.yang.binary: support serializing leafrefs
• 68cd6c4: snabb-snabbflow-v1.yang: use leafrefs where sensible
• 1553134: lib.ptree: fix a bug where bogus PID was logged
• 5923add: ipfix probe: cleanup code, optimize operation without software rss
• 48a1747: program.ipfix.tests.test: tweak tolerance
• 004b6b8: lib.yang.binary: bumb version
• 052d8c2: lib.interlink: fix queue size race condition
• 37628fc: core.shm: test parameterized types
• acf5d80: apps.lwaftr.binding_table: keep ctable defaults originally used in lib.yang.data
• 84d59cd: lib.interlink: move size to same cache line as state
• 0183beb: lib.interlink: keep local copies of mask
• 3b3c3de: core.group_freelist: keep local copies of mask
• 176b126: lib.interlink: edits
• 58be8ee: core.group_freelist: edits

Snabb 2022.10 “Alpina”

This release introduces a YANG schema for the multi-core scalable IPFIX probe developed by @alexandergall at SWITCH, “Snabbflow”. Additionally we have written a tutorial for getting up and running with Snabb lwAFTR. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Fix lwAFTR trace performance issues (for release Alpina) by @eugeneia in #1488
• WIP: YANG schema for snabb ipfix (“Snabbflow”) by @eugeneia in #1481
• lwAFTR 100G related fixes by @eugeneia in #1482
• Tutorial: Getting Started with Snabb lwAFTR by @eugeneia in #1483
• apps.mellanox: use default enable_counters setting by @eugeneia in #1484
• core.app: fix a bug related to getting app zone name by @eugeneia in #1485
• Collected changes for October 2022 release: Alpina by @eugeneia in #1487

Full Changelog: v2022.07...v2022.10

Contributors

Alexander Gall (3):

• 101911a: interlink: fix shmem names in receiver/transmitter shutdown
• 56cdc1d: Adapt ipfix stats program to work with probe_ptree
• 89a218f: map_pci_memory(): set file path, increase timeout

Max Rottenkolber (97):

• a03d3c5: lib.yang: support empty leaf-lists
• 0ffd14c: lib.yang: fix bug where native number keys caused parsing to c(l)tables
• c501f78: lib.yang: fix bug where value_ctype was used in native_key tables
• 2f86ddf: Revert "lib.yang: fix bug where value_ctype was used in native_key tables"
• 402ecdf: lib.ptree.action_codec: support toplevel app classes and nil app args
• bbbea95: program.ipfix.lib: factor pure configure functions
• 507173f: sketch: ipfix probe_ptree
• ea636f4: apps.tap: continuously scan MAC and add overwrite_src_mac option
• 8a5d3e2: apps.tap: add forwarding option
• 11bbb49: program.ipfix: use Tap app features for setting exporter src MAC and sysctls
• c6ab421: App apps.snmp.iftable
• e3f4d77: program.ipfix: use apps.snmp to setup ifmibs
• 4bbd11e: lib.yang.data: do not derive bogus ctype for leaf-lists
• 247761a: lib.cpuset: fix assertion in add, add contains/remove/list methods
• d42c38f: lib.ptree: scan for 'pciaddress' field in acquire_cpu_for_worker
• 6cf6e2f: program.ipfix.probe_ptree: pin to cpus specified in config
• 8d04cd2: ipfix probe_ptree: update example config
• 069264f: ipfix probe_ptree: start with name
• c814c43: lib.ptree: add rudimentary worker-restarting facility
• f3fd7e7: ipfix probe_ptree: use lib.ptree worker restarts
• a757a8e: ipfix probe_ptree: log_level=INFO, rename schema variable
• 0996502: lib.ptree: manage worker restarts per worker configued via worker_opts
• 97bba23: ipfix probe_ptree: only restart dedicated exporter processes
• 2d61cfb: apps.tap: overwrite_dst_mac not overwrite_src_mac
• d585c2f: ljsyscall: fix bug in S.sysctl where we wrote at position ~= 0
• ab14ac3: ipfix probe_ptree: canonicalize singular list names in schema/config
• 88312f5: ipfoc probe_ptree: more schema cleanup
• 42a58ee: ipfix: probe_ptree: [wip] schema documentation
• 3a1055e: core.app: setvmprofile in tick
• aec36e6: ipfix probe_ptree: remove use-maps from schema
• 5490392: ipfix probe_ptree: remove instance weight leaf from schema
• 2fe881b: ipfix probe_ptree: add scan-protection/enable leaf
• 6635812: ipfix probe_tree: raise exporter MTU default to 1500
• 8169068: ipfix probe_tree: remove add-packet-metadata option
• 9a5d18e: ipfix probe_tree: more schema documentation
• c3ce608: ipfix probe_tree: allow maximum ConnectX recvq_size in schema
• 3630cf7: core.lib: equal: check object identity early
• 4571335: apps.ipfix: allow full reconfig while keeping flow sets intact
• 749ce28: ipfix probe_ptree: support configs without software RSS classes
• 7d7b0ed: apps.ipfix: fix bugs in counter initialization
• cfc5095: lwaftr: fix bug in migrate_device_on_config
• 192a0a3: lwaftr: tutorial wip
• b1d7eb9: lwaftr: tutorial wip 2
• 285f3f3: lwaftr: tutorial wip3
• b969b9d: lwaftr tutorial: flowchart
• a3a8029: lwaftr tutorial: edit lwaftr-veth-env.sh
• c76a58f: lib.ptree.inotify: turn warnings into debug messages behind flag
• 8e94ad8: lwaftr tutorial: supporting programs
• a5523ea: lib.ptree: support scalar value app configs in action_codec
• ff74f68: ipfix probe_ptree: test wip
• 4fea4ea: Revert "ipfix probe_ptree: test wip"
• ebc7413: ipfix probe_ptree: add state counters
• 3e5ac19: wip
• a416c42: wip
• b23fe45: apps.ipfix: assert templates shm doesn’t already exist
• 413cf19: wip
• 680a28b: wip get.state
• b26d015: lib.ptree: add acquire_cpu worker option
• ef3a002: ipfix probe_ptree: do not acquire cpu core for mellanox control worker
• 6eb81db: lib.scheduling/lib.ptree.worker: make jit.opt configurable
• 00112e1: ipfix probe_ptree: run workers with custom jit options and engine.busywait=false
• de3a225: ipfix probe_ptree: key rss groups / instances in snabbflow-state with id instead of pid
• 02dcd33: ipfix probe_ptree: device queue stats for rss groups
• acf52dd: ipfix probe_ptree: fix bug introduced in de3a225
• 1fd8e51: apps.connectx: fix stop() bugs
• f346467: ipfix probe_ptree: normalize device, app, and input link names
• 29dfa83: ipfix probe_ptree: guard against duplicate devices
• 5187a9e: ipfix probe_ptree: hide debug prints
• 89581cf: apps.mellanox.connectx: fix bug in IO:stop()
• 83b9f1e: apps.mellanox: use default enable_counters setting
• e2890f4: apps.mellanox: add test case for rxdrop and per-queue rxdrop counters
• c0a4963: core.app: fix a bug related to getting app zone name
• 00d2337: ipfix probe_ptree: warn on missing traffic class, throw error on missing exporter
• 61b67a3: Replace ipfix probe with probe_tree, remove probe_rss
• 22db66a: ipfix probe: add common scheduling CLI opts
• e930e4e: ipfix probe: enable VMProfile by default
• 7228010: apps.mellanox: make sync interval configurable, add :report()
• 5feed3d: packetblaster: support connectx, increase send desciptors in loadgen mode
• bfa68ba: program.config: do not exit on error in call_leader
• da48397: Add program/ipfix/tests/bench.snabb
• 81bd389: apps.basic: fix bug in Truncate
• 1ef5bee: packetblaster ipfix: generate flows from pcap on multiple cpu cores
• 2aae05b: ipfix probe: update bench.snabb to use packetblaster ipfix
• f2870ac: program/ipfix/tests: check in truncate_pcap script
• d64e23c: ipfix probe: add --rss-goups option to bench.snabb
• 97e9369: program.ipfix.lib: remove dead code
• c508b40: ipfix probe: add test.snabb
• 7c5ee4e: ipfix probe: update collector-test.sh
• a8c2f24: ipfix probe: remove generate_packets.py
• cdf1c3b: ipfix probe: cleanup tests
• 86e14ed: ipfix probe: update README.md
• 609716d: lwaftr tutorial: fix typos, reference schema instead of source
• 70785be: lwaftr tutorial: remove TODOs
• 5a4d0b1: apps.ipfix: fix bug in selftest
• 18d2631: snabb config listen: fix bug introduced in bfa68ba
• 5fe89b5: apps.lwaftr: do not flush lookups early
• 4706127: lib.ctable/lib.hash.siphash: optimize traces

Snabb 2022.07 “Parmigiano”

This release upstreams the multi-core scalable IPFIX probe developed by @alexandergall at SWITCH, “Snabbflow”. Additionally, we have extended the core.app interface to support the new tick and per-link push methods. Other notable changes include improved inter-process link scaling, and a new benchmark for our Connect-X driver to be used in CI. As usual this release also comes with various minor improvements and bug fixes.

What's Changed

• Upstream ipfix / snabbflow into max-next by @eugeneia in #1479
• Use MPMC bounded queue for group freelist by @eugeneia in #1472
• core.app 3.0: tick method, advanced app linking features by @eugeneia in #1475
• map_pci_memory: workaround for kernel race condition by @eugeneia in #1476
• lwaftr bench: build pcaps in generate-configuration, print summary by @eugeneia in #1477
• Collected changes for July 2022 release: Parmigiano by @eugeneia in #1478
• apps.mellanox.connectx: benchmark matrix by @eugeneia in #1469
• ConnectX: various driver enhancements by @eugeneia in #1473
• default.nix: remove obsolete brute-force shebang patching (breaks wit… by @eugeneia in #1474
• Fixes for lwAFTR in many-worker configurations by @eugeneia in #1480

Full Changelog: v2022.01.1...v2022.07

Contributors

Alexander Gall (105):

• 0f5068d: lj_trace.c: unpatch blacklisted bytecodes when flushing traces
• 5df67e8: lj_trace.c: Add penalty to trace abort message
• b19d442: intel_mp: fix range of array registers for i210
• 51214ea: intel_mp: add sanity checks for the number of rx/tx descriptors
• d4a9871: intel_mp: add queue enable counters, fix rxdrop counter
• fe709ed: lib.protocol.ipv4: avoid allocation for pseudo-header
• d15cbff: socket.raw: don't use select()
• 54780c2: ctable: fix power-of-2 issue
• 3002614: Add optional callback to the CTable:resize() method
• 0cda3d1: lib: disable tail-call optimization for endian conversions
• 62d0e72: Avoid factor-of-2 headroom in ctable
• 4c15c4e: Check bound of values in lib/lpm/lpm4_248.lua
• d4f8914: lib.lpm4_248: implement search in pure Lua
• 30b1e6f: Flush JIT traces after ctable resize
• d59aaff: Export flow before resetting fields
• 5ec2a64: Properly count records for IPFIX and NetFlow v9
• f4e93cd: Send full-sized export packets
• 556bc58: Get flow octet counter from IP header
• 246655c: Add tcpControlBits to flow sets, factorize transport processing
• 132f5b9: Allow multiple files as input for make_ipfix_element_map()
• e2587ae: Delegate exporting packets to FlowSets
• f40c1b1: Add counters
• 605dac0: Add sub-program to print stats for running ipfix processes
• a25d353: Add take_all() method to token bucket
• e0d93bc: Make table scan time configurable
• 30b7c4f: Support VLAN-tagged traffic, IPv6 extension headers
• b36e45b: Support PEN, strings and octetArray in templates
• d72d42d: Add optional state variables to flow cache value
• 46263a7: Add optional template-specific stats counters
• 57b71a0: Refactor template definitions
• 045f48e: Add templates for HTTP
• 2575d94: Add templates for DNS
• 9e6d48b: Add options to ipfix probe program, support routed tap interface
• 463022d: Add extended template
• 7f6c241: Switch to packet-metadata module from RSS
• 24a5063: Allow multiple IPFIX instances in the same process
• 8178577: Add probe_rss ipfix sub-program
• 84b486a: Disable map logging in extended template by default
• 7a9e3c8: Add SNMP Interface MIB for PCI devices
• 5b9e56e: program.ipfix: redesign RSS framework
• a15f4f3: program.ipfix: support hardware RSS in probe
• 2c05b4c: program.ipfix: fix stats sub-program for redesign
• 95ff0ca: program.ipfix: integrate mellanox driver
• 5151c96: program.ipfix: include PCI stats from control app
• 560e5b1: program.ipfix: increase receive buffers for mellanox driver
• a9e9f7f: program.ipfix: adapt to interlink semantics of v2018.06
• 28cca93: apps.ipfix: tune log messages
• 84cbf0d: apps.ipfix: fix selftests
• a2097f3: core.app: skip array indices when traversing input links
• 395a4f8: core.app: assert that links are connected only once
• 2c7929c: core.app: support link-specific push methods
• 565d7e6: apps.basic: apply link push methods to basic apps
• ea076d8: VlanMux: optimize tag pop/push
• bc36ddb: VlanMux: apply specialized push methods
• 1cfadb3: core.app: only call push() if the link is not empty
• 5fb9816: Add housekeeping() method to apps
• 233c894: Adapt to new versions of logger and token bucket
• be1f65c: apps.vhost: suppress -Wstringop-truncation
• 33eefac: lib.hardware.pci: add new device to supported Mellanox cards
• b655643: apps.ipfix, apps.rss: adapt to specialized push link semantics
• 40373df: program.iptix.probe_rss: add JIT configuration for RSS workers
• 14e32eb: apps.rss: mitigate unbiased branches in metadata handling
• c113670: apps.rss: explicitely unroll the loop over packet queues
• 744d6d5: apps.rss.rss: Preserve order in classes_active, fix explicite loop unrolling
• 3647611: program.ipfix.probe_rss: make receive queue size configurable for mellanox
• 72f5482: program.ipfix.stats: use global counters when per-queue stats are not available
• c24d55c: lib.interlink: increase queue size to allow bursts for ipfix
• b279790: Make be1f65c conditional for GCC >= 8
• 9bfdaae: program.ipfix: add IF MIB for routed tap interfaces
• 670c411: apps.rss.rss: use specialized function to copy addresses for hashing
• 54409f8: core.packet: increase maximum number of allocated packets
• a1780b3: Fix transitions in CXQ state machine
• 90c5762: map_pci_memory: wait for kernel driver to release the device
• 87c1a0e: apps.mellanox.connectx4: unbind device in IO:new()
• 08f25ca: apps.intel_mp.intel_mp: fix device unbinding for Intel driver
• b47b38d: apps.intel_mp.intel_mp: remove disable bus master cleanup
• 8f1a480: apps.ipfix.ipfix: move record expiration to housekeeping
• da76c64: apps.mellanox.connectx4: fix mcast counters
• 5f05dab: lib.hardware.pci.map_pci_memory: only wait if first call failed
• 15ef82b: program.ipfix.probe_rss: use pools to assign collector address and port
• afbe2bd: apps.rss.rss: Add VLAN match capability
• 6d5bafa: program.ipfix: increase logging rate
• eb659ba: program.ipfix.probe_rss: add observation domain to log output
• 58ce70e: program.ipfix: disable date in log messages by default, fix READMEs
• 432f38a: Add scan-protection feature
• 541c0d8: interlink: fix shmem names in receiver/transmitter shutdown
• b9c499e: program.ipfix.probe_rss: restart dead workers
• 429fa21: program.ipfix: make maximum load-factors of hash tables configurable
• 276a26d: ipfix: restrict flow-rate tracker to signature flows
• aafd550: program.ipfix.probe_rss: scale scan-protection rates with number of processes
• 3620245: program.ipfix.probe_rss: add CPU pool and NUMA binding
• bc9211f: Untabify
• 5a47fde: program.ipfix.probe_rss: make CPU-pinning configurable
• 1f082b8: program.ipfix.probe_rss: increase logging rate limit for master process
• 4af5345: program.ipfix: rename ConnectX4 -> ConnectX
• 4296bfa: Don't export empty flows
• fc407c2: Use proper flow start time after active timeout
• caa11e1: lib.ctable: fix hugepages allocation size
• 64a62f2: apps.mellanox: Make RX/TX flow control configurable, default to off/off
• 4c1571f: apps.mellanox: Add per-queue drop stats
• 69a15b9: apps.mellanox: Use lib.parse in ConnectX:new() and IO:new()
• 3a5b5ae: apps.mellanox: Use consumer counter in cxq to simplify ownership handling
• be7a343: connectx.lua: Add basic event handling
• 7c269fa: map_pci_memory: wait for kernel driver to release the device
• 1c7497c: lib.hardware.pci.map_pci_memory: only wait if first call failed

Andy Wingo (2):

• 907ce81: Use binary search for all lookups
• b457b6e: Add a type cache, in case that's important

Luke Gorrie (5):

• b29666f: lj_trace.c: Clear penalty slot after successful trace
• b4060ef: lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
• 0a90764: fixup! lj_jit.h: Increase HOTCOUNT_MAX and expand HotPenalty.val
• c685d8b: lj_trace.c: Reset hotcount table after flush
• 5443940: lj_record.c: Relax heuristic for root trace meeting JIT loop

Max Rottenkolber (76):

• bd5d4ec: wip
• 2f6b983: wip 2
• e301b53: mellanox benchmark numa
• aaea69b: add Source software benchmark to rule it out as a bottleneck
• 7634573: measure rx at sink
• 7f967e4: apps.mellanox: packetblaster
• 9adad52: mellanox benchmark: use packetblaster
• 350aea9: apps.mellanox.connectx: cleanup and optimization
• 3d86feb: mellanox benchmark: use stats counters instead of querying HCA
• ae5121b: mellanox benchmark: cleanup output
• b1cf848: mellanox benchmarks example scripts
• 87207cf: apps.connectx: remove dead code from packetblaster
• 0ab555f: mellanox benchmark: add --opt option to supply driver config options
• 312b740: mellanox benchmark: add forwarding mode
• 28f249a: mellanox benchmark: add --nmacs, --nvlans paremetrs, remove source-sofftest
• 4fa1d01: mellanox benchmark: fix up previous commit
• 6c69c04: mellanox benchmark: add sink mode, sink halts
• 7195667: mellanox benchmark: IMIX source
• f799af2: apps.interlink: add instrumented latency benchmark
• fbc13ab: core.packet: factor packet reclaimation from group_fl into reclaim_ste...

Snabb 2022.01.1 “Octarina”

Fix up .version for Octarina release.

See Snabb 2022.01 “Octarina”

Snabb 2022.01 “Octarina”

This release contains the a new driver for Mellanox Connect-X series NICs, an improved Intel AVF driver with support for RSS, and an experimental XDP driver. As usual this release also brings many smaller improvements and bug fixes.

What's Changed

• Mellanox ConnectX-4 Driver (WIP) by @capr in #918
• Completed Mellanox initialization sequence by @lukego in #943
• Upstream lwAFTR 2022.01.x by @eugeneia in #1465
• Fixes for regressions in max-next headed for Octarina release by @eugeneia in #1466
• Collected changes for Jan 2022 release: Octarina by @eugeneia in #1463
• Snabb 2022.01 “Octarina” by @eugeneia in #1467
• XDP app: driver for Linux AF_XDP sockets by @eugeneia in #1450
• intel_avf: fix integration breakage and free resources on shutdown (stop()) by @eugeneia in #1451
• Various YANG fixes related to choice statements and the consistency checker by @eugeneia in #1452
• Add filter option to XDP app for punting packets to Linux by @eugeneia in #1454
• lib.numa: gracefully handle failure to read CPU performance governor by @eugeneia in #1455
• lib.ptree: cleanup obsolete aggregated stats by @eugeneia in #1456
• arch/checksum: fix bug in carry propagation by @eugeneia in #1457
• lib.ctable: fix hugepages allocation size by @alexandergall in #1459
• core.packet: fix account_free physical capacity formula by @eugeneia in #1460
• Driver for Mellanox Connect-X 4/5/6 cards by @eugeneia in #1461
• Updated Intel AVF driver with VLAN+RSS support by @eugeneia in #1462
• Tracing JIT stability improvements from lwAFTR by @eugeneia in #1464

Full Changelog: v2019.11...v2022.01

Contributors

Alexander Gall (24):

• 48f6080: Cherry-picked apps.melanox from lukego/debug100g
• 4474000: Uncommited changes from lukego
• 580cf9b: connectx4.lua: minor changes
• 10ceea4: connectx4.lua: fix alignment of HCA command queue
• 1b9cf2b: connectx4.lua: fix RSS
• 8332671: Register ConnectX5 device in lib.hardware.pci
• 263919a: connectx4.lua: identify driver for device_info
• d57c67a: connectx4.lua: add statistics counters
• 22af847: connectx4.lua: use async commands to read stats counters
• a949245: connectx4.lua: use a factory to create HCAs
• b4d9252: connectx4.lua: fix bug with async commands when debug_hexdump is enabled
• e2eecff: connectx4.lua: fix page sizes in CREATE_{C,R}Q
• 76c0c80: apps.vhost: suppress -Wstringop-truncation
• 56884a7: apps.mellanox.connectx4: add MT28800 as supported device
• a0e8585: apps.mellanox.connectx4: add per-queue drop counters for ConnectX5
• fc37be4: apps.mellanox.connectx4: unbind device in IO:new()
• adc7d3d: apps.mellanox.connectx4: fix mcast counters
• 2219748: Fix transitions in CXQ state machine
• 6cdcb40: apps.mellanox: free CXQ during process shutdown
• c579524: apps.mellanox: use correct WQE stride in RX queues
• 556e427: mellanox: complete renaming ConnectX4 -> ConnectX
• a348899: lib.ctable: fix hugepages allocation size
• a86c61e: connectex.lua: extend RSS hashing to non-TCP/UDP IP packets
• 5f10b3b: connectx.lua: convert tabs to spaces

Andy Wingo (5):

• 9a787b5: Fix snabb top interface view for multiple queues
• 7a0c6c8: Disable vmprofile on lwAFTR by default
• 648c67a: Add 2019.06.02 changelog entry
• 6524dec: fix typo
• 828fa4f: lwAFTR version 2019.06.02

Cosmin Apreutesei (6):

• 45d0696: stub module and readme
• 8720a60: pci init
• cd3a103: vendor/dev id
• dfbd705: firmware revision register
• 34929f1: print firmware revision maj and min
• fe8065f: init procedure until QUERY_HCA_CAP

Luke Gorrie (24):

• a6973da: lib.hardware.pci: Add reset_device()
• ccdf38e: mellanox: Added text/hexdump trace of Linux driver
• 276f4fa: connectx4: Progress on initialization routine
• 6a426bd: connectx4: Translate virtual to physical on demand
• f349c41: core.memory: Add 'align' argument to dma_alloc()
• 8c55f66: connectx4: Request alignment on memory.dma_alloc()
• 2c1a92d: connectx4: Add multi-mailbox command support
• 6abbd96: connectx4: Successful QUERY_HCA_CAP query
• 92ec410: connectx4: Command mailbox chaining fixes
• 13e6550: connectx4: Set debug = false
• cd8e30e: connectx4: Remove stray print statements
• a285999: connectx4: Move checkstatus() into post()
• 4575dc7: connectx4: Device initialization fixes
• 8c0be9d: connectx4: Comment on ommitted SET_DRIVER_VERSION
• f196ac7: connectx4: Added CREATE_EQ (event queue) command
• c1cee02: connectx4: Added initial VPORT commands
• 51d4d45: connectx4: Give 'lock' argument to map_pci_memory
• 7659eb6: connectx4: Clean driver initialization
• 21d0dc3: connectx4: Refactored for multiprocess operation
• f08da3c: connectx4: IO app now has only 1xSQ and 1xRQ
• ffe4a22: connectx4: Add 'mtu' paramter (default 9500)
• 06c43f0: connectx4: Cleaner (simplistic) selftest method
• e367b6b: connectx4: Fix recv completion queue wrap-around
• 42c2196: mellanox: Rename ConnectX4 -> ConnectX (because it works with 5+ too)

Max Rottenkolber (150):

• 77e548c: Squashed 'lib/ljsyscall/' changes from d78235c..e587f8c55
• 09b29d9: ljsyscall: add some XDP constants
• 3f82010: ljsyscall: fix a bug in getsockopt when querying for structs
• 1e176cb: ljsyscall: lookup strflags in linux bpf(2) wrappers
• 573a29b: ljsyscall: add BPF_MAP type XSKMAP
• 63f0cfe: apps.test.synth: add packets option
• f535525: apps.test.synth: emit all given packets
• 6174563: core.packet: expose constants for packet alignment and headroom
• 984d4a4: apps.xdp: driver for AF_XDP sockets
• e056fa1: Remove src/xdp.snabb (accidentally checked in xdp proof of concept)
• ee4e42d: core.packet: expose free_internal
• b677292: apps.xdp: fix free stats accounting, cleanup fill/completion mngmnt
• 1b3593f: apps.xdp: add duplex test case, fix random_v4_packets
• 22d754c: apps.xdp: cleanup bpf_ld_imm64/LD_MAP_FD construction
• 4e31856: apps.xdp: generate more descriptive errors on bind(2) failure
• e53f314: apps.xdp: fixup comment about engine breath stimulation
• d225e75: apps.xdp: make num_chunks configurable in snabb_enable_xdp(opt)
• d44fb1d: apps.xdp: do not leak packets in XDP:stop()
• f8cb80c: apps.xdp: munmap(2) rings in XDP:stop() (do not leak rings mappings)
• e06c82f: apps.intel_avf: use new PCI acccess API (fix integration breakage)
• 7722f0d: apps.intel_avf: free resources on shutdown (fix leak)
• 6c330ec: lj_syscall: add BPF_OBJ_* syscall wrappers, extend bpf_attr union
• ea955a8: apps.xdp: allow sharing interfaces between Snabb processes
• 01f11e9: pflua: do not cdef struct bpf_ins (collides with ljsyscall)
• ae35674: core.app: add clearvmprofiles() function
• 1124b80: lib.timers.ingress_drop_monitor: clear vmprofiles on flush
• 8f2a216: lib.timers.ingress_drop_monitor: reset last_value on drop-free period
• 63968f6: raptorjit: clear all hotcounts every second
• dfadac8: raptorjit: amend prev. commit, also clear trace exit hotcounts
• b7c50cb: raptorjit: amend prev. commit, do not clear SNAPCOUNT_DONE
• bd84224: apps.xdp: expose driver reference
• 1a9eba5: lib.scheduling: add enable_xdp option
• 18cc365: apps.xdp: kick tx queue if not empty
• 19072b4: lib.yang.data: reorder translated members before printing
• ae4f734: snabb config: use consistency checker in command parsing
• 05b495e: lib.yang.path_data: expand choices in consistency checker
• e96ae1a: lib.yang.data: rework 19072b4 (remove order parameter) add tests
• 95d3a74: lib.yang: remove leftover debug print
• 3fdf926: pflua: fix typo bug in native codegen (xor/or emission)
• 07acce2: pflua: make regalloc architecture independent
• cd4d773: apps.xdp.bpf: add basic disassembler, endiannes opcodes
• 48ff71c: apps.xdp.pf_ebpf_codegen: add eBPF backend for pflua
• c158ac8: apps.xdp: add filter option (offload non-matching packets to kernel)
• 6b5851a: lib.numa: gracefully handle failure to read CPU performance governor
• 16c99ed: lib.ptree: cleanup obsolete aggregated stats
• 0068df6: arch/checksum: fix bug in carry propagation
• 3461a09: apps.xdp: only run rxtx_match selftest if SNABB_XDP_NQUEUES=1
• 00cac36: apps.xdp: debug stop()
• 8d4ee13: apps.xdp: mark stop() NYI / not supported
• 2bcc644: lwaftr --xdp sketch
• dca3004: snabb-softwire-v2: add default for leaf error-rate-limiting/packets
• 8c2000b: apps/ipv4,6/reassemble,fragment: handle padded packets
• 561b52a: lwaftr/setup: fix interfaces for bump-in-the-wire Interfaces used in bump-in-the-wire configurations were swapped, possibly during a prior refactor. Undo this.
• 1cdc635: snabb-softwire-v2.yang: documentation edits
• 6bcce09: lib.numa: gracefully handle failure to read CPU performance governor
• 191581a: apps.lwaftr.ndp: send NS via solicited node multicast
• 4817ae8: lwaftr: support vlans in XDP mode
• 004587b: lwaftr: automate linux interface setup for XDP
• bce8579: snabb-softwire-v2: allow more than two queues
• 909dca7: lib.protocol.ethernet: add ptoi used by connectx driver
• 6bca98b: apps.mellanox.connectx: fixup lib.hardware.pci usage
• 832ea9d: apps.mellanox.connectx...

Snabb 2019.11 “Efraasia”

Lots of new gadgets in this release: a Snabb driver for Intel AVF based Virtual Functions (virtualized NICs) such as provided by Intel X710 and XL710 based NICs, a new optimized Poptrie implementation for all your longest-prefix-match needs (lib.poptrie), VMDq support for Intel i350 based NICs, support for 256-bit AES keys in lib.ipsec, and much broader support for YANG language features in lib.yang. As usual, this release includes numerous additional small improvements, optimizations and bug fixes.

Changes

• #1314 #1381 #1382 #1438 Additions , optimizations and bug fixes for lib.ipsec. New: support for AES 256 (256-bit keys).
• #1443 Fixed a bug in statistics collection in apps.intel_mp (Intel 52899 driver)
• #1442 Fixed a bug where belated opening of auditlog would fail due to initial JIT activity (open it earlier)
• #1397 Added a fast, generic Poptrie implementation (lib.poptrie)
• #1390 Fixed a bug that prevented shared packets from being released to the group when a process is stopped
• #1416 Added support for YANG enumeration types to lib.yang
• #1357 Various bug fixes and improvements to snabb loadtest
• #1431 Fixed a bug in core.memory that caused SIGSEGV handling to fail to recent RedHat Linux distributions
• #1427 Fixed a crash in snabb ipfix probe when the duration flag was not provided
• #1423 lib.hash.siphash: improved compatibility with reference implementation
• #1421 core.app: simplified loop structure of engine to help the JIT find better traces
• #1353 Added VMDq support for Intel i350 based NICs to apps.intel_mp, and fixed some statistics collection bugs for i350/i210 based NICs
• #1376 lib.yang: changed implementation to use native Lua number keys for parsed YANG data where possible
• #1435 lib.yang: added support for unique, min/max, and choice keywords
• #1436 apps.intel_mp: make sure PCI device is unbound from Linux driver before mapping its memory into the Snabb process (previous behavior caused crashes with recent Linux kernels, see issue #1286)
• Igalia#1231 lib.fibers.timer: fixed a bug in timer event promotion
• Igalia#1224 lib.yang: added support for if-feature expressions
• #1448 Merged lwaftr release v2019.06.01 into Snabb
• #1444 apps.intel_mp: added support for Tx->Rx MAC Loopback mode, and fixed related bugs
• #1446 Added apps.intel_avf, a driver for Intel AVF capable NIC virtual functions (Intel X710, XL710, ...)
• #1445 #1433 #1427 Various minor bug fixes and improvements
• #1432 Bug fixes and improvements to SnabbBot, our CI robot

Contributors

Alexander Gall (4):

• 5724ff2: lib.yang.schema: implement semantics for enumeration type
• 4989b99: lib.yang.data: add test-case for enumeration type
• 8c616b0: core.app: use implict pull/push loops in breathe()
• edc07fc: lib.hash.siphash: add option to comply to reference implementation

Andy Wingo (28):

• a805f92: Re-enable NUMA memory binding on newer kernels
• 3809be2: Detect version mismatch when loading compiled YANG configurations
• 69764b5: Remove bogus lwaftr benchmarking file
• 7f2a1d2: Fix PCI affinity check for --preferred NUMA binding
• a27d8b3: Add more system performance diagnostics
• ecd443c: Detect irqbalance presence and suggest remedy
• 9f7177b: Fix uninstall advice for irqbalance
• e1af4f9: Improve snabb lwaftr run --help
• a004e9a: Fixup to snabb lwaftr run --help
• fbf585e: Refactor packetblaster lwaftr to re-use vlan, arp, ndp, v4v6
• 7886344: Separate B4 and internet lwaftr load generation apps
• 14c7255: Allow different IPv4 MAC addresses and VLAN tags; fix bugs
• b63f196: Fix whitespace
• 1b74cf9: Fix source IPv6 from B4, and better interleaving of streams
• e7a60d3: Fix frame sizes. Update test files for fresh checksums. Allow for ordering differences between v4 and v6 packets. Stop in a more robust way.
• 94d123a: Fix inc_ipv4
• f1050ec: Start ingress drop monitor in grace period
• bd822d9: Bump snabb-softwire-v2 revision dateD
• 516306a: Add v2018.09.03 changelog
• acd54d9: Update version to v2018.09.03
• b4c9073: Fix promotion of lib.fibers.timer events from outer to inner wheel
• 0e9e5d9: Make memory monitor a timer instead of an app
• cc4acb5: Revert "lib.ptree: do not aggregate (ignore) counters that are symlinks"
• 79075f5: Snabb lwAFTR 2019.06.01
• 1577238: Fix markdown syntax
• be06560: Add link to upstream release notes
• a283e7b: Add link to YANG module
• c2015ea: Fix release benchmarks to always specify CPU

Asumu Takikawa (1):

• 8e8ad64: ipfix: fix done state checking in ipfix probe

Luke Gorrie (1):

• 70dd879: README.md: Update Slack invite link

Max Rottenkolber (160):

• 25a8d95: snabb_bot: delete $tmpdir on startup
• 971627f: snabb_bot: configure git user name and email
• 29e2303: snabb_bot: add timeout to cushion broken test cases
• 6d1594a: snabb_bot: print SNABB_PCI_INTEL1G{0,1} / snabb_doc: set git user/email
• 44f6ab0: snabb_doc: fixup git user/email config
• 59b1b82: apps.ipsec: add tunnel mode esp app
• 17e8c6c: snabbmark esp: add rudimentary PMU analysis
• 461f195: lib.ipsec.aes_128_gcm: add rudimentary PMU profiling to selftest
• 9d14b98: snabbmark esp: fix bug where packets of wrong size were constructed
• 7f75dc1: snabbmark esp: minor changes
• bb72e66: lib.ipsec: add missing licensing headers
• 53ae8bd: lib.ipsec.esp: correct bogus note about IP fragment handling
• 531170a: lib.ipsec.esp: avoid lib.protocol.* on the fast-path
• 43d498a: lib.ipsec.esp: use full 16 (instead of 12) byte ICV
• 66a0dbb: lib.ipsec.esp: keep lib.protocol.esp*
• 65f0a75: snabb_bot: fix bug that compared benchmark results with wrong target
• f49720a: snabb_bot: make checkout/merge more robust
• 2814b7b: intel_mp: fix broken per queue stat accumulation
• 9c4e3ee: intel_mp: VMDq support for i350
• 33c0912: apps/intel_mp: RQDPC is actually an RC register on the i350
• 283d6a8: loadtest find-limit: fix some edge cases and bugs
• 90c3be7: snabb_bot: fix bug in benchmark regression testing
• 32586f3: lib.yang.data: represent some number-keyed lists as Lua tables
• 8269afe: lib.yang.binary: support number-keyed Lua tables in ad-hoc grammars
• 07821cf: lib.yang.path_data: s/string_key/native_key
• 796c177: lib.ipsec.esp: optimize padding by specializing for power of two
• 5b86f94: lib.ipsec.esp: fix a packet leak in :resync()
• 2bd39ae: intel_mp: fix 1G blast selftest via use of RateLimitedRepeater
• 9ea857e: intel_mp: port more vmdq tests to 1G cards.
• afc971b: intel_mp: revert queuing -> queueing spelling change from 9c4e3ee
• 691b5a1: lib.yang.data: return single value from normalize_id
• 1bff543: lib.yang.path_data: add unique restriction consistency checker
• 5d00ae9: lib.yang.path_data: run consistency_checker in parser_for_grammar
• c7a655e: lib.ptree: run consistency checker on initial_configuration
• 1c9642e: core.packet: release packets to group freelist on shutdown
• ac45e3f: lib.interlink: fix an off-by-one bug in the tostring method
• 5533782: core.packet: do not append .freelist in freelist_create/open)
• 78cf3c3: main.shutdown: unlink supervisor shm root
• ace9b32: Revert "main.shutdown: unlink supervisor shm root"
• c6b32f4: core.packet: require explicit initialization of packets freelist
• 482c3a5: Work-in-progress implementation of Poptrie
• 3234b1f: Add basic documentation for lib.poptrie
• 9343b36: lib.poptrie: fix building empty RIB and repeated calls to build()
• 4e48a25: lib.poptrie: refactor :build() into smaller methods...
• 77cd0a7: lib.poptrie: implement direct pointing
• b5fcbfb: lib.poptrie: do not print zero value leaves in fib_info
• 92bc6c6: lib.poptrie: optimize build(), cover direct pointing in PMU analysis
• 3fcc794: lib.poptrie: make s (bits used for direct pointing) configurable
• db1320f: snabb_bot.sh: do not attempt to initialize submodules
• 703eb9a: intel_mp: work around flaky mirror test on i350
• ab77197: lib.yang.data: make table_native_key a bit easier to follow
• 17871d4: lib.yang.binary: minor coding style change
• d29763d: intel_mp: fix a bug in VMDq BAM (bcast accept) handling for the i350
• 6be165e: intel_mp: add VMDq broadcast tests
• 69b6ca1: intel_mp: fix selftest from the future
• 7b52cf8: lib.yang.data: fix bug that prevented top-level choices being parsed
• 0893b16: lib.yang.data: fix bug when parsing list members of choice cases
• 5eb3857: lib.yang.data: correctly parse choice case short form.
• 6ff14ef: lib.poptrie_lookup: incorporate peter’s suggestions
• bed7ec2: lib.poptrie_lookup: comment formatting
• ab0cfaf: lib.poptrie: extend to support lookup32/lookup128
• 4e71680: lib.poptrie_lookup: fix bug where offset was not set correctly
• 6211b05: lib.poptrie_lookup: optimize lookup128, prefer 32-bit operations
• 4f8ed1f: dasm_x86: import ADX/BMU/FMA3 instruction support from LuaJIT:v2.1
• 7283f71: lib.poptrie_lookup: use BMI2 instructions when available
• 5ca4578: lib.poptrie: avoid cache thrashing in micro benchmark
• 55f7788: lib.ptree.support: fix bug related to native table keys
• 355b033: lib.poptrie: fix out of bounds read in add
• 4c88093: lib.poptrie: avoid slow down incurred by ASM routines as upvalues
• ff13dd2: !fixup previous commit
• 2b613ad: lib.poptrie: reformulate extract
• e22dcd8: lib.poptrie: add test scaffolding for testing different configs
• 4e9379b: lib.poptrie: update a comment
• 7f92167: lib.poptrie: cache generated ASM code
• 8419101: lib.poptrie: expose micro-benchmark parameters via env vars
• 6a684a9: lib.poptrie: fix up to previous commit
• 125facd:...

Snabb 2019.06 “Deltadromeus”

This long awaited release is the first Snabb to ship with RaptorJIT and runtime support for the Studio diagnostics environment. The Snabb top monitoring tool is replaced bya more interactive rewrite, and its basic SHM object listing functionality is spun out to the new snabb shm utility. In addition to bug fixes and optimizations, this release also includes improvements to lib.yang and our IPv6 fragmentation apps as well as support for current versions of nDPI.

Changes

• #1316 Integrate RaptorJIT with Snabb
• #1401 Merged lwAFTR release v2018.09.02
• #1386 Add snabb shm: simple tool to print information about Snabb SHM objects
• #1384 apps.ipv6.reassemble: add expiration of incomplete reassembly buffers
• #1383 Add path MTU discovery to IPv6 fragmenter (apps.ipv6.fragment)
• #1275 lib.checksum: optimize IP checksum computation using DynASM
• #1368 Support nDPI 2.X in ljndpi
• #1380 Fix a bug in apps.vhost.vhost_user with regard to NUL-termination
• #1385 Move logger in core.lib to lib.logger
• #1392 VlanMux: optimize tag pop/push (apps.vlan)
• #1394 #1395 #1396 Ctype diversity related optimizations

Contributors

Alexander Gall (22):

• 796597d: lj_trace.c: unpatch blacklisted bytecodes when flushing traces
• 2ad6817: ipv6.reassembly: avoid packet cloning
• 0463915: ipv6.reassembly: expire stale reassembly buffers
• ffd584c: ipv6.reassembly: reduce scope of local variables in handle_fragment()
• 66a7575: Add PMTUD to apps.ipv6.fragment
• a9a4b72: ipv6.fragment: improve performance, reduce garbage
• 7f0703a: ipv6.reassembly: perform reassembly in separate loop
• dd9bb12: Refactor the logger facility
• 2bbaa39: lib.tsc: cache calibration of rdtsc
• 477b4c5: VlanMux: optimize tag pop/push
• 8f76782: intel_mp: fix ctype diversity for txdesc_t
• cff919c: packet: cast pointer arg in from_pointer()
• f896458: datagram: canonicalize pointers in packet append() and prepend() calls
• d06314c: lwaftr.ctable_wrapper: add cleanup function to random eviction
• a44d8e1: apps.ipv6.fragment: canonicalize pointers, simplify ICMP matching
• 4e28690: apps.ipv6.reassemble: fix packet leak
• d7a51c6: apps.ipv6.reassemble: avoid spurious sort
• 2e5b5e8: apps.ipv6.reassemble: copy header from first fragment
• f7a00ff: apps.ipv6.fragment: fix bug in ICMP type match
• 57cc280: apps.ipv6.reassemble: remove over-eager optimization of module operator
• d054ec0: apps.ipv6.reassemble: use max_payload constant in packet size check
• 69473eb: apps.ipv6.reassemble: fix check for valid packet length

Andy Wingo (122):

• 634ab24: Remove duplicate lj_auditlog.c
• 2914e5a: Make file-local auditlog variables static
• 8119e5d: Remove references to the profiler, -jv, -jdump, and the like
• bbd7635: Remove references to -jv, -jdump, and the like
• 6943135: Revert "Merge pull request #1040 from lukego/dynasm-gc64-nocoerce"
• a91e457: Add ring buffer module
• 252257c: Add basic IO streams
• 5ceb3c8: Add file stream implementation
• ddb6a8e: Add full replacement for Lua's "io" module
• fed2034: Move io.open implementation to lib.stream.file
• 5ee5357: YANG output byte streams use new Lua streams
• 311bea2: Move tmpfile() impl to file.lua; add optional perms arg
• 8a0b098: More YANG output byte stream refactoring
• 55deb49: Add some helper methods to Lua streams
• b27a00c: Remove calls to :align() output to ffi natural sizes
• a2fda31: Remove unused open_output_byte_stream function
• 0e5f45f: Remove unused lwaftr rangemap load/save capabilities
• a361697: Replace lib.yang streams with lib.stream streams
• 0512a8f: Remove YANG streams
• 810c81e: Add memory-backed stream implementation
• 5d1b830: Migrate YANG and related libs away from ad-hoc string_io_file
• 949e663: YANG syntax parser is now streaming
• 7e45ce3: Fix use of undeclared variable (refactor residue)
• 1a33991: Fix default EOL handling style
• ce3b8ad: Fix error reporting in "listen" test
• 3805a3a: Fix top-level scalar parsers to take stream as argument
• ac8e74e: File streams query OS for appropriate block size
• 91560fa: Adapt "snabb config listen" to streaming parse
• 38f1ad4: Fix conversion of listen to streams
• e135a02: call_with_output_string flushes stream
• 4129dec: Add timer wheel
• 1edd231: Add fibers scheduler
• 1d2e560: Add fibers
• c6beee9: Add concurrent ML operations and channels
• f9a37a6: Move sleep and timeout-related operations to sleep.lua
• 18467fb: Add epoll wrapper
• 28ca806: Rework complete tasks to be more centralized
• 650f7c2: Add epoll-based nonblocking file I/O
• 2974b06: Supervisor process restores blocking flags on stdin,stdout,stderr
• 6bf7425: Add Concurrent ML buffered channel
• 149360b: Convert "snabb config listen" over to use fibers
• 2a5b526: Convert lib.ptree.trace to use new streams
• e26a4b6: Adapt new JSON library to return nil on EOF
• 19e422c: Rewrite "snabb config bench" to use fibers and new streams
• a6654ce: Replace old ptree JSON library
• e1b0324: Fix fibers sleep self-test
• 70fde0b: Add sockets library that integrates with blocking handlers
• fb7c091: Improve error handling in "snabb config" tools
• 007be9b: Add support for orderly scheduler shutdown
• 37053d2: Convert ptree manager to use fibers
• 0d7d94a: Minor cleanup when reading string table
• 7466c74: Fix YANG binary serialization of by-reference cdata leaf values
• 8ef996b: Add assertion to Stream:write_scalar
• fdec9b7: YANG binary serialization allows for >4GB files
• f6f8755: Avoid copy when reading in ctable
• ee1a81a: Use more relaxed NUMA memory binding policy
• 855aae7: Simplify alarm notifications listener
• 3ee13d9: Fix socket API change
• 9bb068e: Add condition variables to fibers
• a6ea6ff: Fix typo in channels self-test
• f3cf804: Update fibers file API to allow wait-for-readable ops
• b47a55f: Add inotify module to ptree
• 90dacea: Add EINTR handling to epoll module
• f4a4ab9: Add :next_entry_time() method to timer wheel
• af60f95: Add scheduler :main method
• b0bc5b3: Use fiber main method in programs
• 598745a: Restore schedule_tasks call on event waiter
• 5426d18: Fix duplicated method
• 032d187: Minor scheduler refactor
• cd99fe3: Inotify is_dir() function uses lstat
• c16edb7: Make lib.comma_value work better for large uint64 values
• 5c8cabd: Rewrite "snabb top" with fibers
• b74014c: Fix error case for histogram summaries
• 4fa0ca8: Arrows change focus in snabb top
• 600cbee: Remove snabbvmx top.
• 270bddb: Fix "snabb lwaftr query"
• 1914a86: Add round-robin database library
• bfe4f3d: Back off RRD version to 3
• c29f095: Refactor ptree manager treatment of counters
• d08292e: Fix reading of "snabb config" configurations from stdin
• 225878d: Make sure recursive inotify drains all nested events before returning
• 57b8c18: Swap argument order for file names and initargs in rrd create
• 5eedfef: Fix RRD bugs
• 8a7ab76: Ptree manager samples worker and aggregated counters into RRDs
• 81a54ed: "snabb top" reads rrd files
• 59d372d: snabb top doesn't monitor itself
• 5d033e1: Add "interface view"
• abe2d58: Improve spacing and column width allocation in snabb top grids
• cf8992f: Switch "snabb top" to prune state tree in UI
• 2e95609: Wrap "snabb top" counter leaves in an object
• 6530a0d: Sort rrd lookup results
• c3fef25: top: Add ability to skim back and forward through time
• 1e77c9d: Add fallback version of directory monitor if inotify is unavailable
• 2ba44e8: Make alarms codec use stream i/o routines
• 6806b59: Tweaks to make alarm encoding have less overhead
• 74e78e3: Cache array types in stream.lua
• d6a8c98: Fix another snabb top problem
• 53f4e05: Update "snabb top" docs
• 607f7c1: Add ptree docs on counter collection and aggregation
• a533ed4: Fix typo
• 26812c6: Remove some indirection in alarm_codec
• 34ec6c8: Fix broken encode_raise_alarm
• 15be803: Simplify alarms API
• 93090ae: alarm_codec doesn't require unpacked args
• 047f6e7: Fix bug introduced in previous commit
• a99cd30: Ptree extends alarms by installing alarm handler
• 3930e90: NDP uses a pre-allocated pseudoheader
• 0cb4122: Remove some loops-within-loops in stream.lua
• dbdb0d1: Micro-optimizations
• 9930715: Fix to alarm:raise
• 84c2901: Expand on some Changelog features
• 6ce2990: Minor wording changes around the RRD feature
• dd54c80: Revert "Show rxbits and txbits instead of bytes"
• 7b75fbf: Ensure vhost-user socket paths are NUL-terminated
• 44ef127: intel_mp app instantiations make stats symlink counter forest
• 5fa0d15: Remove unused symlink creation routines
• edad14b: Minor formatting change
• 71bf219: Update l2vpn for intel_mp stats change
• a24bb8d: Remove extraneous diff
• fa913d5: Remove more extraneous diff
• 8f54869: Revert "Minor formatting change"
• 144c589: intel_mp: Don't eagerly open shared counters

Asumu Takikawa (30):

• b145d27: tap: bump selftest timeout to 0.05 seconds
• c4681a6: Use virtual instead of profile time for vmprofile
• fdf5d02: Restore use of LUAJIT_CFLAGS
• 4bd9d6e: intel_mp: store shm stats in global directory
• 2a638fe: Go back to avoiding setting run_stats
• 0d641de: inotify: adjust is_dir to allow symlinks to dirs
• 373d195: Filter queue counters based on enabled bitmap
• 1c536e2: intel_mp: add a new test for rxcounters & workers
• c40f0ea: snabb top: refactor queue stats filtering
• af8c75f: snabb top: fix worker nesting with shared pci
• 5388c98: intel_mp: add 10g test for shared RX counters
• 6b72001: Use virtual instead of profile time for vmprofile
  ...