Skip to content
Permalink
Browse files

Merge PR #1370 (v2018.09 release) into master

  • Loading branch information...
eugeneia committed Sep 12, 2018
2 parents 732eae9 + a154efc commit 771b55c829f42a1a788002c2924c6d7047cd1568
Showing with 875 additions and 278 deletions.
  1. +1 −1 .version
  2. +4 −0 src/README.md
  3. +1 −1 src/apps/bridge/README.md
  4. +2 −1 src/apps/example/asm.dasl
  5. +2 −0 src/apps/ipfix/README.md
  6. +4 −0 src/apps/vhost/vhost_user.h
  7. +39 −15 src/apps/vhost/vhost_user.lua
  8. +11 −0 src/core/app.lua
  9. +2 −78 src/core/lib.lua
  10. +3 −2 src/core/memory.lua
  11. +3 −0 src/lib/README.ctable.md
  12. +70 −0 src/lib/README.token_bucket.md
  13. +112 −0 src/lib/README.tsc.md
  14. +13 −4 src/lib/ctable.lua
  15. +3 −3 src/lib/hash/siphash.dasl
  16. +1 −1 src/lib/multi_copy.dasl
  17. +32 −7 src/lib/numa.lua
  18. +20 −37 src/lib/pmu.lua
  19. +105 −16 src/lib/pmu_x86.dasl
  20. +2 −1 src/lib/protocol/icmp/header.lua
  21. +37 −0 src/lib/protocol/icmp/ptb.lua
  22. +130 −0 src/lib/token_bucket.lua
  23. +120 −0 src/lib/tsc.lua
  24. +7 −2 src/lib/virtio/net_device.lua
  25. +2 −1 src/lib/virtio/virtio.h
  26. +42 −1 src/lib/yang/data.lua
  27. +1 −1 src/lib/yang/schema.lua
  28. +2 −2 src/program/lwaftr/tests/data/counters/arp-for-next-hop.lua
  29. +2 −2 src/program/lwaftr/tests/data/counters/empty.lua
  30. +2 −2 src/program/lwaftr/tests/data/counters/from-inet-ipv4-in-binding-big-packet-df-set-allow.lua
  31. +2 −2 src/program/lwaftr/tests/data/counters/from-inet-ipv4-in-binding-big-packet-df-set-drop.lua
  32. +2 −2 src/program/lwaftr/tests/data/counters/from-to-b4-ipv6-hairpin-n64.lua
  33. +2 −2 src/program/lwaftr/tests/data/counters/from-to-b4-ipv6-hairpin.lua
  34. +2 −2 src/program/lwaftr/tests/data/counters/from-to-b4-tunneled-icmpv4-ping-hairpin-unbound.lua
  35. +2 −2 src/program/lwaftr/tests/data/counters/from-to-b4-tunneled-icmpv4-ping-hairpin.lua
  36. +2 −2 src/program/lwaftr/tests/data/counters/icmpv6-ping-and-reply.lua
  37. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-infrags-out-1p-ipv6-6-outfrags.lua
  38. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-0p-drop.lua
  39. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-icmpv4.lua
  40. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-1.lua
  41. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-2.lua
  42. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-3.lua
  43. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-4.lua
  44. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-6-outfrags.lua
  45. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-6.lua
  46. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-7.lua
  47. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-8.lua
  48. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-1p-ipv6-echo.lua
  49. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-none-1.lua
  50. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-none-2.lua
  51. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-none-3.lua
  52. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv4-out-none-4.lua
  53. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-0p-ipv4.lua
  54. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-icmpv4-1.lua
  55. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-icmpv6-1.lua
  56. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-icmpv6-2.lua
  57. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-1.lua
  58. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-2.lua
  59. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-3.lua
  60. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-4-and-echo.lua
  61. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-4.lua
  62. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-5-frags.lua
  63. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-5.lua
  64. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-1p-ipv4-hoplimhair.lua
  65. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-none-1.lua
  66. +2 −2 src/program/lwaftr/tests/data/counters/in-1p-ipv6-out-none-2.lua
  67. +2 −2 src/program/lwaftr/tests/data/counters/in-ipv4-ipv6-out-icmpv4-ipv6-hairpin-1-drop.lua
  68. +2 −2 src/program/lwaftr/tests/data/counters/in-ipv4-ipv6-out-icmpv4-ipv6-hairpin-1.lua
  69. +2 −2 src/program/lwaftr/tests/data/counters/ndp-no-na-next-hop6-mac-not-set-2pkts.lua
  70. +2 −2 src/program/lwaftr/tests/data/counters/ndp-no-na-next-hop6-mac-not-set-3pkts.lua
  71. +2 −2 src/program/lwaftr/tests/data/counters/ndp-ns-for-next-hop.lua
  72. +2 −2 src/program/lwaftr/tests/data/counters/ndp-secondary.lua
  73. +2 −2 src/program/lwaftr/tests/data/counters/nofrag4.lua
  74. +2 −2 src/program/lwaftr/tests/data/counters/nofrag6-sol.lua
  75. +2 −2 src/program/lwaftr/tests/data/counters/nofrag6.lua
  76. +2 −2 src/program/lwaftr/tests/data/counters/non-ipv4-traffic-to-ipv4-interface.lua
  77. +2 −2 src/program/lwaftr/tests/data/counters/non-ipv6-traffic-to-ipv6-interface.lua
  78. +2 −2 src/program/lwaftr/tests/data/counters/regressiontest-endaddr.lua
  79. +2 −2 src/program/lwaftr/tests/data/counters/regressiontest-signedntohl-frags-counters.lua
  80. +2 −2 src/program/lwaftr/tests/data/counters/tcp-frominet-bound-ttl1.lua
@@ -1 +1 @@
2018.06
2018.09
@@ -258,6 +258,10 @@ following keys are recognized:
printed.
— Function **engine.stop**
Stop all apps in the engine by loading an empty configuration.
— Function **engine.now**
Returns monotonic time in seconds as a floating point number. Suitable
@@ -9,7 +9,7 @@ arriving on a free input port may be forwarded to all other output
ports. Packets arriving on an input port that belongs to a split-horizon
group are never forwarded to any output port belonging to the same
split-horizon group. There are two `bridge` implementations available:
`apps.bridge.flooding` and apps.bridge.learning`.
`apps.bridge.flooding` and `apps.bridge.learning`.

DIAGRAM: bridge
+----------+
@@ -19,7 +19,8 @@ local asm_status = ffi.new("uint32_t[1]")
|.arch x64
|.actionlist actions
local Dst = dasm.new(actions)
| mov dword [asm_status], 0xdeadbeef
| mov64 rax, asm_status
| mov dword [rax], 0xdeadbeef
| ret
code = Dst:build() -- assign to 'code' to avoid machine code being GC'd
fptr = ffi.cast("void(*)()", code)
@@ -10,7 +10,9 @@ included). The exporter can produce output in either the standard RFC

DIAGRAM: IPFIX
+-----------+
| |
input ---->* IPFIX *----> output
| |
+-----------+
See the `snabb ipfix probe` command-line interface for a program built
@@ -21,6 +21,10 @@ enum {
VHOST_USER_SET_VRING_KICK = 12,
VHOST_USER_SET_VRING_CALL = 13,
VHOST_USER_SET_VRING_ERR = 14,
VHOST_USER_GET_PROTOCOL_FEATURES = 15,
VHOST_USER_SET_PROTOCOL_FEATURES = 16,
VHOST_USER_GET_QUEUE_NUM = 17,
VHOST_USER_SET_VRING_ENABLE = 18,
VHOST_USER_MAX
};

@@ -111,21 +111,25 @@ end

-- Table of request code -> name of handler method
handler_names = {
[C.VHOST_USER_NONE] = 'none',
[C.VHOST_USER_GET_FEATURES] = 'get_features',
[C.VHOST_USER_SET_FEATURES] = 'set_features',
[C.VHOST_USER_SET_OWNER] = 'set_owner',
[C.VHOST_USER_RESET_OWNER] = 'reset_owner',
[C.VHOST_USER_SET_MEM_TABLE] = 'set_mem_table',
[C.VHOST_USER_SET_LOG_BASE] = 'set_log_base',
[C.VHOST_USER_SET_LOG_FD] = 'set_log_fd',
[C.VHOST_USER_SET_VRING_NUM] = 'set_vring_num',
[C.VHOST_USER_SET_VRING_ADDR] = 'set_vring_addr',
[C.VHOST_USER_SET_VRING_BASE] = 'set_vring_base',
[C.VHOST_USER_GET_VRING_BASE] = 'get_vring_base',
[C.VHOST_USER_SET_VRING_KICK] = 'set_vring_kick',
[C.VHOST_USER_SET_VRING_CALL] = 'set_vring_call',
[C.VHOST_USER_SET_VRING_ERR] = 'set_vring_err'
[C.VHOST_USER_NONE] = 'none',
[C.VHOST_USER_GET_FEATURES] = 'get_features',
[C.VHOST_USER_SET_FEATURES] = 'set_features',
[C.VHOST_USER_SET_OWNER] = 'set_owner',
[C.VHOST_USER_RESET_OWNER] = 'reset_owner',
[C.VHOST_USER_SET_MEM_TABLE] = 'set_mem_table',
[C.VHOST_USER_SET_LOG_BASE] = 'set_log_base',
[C.VHOST_USER_SET_LOG_FD] = 'set_log_fd',
[C.VHOST_USER_SET_VRING_NUM] = 'set_vring_num',
[C.VHOST_USER_SET_VRING_ADDR] = 'set_vring_addr',
[C.VHOST_USER_SET_VRING_BASE] = 'set_vring_base',
[C.VHOST_USER_GET_VRING_BASE] = 'get_vring_base',
[C.VHOST_USER_SET_VRING_KICK] = 'set_vring_kick',
[C.VHOST_USER_SET_VRING_CALL] = 'set_vring_call',
[C.VHOST_USER_SET_VRING_ERR] = 'set_vring_err',
[C.VHOST_USER_GET_PROTOCOL_FEATURES] = 'get_protocol_features',
[C.VHOST_USER_SET_PROTOCOL_FEATURES] = 'set_protocol_features',
[C.VHOST_USER_GET_QUEUE_NUM] = 'get_queue_num',
[C.VHOST_USER_SET_VRING_ENABLE] = 'set_vring_enable'
}

-- Process all vhost_user requests from QEMU.
@@ -178,6 +182,26 @@ function VhostUser:set_features (msg)
self.dev:set_features(features)
end

function VhostUser:get_protocol_features (msg)
msg.u64 = 0ULL -- no extensions supported for now
msg.size = ffi.sizeof("uint64_t")
self:reply(msg)
end

function VhostUser:set_protocol_features (msg)
-- ignore protocol features for now (FIXME)
end

function VhostUser:get_queue_num (msg)
-- ignore for now (FIXME)
end

-- Handle VHOST_USER_SET_VRING_ENABLE, which explicitly enables/disables the
-- ring (this msg is only used if VHOST_USER_F_PROTOCOL_FEATURES is used)
function VhostUser:set_vring_enable (msg)
self.vhost_ready = msg.u64 ~= 0
end

-- Feature cache: A kludge to be compatible with a "QEMU reconnect" patch.
--
-- QEMU upstream (circa 2015) does not support the vhost-user device
@@ -137,6 +137,12 @@ function configure (new_config)
counter.add(configs)
end


-- Stop all apps by loading an empty configuration.
function stop ()
configure(config.new())
end

-- Removes the claim on a name, freeing it for other programs.
--
-- This relinquish a claim on a name if one exists. if the name does not
@@ -724,6 +730,11 @@ function selftest ()
main({duration = 4, report = {showapps = true}})
assert(app_table.app3 ~= orig_app3) -- should be restarted

-- Check engine stop
assert(not lib.equal(app_table, {}))
engine.stop()
assert(lib.equal(app_table, {}))

-- Check one can't unclaim a name if no name is claimed.
assert(not pcall(unclaim_name))

@@ -457,84 +457,8 @@ function root_check (message)
end
end

-- Simple token bucket for rate-limiting of events. A token bucket is
-- created through
--
-- local tb = token_bucket_new({ rate = <rate> })
--
-- where <rate> is the maximum allowed rate in Hz, which defaults to
-- 10. Conceptually, <rate> tokens are added to the bucket each
-- second and the bucket can hold no more than <rate> tokens but at
-- least one.
--

local token_bucket = {}
token_bucket.mt = { __index = token_bucket }
token_bucket.default = { rate = 10 }
function token_bucket_new (config)
local config = config or token_bucket.default
local tb = setmetatable({}, token_bucket.mt)
tb:rate(config.rate or token_bucket.default.rate)
tb._tstamp = C.get_monotonic_time()
return tb
end

-- The rate can be set with the rate() method at any time, which fills
-- the token bucket an also returns the previous value. If called
-- with a nil argument, returns the currently configured rate.
function token_bucket:rate (rate)
if rate ~= nil then
local old_rate = self._rate
self._rate = rate
self._max_tokens = math.max(rate, 1)
self._tokens = self._max_tokens
return old_rate
end
return self._rate
end

function token_bucket:_update (tokens)
local now = C.get_monotonic_time()
local tokens = math.min(self._max_tokens, tokens + self._rate*(now-self._tstamp))
self._tstamp = now
return tokens
end

-- The take() method tries to remove <n> tokens from the bucket. If
-- enough tokens are available, they are subtracted from the bucket
-- and a true value is returned. Otherwise, the bucket remains
-- unchanged and a false value is returned. For efficiency, the
-- tokens accumulated since the last call to take() or can_take() are
-- only added if the request can not be fulfilled by the state of the
-- bucket when the method is called.
function token_bucket:take (n)
local n = n or 1
local result = false
local tokens = self._tokens
if n > tokens then
tokens = self:_update(tokens)
end
if n <= tokens then
tokens = tokens - n
result = true
end
self._tokens = tokens
return result
end

-- The can_take() method returns a true value if the bucket contains
-- at least <n> tokens, false otherwise. The bucket is updated in a
-- layz fashion as described for the take() method.
function token_bucket:can_take (n)
local n = n or 1
local tokens = self._tokens
if n <= tokens then
return true
end
tokens = self:_update(tokens)
self._tokens = tokens
return n <= tokens
end
-- Backward compatibility
token_bucket_new = require("lib.token_bucket").new

-- Simple rate-limited logging facility. Usage:
--
@@ -146,12 +146,12 @@ function allocate_huge_page (size, persistent)
local fd = syscall.open(tmpfile, "creat, rdwr", "RWXU")
assert(fd, "create hugetlb")
assert(syscall.ftruncate(fd, size), "ftruncate")
local tmpptr = syscall.mmap(nil, size, "read, write", "shared, hugetlb", fd, 0)
local tmpptr = syscall.mmap(nil, size, "read, write", "shared", fd, 0)
assert(tmpptr, "mmap hugetlb")
assert(syscall.mlock(tmpptr, size))
local phys = resolve_physical(tmpptr)
local virt = bit.bor(phys, tag)
local ptr = syscall.mmap(virt, size, "read, write", "shared, hugetlb, fixed", fd, 0)
local ptr = syscall.mmap(virt, size, "read, write", "shared, fixed", fd, 0)
local filename = ("/var/run/snabb/hugetlbfs/%012x.dma"):format(tonumber(phys))
if persistent then
assert(syscall.rename(tmpfile, filename))
@@ -160,6 +160,7 @@ function allocate_huge_page (size, persistent)
else
assert(syscall.unlink(tmpfile))
end
syscall.munmap(tmpptr, size)
syscall.close(fd)
return ptr, filename
end
@@ -52,6 +52,9 @@ Optional entries that may be present in the *parameters* table include:
2. Defaults to 0.9, for a 90% maximum occupancy ratio.
* `min_occupancy_rate`: Minimum ratio of `occupancy/size`. Removing an
entry from an "empty" table will shrink the table.
* `resize_callback`: An optional function that is called after the
table has been resized. The function is called with two arguments:
the ctable object and the old size. By default, no callback is used.

— Function **ctable.load** *stream* *parameters*

@@ -0,0 +1,70 @@
### Token Bucket (lib.token_bucket)

This module implements a [token
bucket](https://en.wikipedia.org/wiki/Token_bucket) for rate-limiting
of arbitrary events. The bucket is filled with tokens at a constant
rate up to a given maximum called the *burst_size*. Tokens are added
and removed in integer quantities. An event can only take place if at
least one token is available. A burst of back-to-back events is
allowed to happen by consuming all available tokens at a given point
in time. The maximum size of such a burst is determined by the
capacity of the bucket, hence the name *burst_size*.

The token bucket is updated in a lazy fashion, i.e. only when a
request for tokens cannot be satisfied immediately.

By default, a token bucket uses the `rdtsc` time source via the
[`tsc`](./README.tsc.md) module to minimise overhead. To override,
the `default_source` parameter of the `tsc` module must be set
to the desired value.

#### Functions

— Function **new** *config*

Creates an instance of a token bucket. The required *config* argument
must be a table with the following keys.

— Key **rate**

*Required*. The rate in units of Hz at which tokens are placed in the
bucket as an arbitrary floating point number larger than zero.

— Key **burst_size**

*Optional*. The maximum number of tokens that can be stored in the
bucket. The default is **rate** tokens, i.e. the amount of tokens
accumulated over one second rounded up to the next integer.

#### Methods

The object returned by the **new** function provides the following
methods.

— Method **token_bucket:set** [*rate*], [*burst_size*]

Set the rate and burst size to the values *rate* and *burst_size*,
respectively, and fill the bucket to capacity. If *rate* is `nil`,
the rate remains unchanged. If *burst_size* is `nil`, the burst size
is set to the number of tokens that will be accumulated over one
second with the new rate (like in the **new** function).

— Method **token_bucket:get**

Returns the current rate and burst size.

— Method **token_bucket:can_take** [*n*]

Returns `true` if at least *n* tokens are available, `false`
otherwise. If *n* is `nil`, the bucket is checked for a single token.

— Method **token_bucket:take** [*n*]

If at least *n* tokens are available, they are removed from the bucket
and the method returns `true`. Otherwise, the bucket remains
unchanged and `false` is returned. If *n* is `nil`, the bucket is
checked for a single token.

— Method **token_bucket:take_burst**

Takes all available tokens from the bucket and returns that number.
Oops, something went wrong.

0 comments on commit 771b55c

Please sign in to comment.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.