Enhanced Traffic Shaping

[bramhaghosh]

Objective

To enhance Martian’s trafficshape package so that it can simulate desired network conditions more precisely. These features are needed in order to allow developers to write tests that exercise application features that are designed to react to suboptimal network conditions.
Key features of this effort include:

• Apply traffic shaping behaviors on a per URL basis (based on a regex)
• Specify throughput (mbps) for byte ranges
• Specify hangs (with duration) or exits at byte offset

Background

These features are useful to anyone who is interested in writing tests that simulate poor network conditions: especially any tests that are serving video. More specifically, the test authors will be able to specify the “shape” of the video traffic as a configuration in the test and write assertions based on the intended behavior of client in dealing with the traffic.

Currently, the trafficshape package only allows setting the bandwidth and latency for all the requests and responses; the user cannot pick a few resources for which to shape traffic. This is insufficient for the testing situation where one wants to throttle a particular resource while letting the rest of the test infrastructure use the network normally.

The trafficshape package also does not have a way to keep track of what byte offsets are currently being written back to the client for a specific resource. This can be useful when the user wants to perform different actions or have different bandwidths depending on what byte ranges are being written back to the client.

Requirements

• Traffic shaping behavior is configured from within a test by issuing a POST request with a JSON configuration to a new configuration endpoint /configure/trafficshape
• Traffic shaping behaviors are defined on a per resource basis
  • Match on URL or parts of URL (using martianurl.Matcher)
  • Match based on URL regex (needs a new matcher)
• Add the ability to specify
  • Downstream bandwidth for a given byte range
  • Hang for a given time at a specific byte
  • Close connection at a specific byte
  • Number of times to repeat this behavior

Design

JSON Configuration API

{
    "trafficshape": {
        "default": {
            "bandwidth": {
                "up": 10000000,  
                "down": 1000000
            },
            "latency": {
                "up": 1000, 
                "down": 100000 
            }
        },
        "shapes": [{
            "url_regex": "example.com/video/stream",
            "throttles": [{
                "throttle": {
                    "bytes": "100-150000",
                    "bandwidth": 100000
                }
            }],
            "halts": [{
                "halt": {
                    "byte": 200000,
                    "duration": 5000,
                    "count": 1
                }
            }],
            "close_connections":[{
                "close_connection": {
                    "byte": 250000,
                    "count": 1
                }
            }]
        }]
    }
}

Default Latency and Bandwidth

These are the values that will be applied for the byte ranges that do not have throttles specified for them.

Throttle

Every throttle action takes in a byte range and a bandwidth. When the proxy is writing back bytes that fall in the range of a throttle’s byte range for the current url, it will be subject to the corresponding bandwidth.

Halt

Every halt action takes in a byte, duration and count (Explained below). The byte specifies at which byte to halt writing data back to the client and duration specifies how long to remain in this state before starting to write data back to the client.

Close_connection

Every close_connection action takes in a byte and a count (Explained below). The byte specifies what byte the proxy should be about to write back to the client before it should close the current connection with the client.

Count

Note the “count” property in the “halt” and “close_connection” actions . It is possible that in the course of writing the video, the client asks for byte range multiple times (this might happen when the existing connection is interrupted, and the client didn’t read part of the data written back by the proxy already, so it asks again for a range of the data that we have already written). So it is possible that they would want the action to be performed only the first time a specific byte is written back, or possibly more times. That is what the count specifies. A count of -1 (default) will specify that the action will happen everytime the specific byte is written back.

So the example json says:

• Make the default bandwidth 100,000mbs, and do not add any additional latency.
• And for any response containing a substring matching the url regex example.com/video/stream:
  • Write the bytes 100 to 150,000 (150kb) (with 150kb being exclusive) to the client at the bandwidth of 50,000 (50kb) per second. For all the other bytes outside this range, write back at the default bandwidth.
  • Upon reaching the byte 200,000 (200kb), perform an interrupt for 5000ms. Do this only the first time you are writing the byte 200kb back to the client. Do not do anything if you are writing back byte 200kb again.
  • Upon writing the byte 250,000 (250kb), close the current connection with the client. Do this 1 time only the first time when writing the byte 250kb back. (After we do this, the client may start a new connection and request the rest of the video, and we don’t want to perform the “close” again. Note, we almost always overestimate the data we write back to the client, so if we perform the close at byte 250kb, the client would probably ask for bytes, say 200kb onwards in a following request, and we do not want to close the connection again when we reach byte 250kb.)

For the byte_ranges, we support ranges to be expresses as “-5000”, which means “up till but not including byte 5000”, or “5000-”, which would mean from “5000 onwards”.