Can not start muxy proxy with a network_shape configuration. #23

yangyongzhi7 · 2018-08-21T02:45:46Z

My muxy.config:

proxy:
  - name: http_proxy
    config:
      host: 0.0.0.0
      port: 8527
      protocol: http
      proxy_host: 192.168.8.228
      proxy_port: 8527
      proxy_protocol: http
  - name: tcp_proxy
    config:
      host: 0.0.0.0           # Local address to bind to and accept connections. May be an IP/hostname
      port: 9527              # Local port to bind to
      proxy_host: 192.168.8.228     # Proxy server port
      proxy_port: 9527        # Proxied server port
      nagles_algorithm: true  # Use Nagles algorithm?
      packet_size: 64         # Size of each contiguous network packet to proxy
middleware:
  - name: logger
    config:
      hex_output: false
  - name: delay
    config:
      request_delay: 2000
      response_delay: 1500
  - name: network_shape
    config:
      latency:     1000        # Latency to add in ms
      target_bw:   10         # Bandwidth in kbits/s
      packet_loss: 0.9         # Packet loss, as a %
      target_ips:              # Target ipv4 IP addresses/CIDRs
        - "0.0.0.0/0"
      target_ips6:             # Target ipv6 IP addresses
        - "::1/128"
      target_ports:            # Target destination ports
        - "9527"               # - "1:65535"            # Ranges also valid
      target_protos:           # Target protocols
        - "tcp"
        - "udp"
        - "imp"

command

sudo ./muxy_bin proxy --config muxy_middleware.yml

logs

2018/08/21 10:38:45.999520 [INFO]		Loading plugin 	logger
2018/08/21 10:38:45.999540 [INFO]		Loading plugin 	delay
2018/08/21 10:38:45.999549 [INFO]		Loading plugin 	network_shape
2018/08/21 10:38:45.999706 [INFO]		Loading proxy 	http_proxy
2018/08/21 10:38:45.999733 [INFO]		Loading proxy 	tcp_proxy
2018/08/21 10:38:45.999741 [DEBUG]		Delay Symptom - Setup()
2018/08/21 10:38:45.999748 [DEBUG]		NetworkShaperSymptom - Setup()

and Muxy process exited.

I viewed the source code, I think that this code will has a panic:

network_shape.go - 62

executeThrottler(&s.config)

how can I solve this problem? thanks.

The text was updated successfully, but these errors were encountered:

yangyongzhi7 · 2018-08-21T07:04:18Z

I debug this code, and got a error:

func addRootQDisc(cfg *Config, c commander) error {
	//Add the root QDisc
	log.Debug(cfg.Device)
	root := fmt.Sprintf(tcRootQDisc, cfg.Device)
	strs := []string{tcAddQDisc, root, "htb", tcRootExtra}
	cmd := strings.Join(strs, " ")
	log.Debug(cmd)

	return c.execute(cmd)
}

output

2018/08/21 14:46:08.458131 [DEBUG]		eth0
2018/08/21 14:46:08.458140 [DEBUG]		sudo tc qdisc add dev eth0 handle 10: root htb default 1

I execute this command directly, and got a error too:

root@appframe205 muxy]# sudo tc qdisc add dev eth0 handle 10: root htb default 1
RTNETLINK answers: File exists

yangyongzhi7 · 2018-08-21T07:55:32Z

I debug the final executed command like this:

sudo tc qdisc add dev eth0 parent 10:10 handle 100: netem delay 1000ms rate 20kbit loss 0.90%

What is "rate"?
Usage: ... netem [ limit PACKETS ]
                 [ delay TIME [ JITTER [CORRELATION]]]
                 [ distribution {uniform|normal|pareto|paretonormal} ]
                 [ drop PERCENT [CORRELATION]]
                 [ corrupt PERCENT [CORRELATION]]
                 [ duplicate PERCENT [CORRELATION]]
                 [ reorder PRECENT [CORRELATION] [ gap DISTANCE ]]

It looks like that this command has syntax error ?

mefellows · 2018-08-22T05:36:59Z

Thanks for the report. Just an aside, you need to use three backticks (`) for your code gates, otherwise it doesn't format properly. I've updated your comments to aid readability.

In terms of the bug, I'll need to dig into that command as I've forgotten the details of how it runs.

Also, please provided details of your operating environment so we can track versions of OS etc.

yangyongzhi7 · 2018-08-23T03:29:44Z

First of all thank you for the formatting problem!
and I removed the 'rate' parameter from the command and it will start successfully.
comment out the following code:
at vendor\github.com\tylertreat\comcast\throttler\tc.go

func addNetemRule(cfg *Config, c commander) error {
	......

	if cfg.TargetBandwidth > -1 {
		// If you used 'rate' in netem, it will has an error at centos.
		//strs = append(strs, fmt.Sprintf(tcRate, cfg.TargetBandwidth))
	}

	......
}

Operating system

Linux appframe205 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

CentOS release 6.10 (Final)

If you think of my modification method can be used, I'm very happy to submit a pr.
Thanks again!

yangyongzhi7 · 2018-08-23T03:35:46Z

By the way, the version I run here is obtained by compiling the master branch.

mycargus · 2019-05-02T14:47:45Z

Muxy doesn't work for us with a network_shape config either. We're running muxy v0.0.5 in docker on MacOSX 10.4.4.

Here's the config:

---
## Test configuration name. Used for reporting.
name: Upstream Packet Loss

## Test Description. Used for reporting
description: A failing upstream network

## Specify log output level
##
## Log Levels supported:
## Trace (0), Debug (1), Info (2, Default), Warn (3), Error (4), Fatal (5)
loglevel: 0

# Configures a proxy to forward and/or mess with your HTTP requests
proxy:
  - name: http_proxy
    config:
      host: 0.0.0.0
      port: 8181
      # proxy_host, i.e. the service in front of which we will run muxy
      proxy_host: http://mock-api
      proxy_port: 8080

# Proxy plugins
middleware:
  - name: network_shape
    config:
      packet_loss: 0.9         # Packet loss, as a %
      target_ports:            # Target destination ports
        - "8181"
      target_protos:           # Target protocols
        - "tcp"

  # Log in/out messages
  - name: logger

Here are the muxy container logs:

2019/05/02 14:42:43.137492 Setting up PKI for 'localhost'...
2019/05/02 14:42:43.832121 [INFO]		Loading plugin 	network_shape
2019/05/02 14:42:43.832160 [INFO]		Loading plugin 	logger
2019/05/02 14:42:43.832228 [INFO]		Loading proxy 	http_proxy
2019/05/02 14:42:43.832276 [DEBUG]		NetworkShaperSymptom - Setup()
2019/05/02 14:42:43.858740 [INFO]		HTTP proxy listening on http://0.0.0.0:8181
2019/05/02 14:42:43.858878 [TRACE]		sudo tc qdisc show | grep "netem"
sudo tc qdisc add dev eth0 handle 10: root htb default 1
sudo tc class add dev eth0 parent 10: classid 10:1 htb rate 1000000kbit
sudo tc class add dev eth0 parent 10: classid 10:10 htb rate 1000000kbit
sudo tc qdisc add dev eth0 parent 10:10 handle 100: netem loss 0.90%!
(MISSING)Packet rules setup...
Run `sudo tc -s qdisc` to double check
Run `muxy --device eth0 --stop` to reset

Of course there's a chance our config is invalid. 😄

mefellows · 2019-05-09T11:33:41Z

Hi @mycargus - long time no see!

Thanks, yes I recall this being an issue. Let me see if I can repro again and get to the bottom of it.

Weird the issue is happening in Docker though, I only noticed the issue on OSX previously.

* Vendor comcast library and modify (remove exits and update print statements to use logger) * Log messages from comcast appropriately to TRACE See #23

mefellows · 2019-05-09T14:06:38Z

I've created a small branch to test out an alternative approach. See https://github.com/mefellows/muxy/releases/tag/v0.0.6 to test the binary out.

If this proves to be working better, I'll get thiss into mainline.

The main issue is that comcast isn't really designed for lib use in this way - its use of os.Exit() is problematic.

mycargus · 2019-05-09T15:16:15Z

Haha long time indeed. :) Thanks so much for working on this, man! I'll try it out tomorrow (I'm away from my computer today) and share feedback here.

mycargus · 2019-05-09T15:21:18Z

By the way, we're using muxy at Instructure now. Thanks for yet another excellent testing tool!

mefellows · 2019-05-09T20:47:06Z

Oh that's so cool! I'd love to hear a bit more about it next time we chat. In the meanwhile, I'll take another look this weekend across the different OS's. The main change you should see above is that any notices/errors will start to be logged (some always, some others at TRACE level only).

So if it does fail, we'll at least find out why.

mycargus · 2019-05-10T19:58:20Z

Alrighty, I tried out the 0.0.6 muxy release in a docker container. Here are the container logs:

2019/05/10 19:51:57.242275 Setting up PKI for 'localhost'...
2019/05/10 19:51:57.917398 [INFO]		Loading plugin 	network_shape
2019/05/10 19:51:57.917449 [INFO]		Loading plugin 	logger
2019/05/10 19:51:57.917544 [INFO]		Loading proxy 	http_proxy
2019/05/10 19:51:57.917573 [DEBUG]		NetworkShaperSymptom - Setup()
2019/05/10 19:51:57.962878 [INFO]		HTTP proxy listening on http://0.0.0.0:8181
2019/05/10 19:51:57.962881 [TRACE]		sudo tc qdisc show | grep "netem"
sudo tc qdisc add dev eth0 handle 10: root htb default 1
sudo tc class add dev eth0 parent 10: classid 10:1 htb rate 1000000kbit
sudo tc class add dev eth0 parent 10: classid 10:10 htb rate 1000000kbit
sudo tc qdisc add dev eth0 parent 10:10 handle 100: netem loss 0.90%!
(MISSING)Packet rules setup...
Run `sudo tc -s qdisc` to double check
Run `muxy --device eth0 --stop` to reset

I then ran sudo tc -s qdisc as suggested:

docker run --rm --privileged muxy bash -c "sudo tc -s qdisc"
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

I'm not sure what to think of it. Lemme know if I can provide any other info that might help!

mefellows · 2019-05-11T05:23:02Z

I've just got it running now in a local Docker environment - things seem to be working nicely. I think it's fixed on OSX now also.

I've added curl and python to the container to do some testing
Build container: docker build -t muxy .
Run container

docker run \
  -p 8181:8181 --name muxy \
  -it \
  --rm \
  -v /tmp/muxy_test/muxy_linux_amd64:/opt/muxy/bin/muxy \
  -v /tmp/muxy_test/config.yml:/opt/muxy/conf/config.yml \
  --privileged \
  muxy bash

Create a few interactive shells i.e. docker exec -it muxy bash
python -m SimpleHTTPServer 8080 in any directory as a proxy target
run ./bin/muxy proxy --config ./conf/config.yml with the below config
in another Docker terminal, run time curl -I localhost:8181 to hit the proxy and witness the delay
Terminate the muxy process and run the curl again to observe conditions resetting

Dockerfile:

FROM debian:latest
MAINTAINER Matt Fellows <m@onegeek.com.au>

RUN apt-get update && apt-get install -y wget unzip iptables iproute net-tools sudo
RUN mkdir -p /opt/muxy/bin

WORKDIR /opt/muxy

ENV PATH /opt/muxy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# For testing / demo purposes
RUN apt-get install -y netcat curl python
VOLUME ["/opt/muxy/conf"]

CMD ["muxy", "proxy", "--config", "/opt/muxy/conf/config.yml"]

Muxy config:

---
name: Upstream Packet Loss

description: A failing upstream network
loglevel: 0

# Configures a proxy to forward and/or mess with your HTTP requests
proxy:
  - name: http_proxy
    config:
      host: 0.0.0.0
      port: 8181
      proxy_host: localhost
      proxy_port: 8080 # proxy the python process

# Proxy plugins
middleware:
  - name: http_tamperer
    config:
      response:
        status: 500

  - name: network_shape
    config:
      packet_loss: 40 # Packet loss
      target_ports: # Target destination ports
        - "8181"
        - "8080"
      target_protos: # Target protocols
        - "tcp"
      device: "lo" # shape only the local network, not the default of eth0

  # Log in/out messages
  - name: logger

screenshot where you can see the proxy working (manipulating response code) and also the network shape taking effect:

mefellows · 2019-05-11T05:27:40Z

I also just noticed in your previous comments that you ran docker run --rm --privileged muxy bash -c "sudo tc -s qdisc" after running the muxy image - if I understand correctly, you've run that in another docker container which won't have those rules applied - hence the response from qdisc doesn't show any rules.

You might want to test the command against the running container i.e. something more like docker exec --it <container id> bash -c "sudo tc -s qdisc"

mefellows · 2019-05-11T05:38:12Z

I've just pushed out an updated v6.0.0 binary pre-release if you'd like to try.

mycargus · 2019-05-13T16:19:03Z

Doh! Good catch, sorry about that. Trying now.

mycargus · 2019-05-13T18:14:54Z

Works great!

Adding the device: "lo" config did the trick.

Thanks for your help!

mefellows · 2019-05-13T21:13:56Z

Fantastic! I wonder if the loopback device will catch others too? I think I added it to the README in the most recent commit but will double check that. I'd love to hear how you're using Muxy and if there is anything I should be adding to make it better / more useful.

mefellows added the Triage label Aug 22, 2018

mefellows added a commit that referenced this issue May 9, 2019

fix(network): prevent exit on network shape errors

a4c36b1

* Vendor comcast library and modify (remove exits and update print statements to use logger) * Log messages from comcast appropriately to TRACE See #23

mefellows mentioned this issue May 9, 2019

network_shape middleware crashes Muxy #22

Closed

mefellows closed this as completed in 72e4787 May 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can not start muxy proxy with a network_shape configuration. #23

Can not start muxy proxy with a network_shape configuration. #23

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

mefellows commented Aug 22, 2018

yangyongzhi7 commented Aug 23, 2018 •

edited

Loading

yangyongzhi7 commented Aug 23, 2018

mycargus commented May 2, 2019 •

edited

Loading

mefellows commented May 9, 2019

mefellows commented May 9, 2019

mycargus commented May 9, 2019

mycargus commented May 9, 2019

mefellows commented May 9, 2019

mycargus commented May 10, 2019

mefellows commented May 11, 2019

mefellows commented May 11, 2019

mefellows commented May 11, 2019

mycargus commented May 13, 2019

mycargus commented May 13, 2019

mefellows commented May 13, 2019

Can not start muxy proxy with a network_shape configuration. #23

Can not start muxy proxy with a network_shape configuration. #23

Comments

yangyongzhi7 commented Aug 21, 2018 • edited by mefellows Loading

yangyongzhi7 commented Aug 21, 2018 • edited by mefellows Loading

yangyongzhi7 commented Aug 21, 2018 • edited by mefellows Loading

mefellows commented Aug 22, 2018

yangyongzhi7 commented Aug 23, 2018 • edited Loading

yangyongzhi7 commented Aug 23, 2018

mycargus commented May 2, 2019 • edited Loading

mefellows commented May 9, 2019

mefellows commented May 9, 2019

mycargus commented May 9, 2019

mycargus commented May 9, 2019

mefellows commented May 9, 2019

mycargus commented May 10, 2019

mefellows commented May 11, 2019

mefellows commented May 11, 2019

mefellows commented May 11, 2019

mycargus commented May 13, 2019

mycargus commented May 13, 2019

mefellows commented May 13, 2019

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

yangyongzhi7 commented Aug 21, 2018 •

edited by mefellows

Loading

yangyongzhi7 commented Aug 23, 2018 •

edited

Loading

mycargus commented May 2, 2019 •

edited

Loading