Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Customized test displaying error #203

Closed
olg33 opened this issue Mar 24, 2020 · 40 comments
Closed

Customized test displaying error #203

olg33 opened this issue Mar 24, 2020 · 40 comments

Comments

@olg33
Copy link

olg33 commented Mar 24, 2020

Hi,

I just received from a a colleague that is no longer with us a customized test file (.conf) to run bandwidth tests with DSCP marks for different types of services. It worked fine in his VM (Ubuntu 18.06).
Now, once I imported the same file on directory /usr/share/flent/flent/tests of my VM and run the test, it displays the following alarm:

ERROR: Unable to load JSON from 'FILE_NAME.conf' : Expecting value: line 1 column 1 (char 0)

Just compared the customized file with others in the /tests subdirectory but I couldn't find any sentence or command out of the ordinary, so at this point I can't figure out what the issue might be. Part of the problem is the fact that I lack basic documentation on how to create customized flent tests, not even the flent site seems to have any.

Any help on the above issues would be appreciated.

Thank you.
@tohojo
Copy link
Owner

tohojo commented Mar 24, 2020 via email

@olg33
Copy link
Author

olg33 commented Mar 24, 2020

Thanks Tohojo,

That was indeed the problem BUT, now I'm getting the following:

ERROR: Unable to read the test config file '/usr/share/flent/flent/tests/TEST1.conf': 'name 'find_netperf' is not defined'

Please note that this time I only typed the name of the test without the ,conf extension .

On the other hand, I'm definitively interested in trying the new per-flow DSCP marks feature in flent,

I'm aiming at being able to run bandwidth tests where we can create marked traffic, something like:

Traffic Class DSCP Marking

Web Browsing ------> 11
Streaming -----------> 13
Instant Messaging --->15
Collaboration -------> 17
File transfer --------> 19
Default -------------> 10
Scavenger ----------> 1

Do you believe that could be possible? if so, what could be the additional parameters required to, let say, run a ruul test creating traffic with different marks.

The customized file is a modified version of the standard RUUL test which I'm attaching (TEST1,conf) so you can have a better idea of what I'm trying to do. It was made focusing on our particular needs to test throughput availability on remote locations, so I'm not sure how generic it can be to include it in the repository.

Thanks again and looking forward hearing from you.

TEST1.conf.txt

@tohojo
Copy link
Owner

tohojo commented Mar 26, 2020 via email

@olg33
Copy link
Author

olg33 commented Mar 26, 2020

Hi Tohojo,

Do yo believe that this proposed rrul_var test will be able to saturate the link? The main purpose of our testing is to know how much bandwidth we can pull in a single link, so far the rrul test has made a good job.

Thanks.

@tohojo
Copy link
Owner

tohojo commented Mar 26, 2020 via email

@olg33
Copy link
Author

olg33 commented Mar 26, 2020

That sounds great!

Looking forward hearing from your results.

In the above example I showed 7 kinds of traffic with different DSCP masks but hopefully we can mark up to 15 or more.

Thanks.

@olg33
Copy link
Author

olg33 commented Mar 26, 2020

Just tried tcp_nup with the parameters you provided and yes I can make 15 traffic markings. Cool.

@tohojo
Copy link
Owner

tohojo commented Mar 31, 2020

Sorry for the delay in getting to this. Please see if the rrul_var test introduced in d06e0b6 solves this for you.

Specifically, you should be able to run this:

flent rrul_var --test-parameter bidir_streams=7 --test-parameter markings=11,13,15,17,19,10,1

to get what you're after :)

@flent-users
Copy link

flent-users commented Apr 1, 2020 via email

@tohojo
Copy link
Owner

tohojo commented Apr 1, 2020 via email

@flent-users
Copy link

flent-users commented Apr 1, 2020 via email

@tohojo
Copy link
Owner

tohojo commented Apr 1, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 2, 2020

Hi Tohojo,
Sorry I was also busy. Let me tell what I did:

First I copied the new rrul_var.conf test file you created in my local directory /usr/share/flent/flent/tests
Then I typed:

flent rrul_var --test-parameter bidir_streams=7 --test-parameter markings=11,13,15,17,19,10,1 -H 172.11.85.2

But after that, I get the following alarm:

ERROR: Unable to read test config file '/usr/share/flent/flent/tests/rrul_var.conf': 'get_test_parameter() got an unexpected keyword argument 'cast''.

Am I missing any step on adding the new test? Below is the version I'm using:

Flent v1.2.2.
Running on Python 3.6.9 (default, Nov 7 2019, 10:44:02) [GCC 8.3.0].
Using matplotlib version 2.1.1 on numpy 1.13.3.
Using PyQt5 version 5.10.1.

Please let me know if you need further information.

Thank you.

@tohojo
Copy link
Owner

tohojo commented Apr 3, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 4, 2020

Thanks Tohojo,

I just cloned the repository and ran Flent using the run-Flent script. It's awesome! I tested several scenarios and in all cases it worked as expected. I just Have a question, how is the bandwidth assigned to each marked data stream? for instance, in one test I got the following bandwidth for these TCP streams:

TCP download::0 (11) : 85.08 86.78 Mbits/s
TCP download::1 (15) : 52.73 53.28 Mbits/s
TCP download::1 (10) : 57.78 58.45 Mbits/s

Are these values 86.78Mb/s, 53.28Mb/s and 58.45 Mb/s assigned ramdomly? What's the criteria to distribute the bandwitdth this way? or, can it be assigned via parameter, like a percentage of the total bandwidth available?

Finally, I'd like to properly install the cloned version of Flent via "make install". Normally there is an autogen.sh and/or a configure script we run before proceeding with the compilation and file creation -make install- but I can't find these files in the cloned file system. Could you tell me where to find these files or if they have a different name?

Thank you so much for all your kind help.

@flent-users
Copy link

flent-users commented Apr 4, 2020 via email

@tohojo
Copy link
Owner

tohojo commented Apr 6, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 8, 2020

Thanks Toke and Dave

I have been testing the marking version intensively and as mentioned it's working really well. However I just found that when I put a sniffer and graph the marked TCP streams with Netdata, the Tx flow graph shows the marked data, but on the Rx side the traffic returns unmarked.

In order to double-check I captured packets on both the Tx and Rx interfaces. On the Tx side I was able to see the marks:

Differentiated Services Field: 0x98 (DSCP: AF43, ECN: Not-ECT)
Differentiated Services Field: 0x28 (DSCP: AF11, ECN: Not-ECT)
Differentiated Services Field: 0x88 (DSCP: AF41, ECN: Not-ECT)
Differentiated Services Field: 0x48 (DSCP: AF21, ECN: Not-ECT)
Differentiated Services Field: 0x20 (DSCP: CS1, ECN: Not-ECT)

But on Rx, all traffic has the "default" mark:

Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)

That's why I'd like to ask you, is this an expected outcome or should I review my setup?

In my test lab, the client side has the latest cloned Flent version and netperf 2.7.0 while the server site has also netperf 2.7.0 running netserver via the "netserver &" command, which I'm not sure it's enough to send back marked tcp traffic to the client side.

Thanks for your help.

@tohojo
Copy link
Owner

tohojo commented Apr 8, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 8, 2020

The server and client are in different locations connected via corporate VPN, which also may be an issue, so I'll test in a server located in the same building and subnet.

Thanks!

@olg33
Copy link
Author

olg33 commented Apr 27, 2020

Hi Toke,
After testing with two co-located servers and tweaking the switch configuration that connects them, all is working well. Thanks.

Now I'd like to test in a production network. Because we work in a multi-vendor environment (Riverbed, Procera, Idirect, etc) we had to create a customized DSCP naming convention which is based on the TOS Hexadecimal value and its DSCP decimal number, something like this:

https://support.riverbed.com/bin/support/static/n0glmtc5kegfi7htfe7msdsgcb/html/uc6d3bbkbug83sroi75ersdj9i/sc_ug_html/index.html#page/sh_sd/app_tosdscp_qosclass.html

Therefore, instead of AF11, we use as a name the DSCP number 10 whose hex value would be 0x28. We have a total of 12 DSCP values for our traffic markings.

I was thinking that perhaps it's a matter of adding these values to the list of supported symbolic values which in our case may look like this:

MARKING_MAP = {'10': 0x28,
'11': 0x2c,
'13': 0x34,
'15': 0x38,
'17': 0x40,
'19': 0x4c,
'20': 0x50,
'21': 0x54,
'23': 0x5c,
'25': 0x64,
'27': 0x6c,
'29': 0x74}

So that, if I enter parameters:

--test-parameter markings=10,13,15

Packets would carry ToS hex value:

Differentiated Services Field: 0x28 (DSCP: 10, ECN: Not-ECT)
Differentiated Services Field: 0x34 (DSCP: 13, ECN: Not-ECT)
Differentiated Services Field: 0x38 (DSCP: 15, ECN: Not-ECT)

Do you think this is possible? or what would be the right approach?

Thanks a lot.

@tohojo
Copy link
Owner

tohojo commented Apr 28, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 28, 2020

Thanks Toke,
I cloned the repository in a brand new VM, Once there I modified the runners.py file removing the previous MARKING_MAP with my customized values. Then I ran a test using your second proposed method:

./run-flent rrul_var --test-parameter bidir_streams=3 --test-parameter markings=0x28,0x34,0x38 --test-parameter labels='10','13','15' -H 192.168.9.13 -l 60

The packet capture shows the right hex values but seems like the labels are not going thru:

Differentiated Services Field: 0x34 (DSCP: Unknown, ECN: Not-ECT)
Differentiated Services Field: 0x38 (DSCP: AF13, ECN: Not-ECT)
Differentiated Services Field: 0x28 (DSCP: Unknown, ECN: Not-ECT)

On cases where the hex value matches a value in the previous Marking_Map, like 0x38, it fills the DSCP label AF13. I also ran a test with no quotes on the labels but got the same result. Needless to say first method didn't work neither.

./run-flent rrul_var --test-parameter bidir_streams=5 --test-parameter markings=10,13,15,17,19 -H 10.200.9.13 -l 60

So, it seems like I'm missing something. I haven't properly installed flent (setup.py install) as I though modifying the marking_map and using the run-flent command would be enough.

Is that the case?

Thank you.

@flent-users
Copy link

flent-users commented Apr 28, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 28, 2020

Right, using standard markings would make everyone's lives easier. But as mentioned, we have some providers whose equipment don't follow the rules.

https://support.riverbed.com/bin/support/static/n0glmtc5kegfi7htfe7msdsgcb/html/uc6d3bbkbug83sroi75ersdj9i/sc_ug_html/index.html#page/sh_sd/app_tosdscp_qosclass.html

Right now we are stuck with this scheme until we move to something more standard.

Thanks for the info.

@flent-users
Copy link

flent-users commented Apr 28, 2020 via email

@tohojo
Copy link
Owner

tohojo commented Apr 28, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 28, 2020

Where are you getting these values from? Tcpdump? The labels are not sent on the wire at all (and nothing you can do to change that).

The 'labels' parameter should set the plot labels on the Flent plots, though...

It's Wireshark and I'm only showing the relevant information. Ok good to know, the labels are just for displaying purpose, in fact I'm not too worried about the labels as I'm feeding the output to Netdata for graphing.

My main concern is that (at least for me) the new MARKING_MAP doesn't seem to be working.
MARKING_MAP = {
'10': 0x28,
'11': 0x2c,
'13': 0x34,
'15': 0x38,
'17': 0x40,
'19': 0x4c,
'20': 0x50,
'21': 0x54,
'23': 0x5c,
'25': 0x64,
'27': 0x6c,
'29': 0x74}
If I run the following test (after editing the MARKING_MAP with custom settings):

./run-flent rrul_var --test-parameter bidir_streams=5 --test-parameter markings=10,13,15,17,19 -H 192.168.9.13 -l 60

I only see packets marked with Hex values that are not part of the map:

Differentiated Services Field: 0x08 (DSCP: Unknown, ECN: Not-ECT)
Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT)
Differentiated Services Field: 0x0c (DSCP: Unknown, ECN: Not-ECT)

That's why I do believe I'm doing something wrong or maybe there was an additional step other than just editing the Marking_Map.

In fact, using just the hex values works for me, but if someone else that only knows the custom dscp values want to test, that will fail, that's why I'd like to be able to feed those values.

Thank you.

@tohojo
Copy link
Owner

tohojo commented Apr 28, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 28, 2020

Ohh, right, the markings_map is only used for irtt and iperf, for
netperf the values are just passed directly through. Totally forgot
about that. I guess I could fix that, but TBH I'm not terribly keen on
supporting non-standard diffserv markings in this way.

You are totally right, I'm not expecting you to do my job, that's unfair and you have been very generous already. I'm just looking for clues (like the one provided) to fix my setup.

Just checking, but when you say "someone else" you mean someone else who
has access to that same system, right?

Correct. Some Jr. members of my engineering team are only familiar with decimal values, no hex, so I want to help them out on that.

@flent-users
Copy link

flent-users commented Apr 28, 2020 via email

@tohojo
Copy link
Owner

tohojo commented Apr 28, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 28, 2020

It definitively will work for me and for other general situations as I may not be the only person that is using non-standard DSCP markings.

@olg33
Copy link
Author

olg33 commented Apr 29, 2020

if you are in search of entertainment, have 'em watch:
https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/

Thanks Dave, and congrats for such an illustrative presentation :)

@tohojo
Copy link
Owner

tohojo commented Apr 29, 2020

Right, pushed as cf4f3da - the commit message
explains the usage :)

@olg33
Copy link
Author

olg33 commented Apr 29, 2020

The marking names can also be specified in the RC file; the rcfile with the
definitions above would look like this:

[global]
marking_names=foo=0x28;bar=0x23

Thanks Toke, just to double-check, the RC file you are referring to is the ~/.bashrc file or something else?

@tohojo
Copy link
Owner

tohojo commented Apr 29, 2020 via email

@olg33
Copy link
Author

olg33 commented Apr 30, 2020

Perfect. I just ran a test with 12 streams (an extreme scenario for me) and all went well. Checking the packet capture I could confirm that every stream was marked with the hex value defined in the RC file. I'm glad that I was the first beneficiary of this new feature.

Thanks for all the help!

@tohojo
Copy link
Owner

tohojo commented Apr 30, 2020 via email

@tohojo tohojo closed this as completed Apr 30, 2020
@flent-users
Copy link

flent-users commented Apr 30, 2020 via email

@olg33 olg33 mentioned this issue Dec 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tohojo @flent-users @olg33