twamp toolchain & tests #16
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run the command: sed -i 's/^\."/\.\\"/' *.man *.ms over the man pages to eliminate warnings about macro " being undefined. The macro for a comment is "/ instead.
Initial changes to support RFC5357 TWAMP protocol which is based on and very similar to the OWAMP protocol.
Make twamp create a socket listening on the TWAMP TCP port, and allow it to accept connections. Create control state for the connection which is flagged as for a two way connection.
Allow twping to open a socket connecting to a TWAMP server, using the well-known TWAMP TCP port by default. Flag the created control state as for a two way connection.
Handle the new TWAMP TestRequestTW message, which is similar to the TestRequest message but without the Schedule Slot Descriptions and only one HMAC. Send negative accept messages for unexpected messages in the TWAMP server. Don't handle FetchSession message in the TWAMP server, and don't send StopSessions messages. The StopSessions message for twoway connections doesn't have any session description records.
Adapt twping so that it requests just one two-way test session, rather than a sender and a receiver session that owping does. Remove command line options that don't make sense as a result. Don't send a SID in the session request message for TWAMP. Ensure it's filled in based on the value in the response. Disallow use of OWPFetchSession for TWAMP control sessions.
Initialise endpoint for twoway sessions, using twoway packet size to allocate payload buffer and set expected payload length. Re-use existing code for creating datafile for two-way client and setting socket options for receiving TTL and ensuring send/receiver buffers are large enough. Setup a second AES key for the two-way test sessions so that they can both encrypt and decrypt packets.
Implement reflector for TWAMP server and TWAMP client sender/receiver.
Create a new format for two-way records, consisting simply of two one-way records. Use the top bit of the version field to distinguish between the two formats without having to define a new header format. Add two-way stats records from the two-way client. Upon summarising, display two-way packet delay variance calculated from round trip time (client received - client sent) minus the packet processing delay (reflector sent - reflector received), with the error being accumulated from these. Display the min/max reflector processing time. Reorder buffers aren't needed for TWAMP as twping waits for a sent packet to be received before sending the next packet, so disable the use and reporting of them for TWAMP.
Don't use owampd config, policy and state files for twampd. Make it use its own ones. Fix memory leaks in OWPDPolicyInstall.
Add documentation for twampd, extracting out command line options common to both that are more likely than other details to change in the future. Add documentation for twping based on owping.
Add support for configurable REFWAIT timeout (idle test session timeout) as per TWAMP RFC. Add a new global config option, testtimeout to change this from the default.
The min, median and max delay values for the forward and backward directions are now displayed after a twping session. PDV for the forward and backward directions is also now shown. Min/max processing delay is now shown in the machine readable output. Median values for all directions are now displayed in the machine readable output. PDV is now shown in the machine readable output.
This prevents a "TTL not reported" message when there are no hops. This also ensures that the min and max TTL and buckets are displayed in the machine readable output. MAXTTL in the machine readable output now displays the maximum TTL, not the minimum TTL.
Previously, a maximum of 255 counts could be represented for each TTL value. This commit ensures that a count can be represented for each test packet for sessions with up to the maximum allowed number of test packets.
Both the forward and backward hop count is now displayed after a twping session. In the machine readable output, the min and max TTL is shown for both directions. The bucket output now also shows counts for both directions.
Duplicate packet statistics are now calculated and displayed for both the forward and backward directions.
|
Tried to test with twamp-gui but it does not work in either direction: 100% packet loss. The TCP control connection setup seems to work, but there is something broken with the UDP test packets. |
|
I found |
|
Communicating with Cisco CSR1000v virtual router also shows some protocol problems: Testing the same router with twamp-gui works without errors. |
|
Hello, I'm the original author of some of these patches. Thanks for picking these up and working at integrating them. I've pushed some additional patches here: https://github.com/deastoe/owamp/commits/twamp-fixes. These should fix the inter-op issues with the Cisco CSR1000v and with the twamp-gui sender. A quick test using |
|
After adding |
|
Some tests are still failing when building as normal user, Erik can you check that? |
|
Do these tests fail every time you run make check? So far I'm not able to reproduce this ... under which environment do you run the tests? Incidentally, the "waiting for server: No such file or directory" messages are expected and are not an error. The unit test just tries a few times waiting for the simulated server to startup before starting the ping client, and for simplicity prints errstring until it connects. Those messages don't appear in the production code. |
|
On Mon, Sep 11, 2017 at 06:46:21AM -0700, Erik Reid wrote:
Do these tests fail every time you run make check? So far I'm not
able to reproduce this ... under which environment do you run the
tests?
Incidentally, the "waiting for server: No such file or directory"
messages are expected and are not an error. The unit test just tries
a few times waiting for the simulated server to startup before
starting the ping client, and for simplicity prints errstring until it
connects. Those messages don't appear in the production code.
I run the build under Debian jessie. The message "waiting for server"
appears once but than the test exits with 1. Maybe the test could print
out some more info so we can figure out what is failing?
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
I added a few progress messages in a temporary branch called twamp_test_debugging. Can you try this on your system? I also noticed one branch where it could have been returning with an error and a proper message wasn't printed, so this might be where it's failing in your environment. |
|
Here is the output now: |
|
maybe a timing problem ...? I added a few more debug messages on that branch. Can you try again on your system? |
|
The tests are passing with this patches now, can you make the threads behave so we don't have this timing problems? |
|
I took out most of those debugging messages to ensure I found the race condition and the additional printing isn't affecting the timing. I put the change on the new branch: twamp_test_bugfix Can you confirm the test pass on your system using this branch? |
|
On Mon, Sep 11, 2017 at 08:21:57AM -0700, Erik Reid wrote:
I took out most of those debugging messages to ensure I found the race
condition and the additional printing isn't affecting the timing.
I put the change on the new branch: twamp_test_bugfix
Can you confirm the test pass on your system using this branch?
Yes, the tests seem to work now. Can you add this into the twamp
branch as I have some compile errors that I would like to fix.
BTW, what OS and compiler are you using for twamp?
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
merged that change onto twamp branch the build/test environment i'm working on right now is ubuntu xenial64, gcc 5.4.0, automake 1.15 ... I also see those warnings and will fix them today |
|
I pushed some changes that fix a few of those warnings. Which gcc -std option were used to generate those errors about loop variables? The gcc options pasted into the sample don't generate those in my current environment (which uses default -std value of gnu11). Requiring a very old standard like c90 then I can see those errors, as expected, but then the rest of the project can't be compiled. |
|
On Tue, Sep 12, 2017 at 04:30:28AM -0700, Erik Reid wrote:
I pushed some changes that fix a few of those warnings.
Which gcc -std option were used to generate those errors about loop
variables? The gcc options pasted into the sample don't generate
those in my current environment (which uses default -std value of
gnu11). Requiring a very old standard like c90 then I can see those
errors, as expected, but then the rest of the project can't be
compiled.
Not sure why the errors are appearing on Debian jessie (gcc 4.9.2),
but since the build was working without the twamp patch we need to
keep it like that. I've pushed 2 commits to twamp branch to fix
these problems. What remains are some warnings but they are comming
from libI2util.a so we can't fix them here.
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
Ok, that makes sense now. The gcc default language standard was changed from gnu90 to gnu11 in gcc 5.1.0. |
|
On Wed, Sep 06, 2017 at 02:36:07PM +0000, deastoe wrote:
A quick test using `twping` towards the twamp-gui reflector points to
an issue in twamp-gui. The reflected test packets had the error
estimate fields set to all zeros, which `twping` detected as invalid
and aborted the session.
I sent some patches to twamp-gui to fill the error estimate fields, so
the direction twping -> twamp-responder should work now, please test.
The other way aroud (twamp-client -> twampd) still has problems. The
problem seems to be in twampd this time because it does not recognize
sender/receiver address fields set to 0 and just drops all incoming
tests packets because of this:
The Sender Address and Receiver Address fields contain, respectively,
the sender and receiver addresses of the endpoints of the Internet
path over which a TWAMP-Test session is requested. They MAY be set
to 0, in which case the IP addresses used for the Control-Client to
Server TWAMP-Control message exchange MUST be used in the test
packets.
So we need to make a patch that would take the addresses from the TCP
control connection and use them for the UDP test packets if the
0-address was used during test setup.
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
On Tue, Sep 12, 2017 at 08:11:52AM -0700, Erik Reid wrote:
Ok, that makes sense now. The gcc default language standard was
changed from gnu90 to gnu11 in gcc 5.1.0.
Yes, the build works now on Debian wheezy and jessie, so we can try
the packaging now. Can you patch the configure script to check
for libssl-dev like we discussed?
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
Thanks - tests now appear to complete: Looks like
I have put out a fix for this in PR #21. |
|
On Wed, Sep 13, 2017 at 11:57:38AM +0000, Duncan Eastoe wrote:
Thanks - tests now appear to complete:
```Approximately 13.0 seconds until results available
Directional delays may be inaccurate due to out of sync clocks!
--- twping statistics from [10.10.1.2]: to [10.10.1.1]: ---
SID: 00000000000000000000000000000000
first: 2017-09-13T11:46:28.306
last: 2017-09-13T11:46:37.988
100 sent, 0 lost (0.000%), 0 send duplicates, 0 reflect duplicates
round-trip time min/median/max = 0.389/1.2/19.8 ms, (err=0.0322 ms)
send time min/median/max = 1.86e+03/1.86e+03/1.88e+03 ms, (err=0.0161 ms)
reflect time min/median/max = -1.86e+03/-1.86e+03/-1.86e+03 ms, (err=0.0161 ms)
reflector processing time min/max = 0/1 ms
two-way jitter = 4.5 ms (P95-P50)
send jitter = 3.1 ms (P95-P50)
reflect jitter = 3.5 ms (P95-P50)
send hops = 255 (consistently)
reflect hops = 191 (consistently)
```
Looks like `twamp-responder` is not generating/returning a session ID,
and also not setting the reflected test packet TTL values as per the
RFC:
Sent them a patch to return TTL of 255 since the real TTL value is
not available from the QUdpSocket class. Is there something wrong
with the timestamps too since this message appears:
Directional delays may be inaccurate due to out of sync clocks!
> So we need to make a patch that would take the addresses from the
> TCP control connection and use them for the UDP test packets if the
> 0-address was used during test setup.
I have put out a fix for this in PR #21.
Just merged this, thanks again :)
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
Do we want to use a new port range for TWAMP UDP tests or reuse the range from OWAMP? |
Thank you.
This message appears when any calculated directional delay is negative. This can occur for a number of reasons such as unsynced clocks, or when the latency between the sender and reflector is so small that the difference between the clocks of the sender and reflector becomes significant. Therefore you can see this message when using |
|
On Mon, Sep 11, 2017 at 02:14:14PM +0000, Erik Reid wrote:
I added a few progress messages in a temporary branch called
twamp_test_debugging. Can you try this on your system?
I also noticed one branch where it could have been returning with an
error and a proper message wasn't printed, so this might be where it's
failing in your environment.
One of the tests is failing on i386:
$ test/owstats
owstats: FILE=stats.c, LINE=1634, OWPStatsParse: Invalid sample range [4294967295,109]
OWPStatsParse failed
Can you check if this can be fixed?
…--
Valentin Vidic
Computer Systems Engineer - Expert
Department of Computer Infrastructure and Services
Croatian Academic and Research Network - CARNet
Josipa Marohnica 5, HR-10000 Zagreb, Croatia
tel: +385 1 6661 714, fax. +385 1 6661 635
gsm: +385 91 2480 919
www.CARNet.hr
|
|
If the test mesh is in a good state we could merge this to see if any problems with owamp appear? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The patch submitted some time ago by Robert Shearman and Duncan Eastoe was essentially working and stable. This patch includes those changes and unit tests for both owamp and twamp.