Greentea netsocket more tests #6665

VeijoPesonen · 2018-04-18T12:04:52Z

Description

Improving Greentea netsocket test case coverage

Pull request type

[X] Test case
[ ] Fix
[ ] Refactor
[ ] New target
[ ] Feature
[ ] Breaking change

VeijoPesonen · 2018-04-18T12:08:12Z

@kjbracey-arm would you please add DO NOT MERGE-label. There are more test cases to come. @SeppoTakalo, @jarlamsa, @juhaylinen, you can start reviewing if you want but it's not expected. At this stage I just want to make the stuff visible so you can see how things are proceeding.

VeijoPesonen · 2018-04-18T12:14:57Z

TESTS/netsocket/tcp_chargen_recv100k/main.cpp

Implementation provided by @SeppoTakalo

VeijoPesonen · 2018-04-20T14:23:42Z

tools/test_configs/ESP8266Interface.json

This won't work. Only IP address is accepted.

Where this limitation is?
Socket API is perfectly capable of accepting domain names.

Need to figure it out.

SeppoTakalo · 2018-04-20T14:35:47Z

TESTS/netsocket/tcp/tcpsocket_recv_100k.cpp

SocketAddress does not resolve names. It is just a placeholder for IP addresses.
Use NetworkInterface::gethostbyname() or feed the string to Socket which translates its string through the gethostbyname()

So for example TCPSocket::connect(const char *host, uint16_t port) does the translation.

Ok, I just need to remember to make the same change to all netsocket test cases.

VeijoPesonen · 2018-05-11T11:20:04Z

~~TCPSOCKET_RECV_100K_NONBLOCK and TCPSOCKET_RECV_100K are still unstable but people can start to review the other tests.~~

Wrong lwIP configuration (#define TCP_QUEUE_OOSEQ 0) was causing the issue

SeppoTakalo

Add error handling.

There are multiple cases where you call socket functions without testing the return code.
For example:

get_interface()->gethostbyname(...);
sock->open(...);
sock->connect(...);
sock->close();

Every socket call should be tested and compared against acceptable return values. Not only those that transfer the data, but also ones than translate names or create connection.

SeppoTakalo · 2018-05-14T13:23:52Z

TESTS/netsocket/tcp/main.cpp

Does the TEST_ASSERT() propagation work in greentea_setup()?
Are we able to see whether device did not join the network and therefore tests failed to run?

Yes, we are able to see if the problem arises from the handler. But there is still need to check the logs.

PS C:\ws\mbed-os> mbedhtrun -f .\BUILD\tests\K64F\ARM\TESTS\netsocket\udp\udp.bin -d D: -p COM14:9600 [1526317024.03][HTST][INF] host test executor ver. 1.2.0 [1526317024.03][HTST][INF] copy image onto target... 1 file(s) copied. [1526317032.47][MBED][WRN] Target ID not found: Skipping flash check and retry [1526317032.47][HTST][INF] starting host test process... [1526317033.09][CONN][INF] starting connection process... [1526317033.09][CONN][INF] notify event queue about extra 60 sec timeout for serial port pooling [1526317033.09][CONN][INF] initializing serial port listener... [1526317033.09][SERI][INF] serial(port=COM14, baudrate=9600, read_timeout=0.01, write_timeout=5) [1526317033.09][SERI][INF] reset device using 'default' plugin... [1526317033.11][HTST][INF] setting timeout to: 60 sec [1526317033.36][SERI][INF] waiting 1.00 sec after reset [1526317034.36][SERI][INF] wait for it... [1526317034.36][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed [1526317034.36][CONN][INF] sending up to 2 __sync packets (specified with --sync=2) [1526317034.36][CONN][INF] sending preamble '0ea1005b-0e2c-4dec-a15f-c947b6556639' [1526317034.36][SERI][TXD] {{__sync;0ea1005b-0e2c-4dec-a15f-c947b6556639}} [1526317034.49][CONN][RXD] mbedmbedmbedmbedmbedmbedmbedmbed [1526317034.54][CONN][INF] found SYNC in stream: {{__sync;0ea1005b-0e2c-4dec-a15f-c947b6556639}} it is #0 sent, queued... [1526317034.56][HTST][INF] sync KV found, uuid=0ea1005b-0e2c-4dec-a15f-c947b6556639, timestamp=1526317034.540000 [1526317034.57][CONN][INF] found KV pair in stream: {{__version;1.3.0}}, queued... [[1526317034.59][HTST][INF] DUT greentea-client version: 1.3.0 1526317034.59][CONN][INF] found KV pair in stream: {{__timeout;120}}, queued... [1526317034.59][HTST][INF] setting timeout to: 120 sec [1526317034.62][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued... [1526317034.62][HTST][INF] host test class: '<class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>' [1526317034.62][HTST][INF] host test setup() call... [1526317034.62][HTST][INF] CALLBACKs updated [1526317034.62][HTST][INF] host test detected: default_auto [1526317078.79][CONN][RXD] :46::FAIL: Expected 0 Was -3004 [1526317078.86][CONN][RXD] >>> failure with reason 'Assertion Failed' during 'Test Setup Handler' [1526317078.90][CONN][INF] found KV pair in stream: {{max_heap_usage;0}}, queued... [[1526317078.91][CONN][INF] found KV pair in stream: {{reserved_heap;0}}, queued... 1526317078.91][HTST][ERR] orphan event in main phase: {{max_heap_usage;0}}, timestamp=1526317078.897000 [1526317078.91][HTST][ERR] orphan event in main phase: {{reserved_heap;0}}, timestamp=1526317078.913000 [1526317078.93][CONN][INF] found KV pair in stream: {{end;failure}}, queued... [1526317078.94][CONN][INF] found KV pair in stream: {{__exit;0}}, queued... [1526317078.95][HTST][INF] __exit(0) [1526317078.95][HTST][INF] __notify_complete(False) [1526317078.95][HTST][INF] __exit_event_queue received [1526317078.95][HTST][INF] test suite run finished after 44.36 sec... [1526317078.96][CONN][INF] received special even '__host_test_finished' value='True', finishing [1526317078.98][HTST][INF] CONN exited with code: 0 [1526317078.98][HTST][INF] Some events in queue [1526317078.98][HTST][INF] stopped consuming events [1526317078.98][HTST][INF] host test result() call skipped, received: False [1526317078.98][HTST][INF] calling blocking teardown() [1526317078.98][HTST][INF] teardown() finished [1526317078.98][HTST][INF] {{result;failure}}

I'll add the required return value checks. ~~I wouldn't prefer not to because it will just add clutter on the test cases and assertions would reveal problems arising during setups.~~ Part of the test is to check that the desired behavior is the result of correct setup.

SeppoTakalo · 2018-05-14T13:27:03Z

TESTS/netsocket/udp/udpsocket_echotest.cpp

Are you ignoring the sendto() failures?

Yes, I am because buffer comparison will reveal if data was not transmitted and received - retries are allowed. Checking for errors wouldn't make any difference.

cmonr · 2018-05-14T16:29:42Z

@VeijoPesonen Was there a particular reason why this PR was give a new category?

@SeppoTakalo @kjbracey-arm Is the intention to get this in for 5.9?

VeijoPesonen · 2018-05-14T16:54:58Z

@cmonr My changes are not fixes, refactoring of existing code, breaking change to existing code, new features and do not consider new targets. I decided to add a new category to evoke this discussion. I would say there is need to have its own category for test cases. Unless new test case is to be treated like a new feature.

VeijoPesonen · 2018-05-16T09:01:49Z

Please let me know when it's fine to squash those fixup commits.

SeppoTakalo · 2018-05-16T10:59:01Z

It is OK to squash when ever you feel so. And please rebase on top of master.
Start checking of the build failures.

For example:

Build failures:

  * K64F::ARM::TESTS-NETWORK-WIFI

        Building project wifi (K64F, ARM)

        Scan: ARM

        Scan: wifi

        Scan: FEATURE_STORAGE

        Scan: FEATURE_LWIP

        Compile [  5.3%]: get_interface.cpp

        [Error] get_interface.cpp@34,0:  #513: a value of type "EthernetInterface *" cannot be assigned to an entity of type "WiFiInterface *"

Once build failures are fixed, ask Kevin to start morph testing for this PR. We need to see which boards start failing.

SeppoTakalo · 2018-05-16T11:46:33Z

Please remove also following tests from the tree:

netsocket/ip_parsing <- This is unittesting the SocketAddress class. Not useful for testing HW.
netsocket/socket_sigio <- This one is obsoleted by non-blocking test-cases
netsocket/tcp_hello_world <- This is obsoleted by other TCP tests

SeppoTakalo · 2018-05-16T11:52:03Z

Can we also refactor testcases to follow naming from the test plan?

So instead of:

void test_udpsocket_echotest();
...
Case cases[] = {
        Case("Echo", test_udpsocket_echotest),

We would use:

void UDPSOCKET_ECHOTEST();
...
Case cases[] = {
        Case("UDPSOCKET_ECHOTEST", UDPSOCKET_ECHOTEST),

That would allow us to publish the test plan without modifications.

VeijoPesonen · 2018-05-16T12:14:41Z

I'll switch the naming convention. I don't see a reason to suppress compiler error for TESTS-NETWORK-WIFI test cases when Ethernet is in use. Maybe it would make more sense to use preprocessor directive explicitly to report an error, or not...

SeppoTakalo · 2018-05-16T12:37:00Z

@VeijoPesonen Also, you need to do the separation of testcase.

Currently in both TCP and UDP you have

#ifndef MBED_EXTENDED_TESTS
#error [NOT_SUPPORTED] Pressure tests are not supported by default
#endif

Which is not what we are aiming for. So remove those.
If device is able to provide network interface, it should be able to run at least some of the testcases.

So do the splitting in the Case case[] table.
For example in tcp/main.cpp

    Case("Echo", test_tcpsocket_echotest),
    Case("Echo non-block", test_tcpsocket_echotest_nonblock),
    Case("Reuse a socket", test_tcpsocket_open_close_repeat),
    Case("Open at least 4 sockets", test_tcpsocket_open_limit),
    Case("Parallel socket thread safety", test_tcpsocket_thread_per_socket_safety),
    Case("Send repeatedly", test_tcpsocket_send_repeat),
#ifdef MBED_EXTENDED_TESTS
    Case("Echo burst", test_tcpsocket_echotest_burst),
    Case("Echo burst non-block", test_tcpsocket_echotest_burst_nonblock),
    Case("Invalid endpoint rejected", test_tcpsocket_connect_invalid),
    Case("Receive 100k from CHARGEN service", test_tcpsocket_recv_100k),
    Case("Receive 100k from CHARGEN service non-block", test_tcpsocket_recv_100k_nonblock),
    Case("Receive in given time", test_tcpsocket_recv_timeout),
    Case("Sending shall not take too long", test_tcpsocket_send_timeout),
    Case("Endpoint initiated close", test_tcpsocket_endpoint_close),
#endif
};

Modify the splitting so that in PR we would run only fast smoke tests that current boards are able to pass. Then later on, in nightly test those extented tests are also enable.

SeppoTakalo · 2018-05-16T12:42:59Z

TESTS/netsocket/tcp/tcpsocket_echotest.cpp

Use same packet ranges as our python tests.
1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,90,100,200,300,400,500,600,700,800,900,1000,1100,1200

So from 1 to 10. Then 10 to 100 in steps of 10. Then from 100 to 1200 in steps of 100.

ok, to be fixed

SeppoTakalo · 2018-05-16T12:46:40Z

TESTS/netsocket/tcp/tcpsocket_echotest.cpp

NSAPI_ERROR_WOULD_BLOCK should not be treated differently in blocking case. All value less than zero are errors in this case.

ok, have to take a look

SeppoTakalo · 2018-05-16T12:48:19Z

TESTS/netsocket/tcp/tcpsocket_echotest.cpp

Just a suggestion, you already have generate_RFC_864_pattern() that can be used for this purpose.

Good idea ~~and to be taken into use~~

SeppoTakalo · 2018-05-16T13:40:41Z

TESTS/netsocket/udp/udpsocket_echotest.cpp

To be sure, clear the rx_buffer, so if we re-send same sized packet with same data, it will be shown.
`memset(rx_buffer, 0, BUFF_SIZE);

SeppoTakalo · 2018-05-16T13:41:28Z

TESTS/netsocket/udp/udpsocket_echotest.cpp

As in TCP case, use same ranges.
(Perhaps update the testplan as well)

SeppoTakalo · 2018-05-16T13:51:15Z

TESTS/netsocket/udp/udpsocket_recv_timeout.cpp

This is weird way of saying "More than 100, but less than 200"
Why not TEST_ASSERT_INT_WITHIN(50, 150, ...)

SeppoTakalo · 2018-05-16T14:05:34Z

Any ideas why WIFI tests now fail to build?
Did we somehow enable the build with K64F?

SeppoTakalo · 2018-05-16T14:11:24Z

@VeijoPesonen
Ah.. looks like in master it does not fail, so you have cherry-picked some changes from Juha.

We cannot merge those in yet, selectively building WiFi tests should be resolved somewhere else.

SeppoTakalo · 2018-05-16T14:13:54Z

@kjbracey-arm @0xc0170
Can you guys help, how can I tell from pre-compilation phase that

#if <some magic here>
#error [NOT_SUPPORTED] This is not a WiFi device
#endif

Other than that, test configuration now picks up interfaces by

interface = MBED_CONF_APP_OBJECT_CONSTRUCTION;

kjbracey · 2018-05-16T14:20:37Z

Prior to "default network interface" stuff, you have to assume it's not a WiFi device, unless someone has specifically told you how to find a WiFi interface. Which would be the 'MBED_CONF_APP_OBJECT_CONSTRUCTION' etc the tests use?

After default network interface, I'm afraid there's still no compile-time method, but WiFiInterface::get_default_instance() would return NULL or not at run-time.

For Ethernet, DEVICE_EMAC basically does the job.

SeppoTakalo · 2018-05-16T15:02:18Z

@VeijoPesonen You need then drop the changes to WiFi tests get_interface() function. That is not universal, and is breaking the build.

SeppoTakalo · 2018-05-16T15:02:57Z

Other than those that I have now commented, all looks good.
So after those are changed, I'm OK with these changes.

udpsocket_sendto_invalid

tcpsocket_send_repeat

udpsocket_sendto_repeat

udpsocket_echotest udpsocket_echotest_nonblock

udpsocket_echotest.cpp udpsocket_echotest_burst.cpp

From lowercase to uppercase and drops 'test_'-prefix. Test case descriptions do also match to TC names now. Additionally all the TCs are not behing MBED_EXTENDED_TESTS anymore.

SeppoTakalo · 2018-05-25T07:21:49Z

Rebased on top of master.

Emac is now in "needs Preceeding PR" label should be removed.

@0xc0170 or @kjbracey-arm please start tests for this.

kjbracey · 2018-05-25T09:48:31Z

It would be nice if this could go to 5.9.0, but I think CI load may prevent it. Labelling as 5.9.1 for now.

cmonr · 2018-05-25T14:23:58Z

Going to attempt to squeeze this in.

/morph build

mbed-ci · 2018-05-25T15:58:01Z

Build : SUCCESS

Build number : 2153
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6665/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-05-25T17:20:42Z

Test : SUCCESS

Build number : 1945
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6665/1945

mbed-ci · 2018-05-25T19:41:46Z

Exporter Build : SUCCESS

Build number : 1782
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6665/

VeijoPesonen commented Apr 18, 2018

View reviewed changes

TESTS/netsocket/tcp_chargen_recv100k/main.cpp Outdated

Copy link

Contributor Author

VeijoPesonen Apr 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation provided by @SeppoTakalo

0xc0170 added do not merge needs: work labels Apr 18, 2018

VeijoPesonen commented Apr 20, 2018

View reviewed changes

SeppoTakalo reviewed Apr 20, 2018

View reviewed changes

kjbracey previously approved these changes May 11, 2018

View reviewed changes

kjbracey mentioned this pull request May 11, 2018

lwIP: Enable TCP out-of-order processing #6882

Merged

SeppoTakalo suggested changes May 14, 2018

View reviewed changes

SeppoTakalo reviewed May 14, 2018

View reviewed changes

VeijoPesonen dismissed kjbracey’s stale review via 021f6cd May 14, 2018 16:32

SeppoTakalo reviewed May 16, 2018

View reviewed changes

mikaleppanen mentioned this pull request May 17, 2018

Greentea UDP/TCP packet pressure test cases are failing on platforms with large memory #6650

Closed

Veijo Pesonen and others added 11 commits May 25, 2018 09:39

Removes obsolete TCP and UDP Greentea test cases

420459b

Adds UDP Greentea test case

096ed1e

udpsocket_sendto_invalid

Adds TCP test case

1933574

tcpsocket_send_repeat

Adds UDP Greentea test case

a2f5ffa

udpsocket_sendto_repeat

Adds UDP Greentea test cases

1fef160

udpsocket_echotest udpsocket_echotest_nonblock

Greentea TCP test cases - check all socket calls

4fa57e3

Fixes Greentea UDP test cases

1b04172

udpsocket_echotest.cpp udpsocket_echotest_burst.cpp

Switches Greentea netsocket test case naming convention

21ec972

From lowercase to uppercase and drops 'test_'-prefix. Test case descriptions do also match to TC names now. Additionally all the TCs are not behing MBED_EXTENDED_TESTS anymore.

Fix Greentea WiFi test build issues

53eb36e

Fixes findings from Greentea netsocket UDP test cases

3e11a3e

Fixes findings from Greentea netsocket TCP test cases

491a7ea

SeppoTakalo dismissed stale reviews from kjbracey and themself via 491a7ea May 25, 2018 07:20

0xc0170 added needs: CI and removed needs: preceding PR labels May 25, 2018

kjbracey added the release-version: 5.9.1 label May 25, 2018

cmonr added release-version: 5.9.0-rc1 risk: A and removed release-version: 5.9.1 labels May 25, 2018

cmonr approved these changes May 25, 2018

View reviewed changes

cmonr merged commit 45dc00d into ARMmbed:master May 25, 2018

cmonr removed risk: A needs: CI labels May 25, 2018

VeijoPesonen deleted the greentea_netsocket_more_tests branch May 26, 2018 17:46

Greentea netsocket more tests #6665

Greentea netsocket more tests #6665

Uh oh!

Conversation

VeijoPesonen commented Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Uh oh!

VeijoPesonen commented Apr 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeijoPesonen commented May 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SeppoTakalo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeijoPesonen May 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo May 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeijoPesonen May 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmonr commented May 14, 2018

Uh oh!

VeijoPesonen commented May 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VeijoPesonen commented May 16, 2018

Uh oh!

SeppoTakalo commented May 16, 2018

Uh oh!

SeppoTakalo commented May 16, 2018

Uh oh!

SeppoTakalo commented May 16, 2018

Uh oh!

VeijoPesonen commented May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SeppoTakalo commented May 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeijoPesonen May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VeijoPesonen commented Apr 18, 2018 •

edited

Loading

VeijoPesonen commented Apr 18, 2018 •

edited

Loading

VeijoPesonen commented May 11, 2018 •

edited

Loading

VeijoPesonen May 14, 2018 •

edited

Loading

SeppoTakalo May 14, 2018 •

edited

Loading

VeijoPesonen May 14, 2018 •

edited

Loading

VeijoPesonen commented May 14, 2018 •

edited

Loading

VeijoPesonen commented May 16, 2018 •

edited

Loading

VeijoPesonen May 16, 2018 •

edited

Loading