Add TLSSocket greentea tests. #9283

michalpasztamobica · 2019-01-08T07:44:56Z

Description

TLSSocket tests based on existing TCPSocket greentea tests and TLSSocket icetea tests.
The tests are only passing with "rtos.main-thread-stack-size": 8192 (6144 if we take out the SIMULTANEOUS test).
HTTPS_GET test is commented as it fails, perhaps due to connectivity reasons - I don't want to break the CI.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

@SeppoTakalo
@KariHaapalehto
@VeijoPesonen
@mtomczykmobica

ciarmcom · 2019-01-08T08:00:31Z

@michalpasztamobica, thank you for your changes.
@VeijoPesonen @SeppoTakalo @KariHaapalehto @mtomczykmobica @ARMmbed/mbed-os-ipcore @ARMmbed/mbed-os-test @ARMmbed/mbed-os-maintainers please review.

mtomczykmobica

Looks good.

TESTS/netsocket/tls/main.cpp

SeppoTakalo

Just a minor changes proposed.

Otherwise looks good.

SeppoTakalo · 2019-01-08T11:32:04Z

TESTS/netsocket/tls/tlssocket_echotest_burst.cpp

+        }
+
+        if (bt_left != 0) {
+            drop_bad_packets(*sock, 0);


This can only happen if recv() returns error and if it does, we should not try to read again.

I prefer this drop_bad_packets() to be removed. Just mark the test failed (with error message, if possible).

TESTS/netsocket/tls/main.cpp

SeppoTakalo · 2019-01-08T11:38:18Z

TESTS/netsocket/tls/main.cpp

+{
+    SocketAddress tls_addr;
+
+    get_interface()->gethostbyname(MBED_CONF_APP_ECHO_SERVER_ADDR, &tls_addr);


Probably a minor thing, but now that we use NetworkInterface::get_default_instance() can we refactor all these get_interface() call to NetworkInterface::get_default_instance()->

I feel like get_interface() is also a residue from time when default network interface did not exist

SeppoTakalo · 2019-01-08T11:39:38Z

TESTS/netsocket/tls/main.cpp

+    if (err != NSAPI_ERROR_OK) {
+        printf("Error from sock.connect: %d\n", err);
+        return err;
+    }


This function is copy&paste from previous, only change is the port address.
Can the port address be a parameter to this and just use the same function?

SeppoTakalo · 2019-01-08T11:41:04Z

TESTS/netsocket/tls/main.cpp

+    Case("TLSSOCKET_OPEN_TWICE", TLSSOCKET_OPEN_TWICE),
+    Case("TLSSOCKET_OPEN_LIMIT", TLSSOCKET_OPEN_LIMIT),
+    Case("TLSSOCKET_OPEN_DESTRUCT", TLSSOCKET_OPEN_DESTRUCT),
+//    Case("TLSSOCKET_HTTPS_GET", TLSSOCKET_HTTPS_GET), // Fails!


// Fails! does not sound very good on public example.

Maybe "work-in-progress, unstable" or something like that.

TESTS/netsocket/tls/tlssocket_echotest.cpp

SeppoTakalo · 2019-01-08T12:07:52Z

TESTS/netsocket/tls/tlssocket_echotest.cpp

+                            tls_global::TLS_OS_STACK_SIZE,
+                            stack_mem,
+                            "receiver");
+        TEST_ASSERT_EQUAL(osOK, thread->start(callback(tlssocket_echotest_nonblock_receiver, &pkt_s)));


New thread per packet?

Sounds a bit overkill...

We could push event to global event loop every time we receive sigio(), and that event could the read until WOULD_BLOCK is received.. but if all data is received, it could set some global event flag to indicate that all data is read.

Just to make sure we have a common understanding of what is going on in this test...

Indeed, a thread is created for every new packet size. But there is always just one receiver thread running at a time (tx_sem.wait will block the main thread's for loop until receiver thread exits). The receiver thread already contains a loop, which is controller with SIGIO (as per the previous comment :) ) and returns once all data is received.
So the only change would be to trigger the receiver's loop using a global event loop instead of triggering it using thread creation - is that right? Or is there some further optimisation which I failed to notice?

SeppoTakalo · 2019-01-08T12:10:43Z

TESTS/netsocket/README.md


-$ sudo nmap -sT -p7,9,13,37 echo.mbedcloudtesting.com
+$ sudo nmap -sT -p7,9,13,37,2007,2009,2013 echo.mbedcloudtesting.com


I can see that there was a error on previous documentation.
This was supposed to do UDP scan, and therefore required sudo

Can you change it to do
sudo nmap -sU instead?

SeppoTakalo · 2019-01-08T12:11:41Z

TESTS/netsocket/tls/tlssocket_endpoint_close.cpp

+ * limitations under the License.
+ */
+
+#include "tls_tests.h""


Douple "" at the end of line

TESTS/netsocket/tls/tlssocket_endpoint_close.cpp

TESTS/netsocket/tls/main.cpp

VeijoPesonen · 2019-01-08T13:02:53Z

TESTS/netsocket/tls/tlssocket_bind_unopened.cpp

+
+    TLSSocket *sock = new TLSSocket;
+    if (!sock) {
+        TEST_FAIL();


TEST_FAIL won't abort execution so adding return-statement would be good here.

Good point, but I removed the bind tests entirely.
However, these same comments apply also to tcp tests.

VeijoPesonen · 2019-01-08T13:20:45Z

TESTS/netsocket/tls/tlssocket_no_cert.cpp

+{
+    TLSSocket sock;
+    TEST_ASSERT_EQUAL(NSAPI_ERROR_OK, sock.open(get_interface()));
+    TEST_ASSERT_EQUAL(NSAPI_ERROR_AUTH_FAILURE, sock.connect(MBED_CONF_APP_ECHO_SERVER_ADDR, 2007));


Would be good to take the port number from app config json-file as this is non-standard port

TESTS/netsocket/tls/tlssocket_send_unconnected.cpp

SeppoTakalo · 2019-01-09T11:23:51Z

@ARMmbed/mbed-os-maintainers @ARMmbed/mbed-os-core Hi guys, do you know that is there any way we could force main stack size to be raised to 8 kB just for these TLSSocket tests?

Currently that is roughly the amount of stack that TLS handshake requires. Therefore we need to resolve the stack issue before merging?

For example, can I manually create a thread that has 8kB of stack, and launch Greentea from there?

SeppoTakalo · 2019-01-09T11:30:02Z

@michalpasztamobica Can you try for example following:


static int run_test(voi) {
    return !Harness::run(specification);
}

static char stack_mem[8192];
int main()
{
    Thread *th = new Thread(osPriorityNormal,
                        8192,
                         stack_mem,
                         "tls_gt_thread");
    th->start(run_test);
    return th->join();
}

If that works, we could perhaps use that to force bigger stack for TLS thread.

michalpasztamobica · 2019-01-09T12:01:22Z

@SeppoTakalo , your idea worked. Thanks!
I also added the echo-server-port-tls and echo-server-discard-port-tls definitions in tools/test_configs/*.json files.
If this is approved we can merge safely.

SeppoTakalo

Now looks OK

OPpuolitaival

Looks good

michalpasztamobica · 2019-01-11T12:13:53Z

@SeppoTakalo , @VeijoPesonen , I updated the README.md file with the TLSSocket test documentation and also amended some mistakes I spotted. I did not open a new PR, as the changes were actually relevant to this PR and it still wasn't merged.
Please, review just that one file once more.

michalpasztamobica · 2019-01-22T07:06:54Z

Fixed the missing whitespace which Travis pointed out.

TESTS/netsocket/tls/main.cpp

TLSSocket tests based on existing TCPSocket greentea tests and TLSSocket icetea tests.

michalpasztamobica · 2019-01-22T11:48:12Z

Travis was giving some odd errors, so I rebased to latest master to check I am missing some CI fix...

0xc0170 · 2019-01-22T12:42:23Z

CI started

mbed-ci · 2019-01-22T15:21:20Z

Test run: FAILED

Summary: 2 of 11 test jobs failed
Build number : 5
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_dynamic-memory-usage
jenkins-ci/mbed-os-ci_greentea-test

0xc0170 · 2019-01-23T13:54:25Z

@michalpasztamobica test failures are related

michalpasztamobica · 2019-01-23T14:39:41Z

The reason is a known issue in IAR: #8306
I admit, I did not test locally with all compilers. IAR compiler doesn't take the configured heap size into consideration and we have seen in icetea tests that opening two TLSSockets consumes too much memory to fit into the IAR's predefined area size.

@SeppoTakalo , in icetea we left the test failing in order not to hide the bug, but I think for greentea we have to exclude the test, otherwise we will break the public CI. Do you agree?

I wrapped the TLSSOCKET_SIMULTANEOUS test with a preprocessor guard:

#ifndef __IAR_SYSTEMS_ICC__
    Case("TLSSOCKET_SIMULTANEOUS", TLSSOCKET_SIMULTANEOUS)
#endif

I checked that it doesn't hurt other compilers and IAR simply runs one test less.

The whole README.md had to be updated to match the internal Confluence documentation, which can now be locked. In the process I also updated any spotted mistakes in tests documentation, removed the obsolete TCPServer tests documentation and added a test which was missing from UDPSocket's main.

michalpasztamobica · 2019-01-29T12:04:14Z

@0xc0170 , can we give this another try? Seppo approved my solution and it worked fine locally.

0xc0170 · 2019-01-29T12:42:50Z

Ci started

mbed-ci · 2019-01-29T14:37:10Z

Test run: FAILED

Summary: 1 of 12 test jobs failed
Build number : 6
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_greentea-test

alekla01 · 2019-01-29T14:40:48Z

jenkins-ci/greentea-test restarted

ciarmcom requested review from KariHaapalehto, mtomczykmobica, SeppoTakalo, VeijoPesonen and a team January 8, 2019 08:00

ciarmcom added the needs: review label Jan 8, 2019

michalpasztamobica force-pushed the tlssocket_greentea branch from da9eb01 to fdec0de Compare January 8, 2019 09:00

mtomczykmobica approved these changes Jan 8, 2019

View reviewed changes

SeppoTakalo reviewed Jan 8, 2019

View reviewed changes

TESTS/netsocket/tls/main.cpp Show resolved Hide resolved

michalpasztamobica force-pushed the tlssocket_greentea branch from fdec0de to 3b654aa Compare January 8, 2019 12:09

SeppoTakalo suggested changes Jan 8, 2019

View reviewed changes

0xc0170 added needs: work and removed needs: review labels Jan 8, 2019

VeijoPesonen reviewed Jan 8, 2019

View reviewed changes

michalpasztamobica force-pushed the tlssocket_greentea branch from 3b654aa to cdfc21d Compare January 8, 2019 15:51

cmonr added needs: review and removed needs: work labels Jan 8, 2019

VeijoPesonen approved these changes Jan 9, 2019

View reviewed changes

michalpasztamobica force-pushed the tlssocket_greentea branch from cdfc21d to 8bd443d Compare January 9, 2019 09:41

SeppoTakalo approved these changes Jan 9, 2019

View reviewed changes

michalpasztamobica force-pushed the tlssocket_greentea branch from 8bd443d to 0a0026e Compare January 9, 2019 11:59

SeppoTakalo approved these changes Jan 9, 2019

View reviewed changes

OPpuolitaival approved these changes Jan 9, 2019

View reviewed changes

michalpasztamobica force-pushed the tlssocket_greentea branch from 3a1d276 to 419ee1d Compare January 22, 2019 07:05

SeppoTakalo reviewed Jan 22, 2019

View reviewed changes

TESTS/netsocket/tls/main.cpp Show resolved Hide resolved

michalpasztamobica force-pushed the tlssocket_greentea branch from 419ee1d to 5b829a2 Compare January 22, 2019 08:56

Add TLSSocket greentea tests.

d568e21

TLSSocket tests based on existing TCPSocket greentea tests and TLSSocket icetea tests.

michalpasztamobica force-pushed the tlssocket_greentea branch from 5b829a2 to c2aafcd Compare January 22, 2019 11:47

0xc0170 added needs: CI and removed needs: work labels Jan 22, 2019

cmonr added needs: work and removed needs: CI labels Jan 22, 2019

michalpasztamobica force-pushed the tlssocket_greentea branch from c2aafcd to af65522 Compare January 23, 2019 14:54

SeppoTakalo approved these changes Jan 24, 2019

View reviewed changes

0xc0170 added needs: CI and removed needs: work labels Jan 29, 2019

cmonr added ready for merge release-version: 5.11.4 and removed needs: CI labels Jan 29, 2019

0xc0170 merged commit 7d036b5 into ARMmbed:master Jan 30, 2019

0xc0170 removed the ready for merge label Jan 30, 2019

michalpasztamobica deleted the tlssocket_greentea branch January 30, 2019 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TLSSocket greentea tests. #9283

Add TLSSocket greentea tests. #9283

michalpasztamobica commented Jan 8, 2019 •

edited

ciarmcom commented Jan 8, 2019

mtomczykmobica left a comment

SeppoTakalo left a comment

SeppoTakalo Jan 8, 2019

SeppoTakalo Jan 8, 2019

SeppoTakalo Jan 8, 2019

SeppoTakalo Jan 8, 2019

SeppoTakalo Jan 8, 2019

michalpasztamobica Jan 8, 2019

SeppoTakalo Jan 8, 2019

SeppoTakalo Jan 8, 2019

VeijoPesonen Jan 8, 2019

michalpasztamobica Jan 8, 2019

VeijoPesonen Jan 8, 2019

SeppoTakalo commented Jan 9, 2019

SeppoTakalo commented Jan 9, 2019

michalpasztamobica commented Jan 9, 2019

SeppoTakalo left a comment

OPpuolitaival left a comment

michalpasztamobica commented Jan 11, 2019

michalpasztamobica commented Jan 22, 2019

michalpasztamobica commented Jan 22, 2019

0xc0170 commented Jan 22, 2019

mbed-ci commented Jan 22, 2019

0xc0170 commented Jan 23, 2019

michalpasztamobica commented Jan 23, 2019

michalpasztamobica commented Jan 29, 2019

0xc0170 commented Jan 29, 2019

mbed-ci commented Jan 29, 2019

alekla01 commented Jan 29, 2019


		$ sudo nmap -sT -p7,9,13,37 echo.mbedcloudtesting.com
		$ sudo nmap -sT -p7,9,13,37,2007,2009,2013 echo.mbedcloudtesting.com

Add TLSSocket greentea tests. #9283

Add TLSSocket greentea tests. #9283

Conversation

michalpasztamobica commented Jan 8, 2019 • edited

Description

Pull request type

Reviewers

ciarmcom commented Jan 8, 2019

mtomczykmobica left a comment

Choose a reason for hiding this comment

SeppoTakalo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SeppoTakalo commented Jan 9, 2019

SeppoTakalo commented Jan 9, 2019

michalpasztamobica commented Jan 9, 2019

SeppoTakalo left a comment

Choose a reason for hiding this comment

OPpuolitaival left a comment

Choose a reason for hiding this comment

michalpasztamobica commented Jan 11, 2019

michalpasztamobica commented Jan 22, 2019

michalpasztamobica commented Jan 22, 2019

0xc0170 commented Jan 22, 2019

mbed-ci commented Jan 22, 2019

0xc0170 commented Jan 23, 2019

michalpasztamobica commented Jan 23, 2019

michalpasztamobica commented Jan 29, 2019

0xc0170 commented Jan 29, 2019

mbed-ci commented Jan 29, 2019

alekla01 commented Jan 29, 2019

michalpasztamobica commented Jan 8, 2019 •

edited