[tutorial] [integration testing] please explain the choice of the "test executables" approach

[dodikk]

@phil-opp , thank you for another great article.

Could you please explain the reason of choosing the "Additional Test Executables" approach over "Read the test function name from the serial port" (and dispatch it to actual Rust functions with match operator or some other means. Basically, that would be a simple interpreter over a serial port).

As far as I remember, UART allows a two-way communication... (and you've also mentioned that fact in the beginning of an article).

P.S. explaining how to build a library and a "hybrid project" clearly adds some education value to the article. Still, please consider highlighting the pros and cons of different test framework implementation approaches. So far It's a bit unclear for the beginners (like me) "what the best practice is".

[dodikk]

A separate ticket for my comment #443 (comment)

[phil-opp]

So far It's a bit unclear for the beginners (like me) "what the best practice is".

Every approach has its advantages and drawbacks. The goal of the post was not to create the perfect integration test framework. It certainly isn't perfect and has a lot of drawbacks. Instead I tried to create something simple that can be easily implemented in the context of a blog post.

I chose to use additional test executables for integration tests to make the tests as separated as possible. Moving the test runner inside the kernel and reading the test names that should be invoked via the serial port does not allow panics or destructive operations inside individual tests (or makes them much more complicated).

Imagine a test that overflows the stack and causes a double fault exception (like in the “Double Faults” post): We can't recover from a double fault, so we can't return back to the test runner because the system is no longer in a runnable state.

[dodikk]

Thank you. That's a good point I have missed since I've been biased by "client-server" and "mobile userspace" (ios/android) testing approaches.

I did not mean to advocate the "interpreter" approach. Neither did I mean to criticise your approach (since I actually liked it a lot). Instead, I just was curious about your reasoning behind that decision.

I guess, explaining this explicitly in the article might be helpful for other beginners (developers with little embedded development or driver development background). What do you think about adding the last paragraph (at least) of your reply to the "Integration Testing" article?

Imagine a test that overflows the stack 
and causes a double fault exception 
(like in the “Double Faults” post): 

We can't recover from a double fault, 
so we can't return back to the test runner 
because the system is no longer in a runnable state.

[dodikk]

Moving the test runner inside the kernel and reading the test names that should be invoked via the serial port does not allow panics or destructive operations inside individual tests

Actually, you have a bit misunderstood my idea. Please take a look at a pseudo code :

// desktop runner

foreach singleTestName in testNames
{
      runQemu();
      serialPort.writeBlocking(singleTestName);
      testResult = serialPort.readBlocking();
}

// embedded

// for integration testing
// one entry point for all cases
// dispatches and runs one test at a time
//
fn _start()
{
      testName = serialPort.readBlocking();
      
      match testName
      {
            "test_something" => {  // run the test case function, write the result to a serial port  }
 
            // a bit more "matching"

            _ => {  // write fail to serial port }
      }
}

We still boot and run each test at a time.
Still, your approach gives more flexibility in terms of redefining panic and interrupt handlers for each test case. And it is way more simple. Now I see that.