ADR 2: How to show test results in the html report

[camilamaia]

Architecture Decision Review - ADR

• How are we going to show the tests in the html report
• Which color should we use?
• How are we going to show each test case?
• How are we going to show if a test passed?
• How are we going to show if a test failed?
• Should we use HTTP methods with color? Yes or No?

This discussion started here

Related ADR: #164

[camilamaia]

I did some tests, they are all ugly 😂 But maybe we can get some ideas of what works and what doesn't work:

https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version1.html
https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version2.html
https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version3.html
https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version4.html
https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version5.html
https://github.com/scanapi/scanapi/blob/adr2/testing_reports/version6.html

I can't attach HTML files, that is why a created a temporary branch just to load this tests.

@barbosa @djalmaaraujo @gillianomenezes any thought?

[camilamaia]

Basically we have three different test status we need to show

1. The overview. If all tests passed. Or, if any test failed

If all tests passed, the test summary should, somehow, show that everything is ok and there is no need to check anything else. It is all green.

2. The Endpoint. We need to show if all tests for that specific endpoint passed (or, if any failed)

Here it is important to provide somehow an easy and fast way for the user to find which endpoint has a failing test. Let's say we have a report with dozens of endpoints. We don't want the user opening all the endpoints trying to find which one failed.

Something written would be a solution. Let's say there is something like this

The user can easily use ctrl + f and search for [failed] to find the endpoints with failing tests.

3. The test itself.

---

We are using different colors for each HTTP method. We want to make clear that they are different.

But, as @barbosa mentioned here, "green/yellow/red colors may give the impression of success/failure for each of the requests being executed."

So how we can show this difference without making this impression of success/failure/skipped? Which colors should we use? I tried some pastel colors, but I did not like the ones I picked

Current we only support five HTTP methods, but there are 9 in total.

---

In summary, it would be nice to discuss:

1. What is the best way to show the overview status?
2. What is the best way to show the endpoint status?
3. What is the best way to show the test status?
4. What is the best way to differentiate the HTTP methods?

ps: please, see the comment above with some suggestions

[djalmaaraujo]

@camilamaia @barbosa
Hey, that's a good discussion. Here's what I would prefer to see in this case:

• I would suggest red/green for the main status of the runner (PASSED/FAILED)
• One-color only for the HTTP methods. I am not sure if the different colors make sense for the report, to be honest. So maybe we could use BLUE (maybe) since blue is not a TRAFFIC color (red/green/yellow)
• For HTTP status 200/404/etc, I would follow the standard industry: 200/2** goes GREEN, anything bad 400/500, etc goes red.

I can provide the palette for this if you agree.

[camilamaia]

Versions with images:

Version 1

Version 2

Version 3

Version 4

Version 5

Version 6

[gillianomenezes]

@camilamaia @barbosa
Hey, that's a good discussion. Here's what I would prefer to see in this case:

• I would suggest red/green for the main status of the runner (PASSED/FAILED)
• One-color only for the HTTP methods. I am not sure if the different colors make sense for the report, to be honest. So maybe we could use BLUE (maybe) since blue is not a TRAFFIC color (red/green/yellow)
• For HTTP status 200/404/etc, I would follow the standard industry: 200/2** goes GREEN, anything bad 400/500, etc goes red.

I can provide the palette for this if you agree.

I agree with this approach. IMHO the HTTP methods in black bold seem better. This is why I vote in the first option, @camilamaia.

[barbosa]

@camilamaia @barbosa
Hey, that's a good discussion. Here's what I would prefer to see in this case:

• I would suggest red/green for the main status of the runner (PASSED/FAILED)
• One-color only for the HTTP methods. I am not sure if the different colors make sense for the report, to be honest. So maybe we could use BLUE (maybe) since blue is not a TRAFFIC color (red/green/yellow)
• For HTTP status 200/404/etc, I would follow the standard industry: 200/2** goes GREEN, anything bad 400/500, etc goes red.

I can provide the palette for this if you agree.

Yeah, I agree with this except for the last item. I believe color variation should be used for one thing and one thing only: indicate success or failure of test results.

I would keep both HTTP methods and status codes in black.

Mixing red/green in more than one context would cause confusion, I think.

[camilamaia]

@djalmaaraujo @gillianomenezes and @barbosa, thank you very much for your feedbacks and opinions. Based on then, I created a new version. What do you think?

Failed Closed

Failed Opened

Passed Closed

Passed Opened

[camilamaia]

@djalmaaraujo give me some help! Look how it is now:

[camilamaia]

@gillianomenezes and @barbosa can I close this ticket? Do you have any other opinions? Thanks!

[barbosa]

it seems the icons are not vertically centered but we can fix this later

[camilamaia]

it seems the icons are not vertically centered but we can fix this later

Great catch 🔍 👀
@barbosa I created a new issue for that: #190

[camilamaia]

@GitVitor check this issue here! It might help you with the new changes on the report 🎨

[GitVitor]

@camilamaia thanks, i had never seen this ADR before;

Just to document, i had some thoughts about how we could improve our UI;

Below a list with some changes that i have work:

1. Change P(assed) or F(ailed) icon;
2. Change font-family to use Gotham;
3. Use diferent colors to represent each HTTP Method (Image 1);
4. Make request content, request response and request test visually grouped (Image 2);
5. Change Green to use the following hex #48c78e;
6. Change Red to use the following hex #f14668;

Image 1

Image 2

[GitVitor]

Based on comments that was sent here, maybe we could apply following changes:

1. Change P(assed) or F(ailed) icon;
2. Change font-family to use Gotham;
   Use diferent colors to represent each HTTP Method;
   Make request content, request response and request test visually grouped;
3. Change Green to use the following hex #48c78e;
4. Change Red to use the following hex #f14668;

[camilamaia]

@GitVitor I love the new ideas! The only concern I had was the colors, as you mentioned before. But I would keep the number 4 Make request content, request response and request test visually grouped (Image 2); I think it is good to see them together. Any reason to not implement it?

Thanks again!

[GitVitor]

Oh number 4 make sense and could be implemented, i make some mistake on remove it 😄