docs: augmenting the test plan with X11 clients #3397

mattkae · 2024-05-23T18:04:46Z

What's new?

Added manual tests for a number of 2.16 regressions (see the release page for more info)

Tickets Created

testing: write a test to validate that custom attributes can flow through the display configuration ubuntu-frame#184
feature: write a robot test to ensure that JetBrains' tools are interactable mir-ci#125
feature: write a robot test to make sure that mattermost can be resized mir-ci#124
feature: testing that waking up a sleeping session works mir-ci#126
feature: test cursor movement speed mir-ci#127
feature: test that miral-app can be quit over VNC mir-ci#128
feature: validate that changing the scale of an output via the display configuration works properly mir-ci#129
feature: validate that different rotations result in the correct output mir-ci#130
feature: write a test for Mir's interactions with X11 apps mir-ci#131

codecov · 2024-05-23T18:32:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.27%. Comparing base (fdeafdf) to head (33dcdea).
Report is 444 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3397      +/-   ##
==========================================
- Coverage   77.27%   77.27%   -0.01%     
==========================================
  Files        1075     1075              
  Lines       68401    68401              
==========================================
- Hits        52857    52855       -2     
- Misses      15544    15546       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AlanGriffiths

I have a problem with this approach: I don't think it is scalable to add manual testing for every bug we fix.

Only if we cannot automate a test and there is a likelyhood of a future regression should we consider adding manual testing

AlanGriffiths · 2024-05-24T10:11:20Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

@@ -108,6 +108,44 @@ For each empty box in the matrix above, ensure that the following applications c
   gnome-terminal
   ```

+5. Test that `X11` apps can be started and can receive input when `XWayland` is supported:


I think using a specific compositor is appropriate here: e.g. the default miral-app

AlanGriffiths · 2024-05-24T10:20:28Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+6. (X11) Test that `mattermost` can be opened and resized. This was first encountered in
+   [#2509](https://github.com/canonical/mir/issues/2509). To test:


I think this was a longstanding bug, not a 2.16 regression.

I don't doubt it needs testing, but this sort of arithmetic error should probably have a unit test.

canonical/mir-ci#124 I think this would be good in mir-ci since this is application-specific

AlanGriffiths · 2024-05-24T10:25:02Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+7. (X11) You will need a CLion account for this one. Test that `clion` menus can be interacted
+   with. This was first encountered in [#2326](https://github.com/canonical/mir/issues/2326),
+   [#3107](https://github.com/canonical/mir/issues/3107),
+   [#3024](https://github.com/canonical/mir/issues/3024). 


I think this was a longstanding bug, not a 2.16 regression. Also, this should be reproducible on any of the free, or free trial, JetBrains tools

canonical/mir-ci#125 I think that this would be good in mir-ci as wel since its app specific

AlanGriffiths · 2024-05-24T10:28:07Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+    2. Set orientation of your outputs in `$FILENAME` [normal, left, right, inverted]
+    3. Confirm that the output has the correctly rotated content


This is easier to test with

miral-app -demo-server --screen-rotation

When you can "Rotate screen on Ctrl-Alt-"

AlanGriffiths · 2024-05-24T10:29:55Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

The only compositor that supports snap-name is Frame. And that gets its display configuration from the display snap configuration option.

canonical/ubuntu-frame#184 This should be in the frame repo

AlanGriffiths · 2024-05-24T10:34:08Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+        gvncviewer localhost
+        ```
+    3. Enter `Ctrl + Alt + Backspace`


Suggested change

3. Enter `Ctrl + Alt + Backspace`

3. "Send Key" `Ctrl + Alt + Backspace`

AlanGriffiths · 2024-05-24T10:34:39Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

This should probably be tested via test automation

canonical/mir-ci#128

Saviq · 2024-05-24T13:24:56Z

I have a problem with this approach: I don't think it is scalable to add manual testing for every bug we fix.

Only if we cannot automate a test and there is a likelyhood of a future regression should we consider adding manual testing

The idea is to record the scenarios we need to extend our testing with, and reduce the test plan as we bridge those gaps. Maybe it's a separate section, or document — things we know we can, and want to, automate? Or maybe they're just issues here, or on mir-ci?

AlanGriffiths · 2024-05-24T15:10:21Z

The idea is to record the scenarios we need to extend our testing with, and reduce the test plan as we bridge those gaps. Maybe it's a separate section, or document — things we know we can, and want to, automate? Or maybe they're just issues here, or on mir-ci?

I agree we should track those we don't (yet) have tests for. I don't agree they belong in the manual Mir test plan.

mattkae · 2024-05-24T19:08:23Z

@Saviq and @AlanGriffiths: I created a number of tickets on mir-ci and one on ubuntu-frame (although maybe that belongs in mir-ci too). I kept out some things that seem to have no feasible automation at the moment, but perhaps they can have something too...

RAOF · 2024-05-27T07:30:10Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+  encountered in [#3238](https://github.com/canonical/mir/issues/3238). To test:
+    1. Sleep your session
+    2. Wake your session up
+    3. Confirm that the compositor is still running and that the contents on the
+       screen(s) match what should be there
+
+    This would be automated by https://github.com/canonical/mir-ci/issues/126.


This seems to be an insufficient description for the test? The bug under test specifically required that you have a clone-mode multi-monitor setup to trigger.

General sleep/wake testing would be valuable, but that's not tied to this specific bug.

Oops right! I will add that to the test description

RAOF · 2024-05-27T07:34:02Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+
+- Confirm that the keyboard works after you have disconnected and reconnected it. This
+  issue was first encountere in [#3149](https://github.com/canonical/mir/issues/3149).


Suggested change

issue was first encountere in [#3149](https://github.com/canonical/mir/issues/3149).

issue was first encountered in [#3149](https://github.com/canonical/mir/issues/3149).

RAOF · 2024-05-27T07:35:31Z

doc/sphinx/how-to/how-to-test-mir-for-a-release.md

+  [#3205](https://github.com/canonical/mir/issues/3205). To test:
+    1. Move the cursor through many different surfaces, including Wayland and X11
+       windows
+
+    This would be automated by https://github.com/canonical/mir-ci/issues/127.


Did you mean to link a different bug here? That bug didn't have anything to do with the cursor moving at a different speed over different surfaces? It was a Mir performance regression, where we did a big CPU copy each frame.

True, the "multiple surfaces" bit was a hallucination on my part

RAOF

Yes, those are reasonable additions

AlanGriffiths

Sure

…6_regressions

mattkae requested a review from a team as a code owner May 23, 2024 18:04

mattkae force-pushed the docs/2.16_regressions branch from 7e3b893 to 6883590 Compare May 23, 2024 18:32

AlanGriffiths requested changes May 24, 2024

View reviewed changes

docs: augmenting the test plan to include 2.16 regressions

1d875a3

mattkae force-pushed the docs/2.16_regressions branch from 6883590 to 1d875a3 Compare May 24, 2024 18:34

moving some tests out to tickets

545674b

mattkae force-pushed the docs/2.16_regressions branch from 68376c5 to 545674b Compare May 24, 2024 19:03

mattkae requested a review from AlanGriffiths May 24, 2024 19:08

RAOF reviewed May 27, 2024

View reviewed changes

pr feedback: 2.16 regression feedback from raof

b35e97d

mattkae requested a review from RAOF May 28, 2024 11:31

Base automatically changed from docs/how_to_test_mir to main May 30, 2024 11:41

mattkae added 2 commits May 30, 2024 08:00

Merge branch 'main' into docs/2.16_regressions

faff925

minor: revert accidental indent

cfbe72f

RAOF approved these changes Jun 4, 2024

View reviewed changes

AlanGriffiths approved these changes Jun 4, 2024

View reviewed changes

AlanGriffiths changed the title ~~docs: augmenting the test plan to include 2.16 regressions~~ docs: augmenting the test plan with X11 clients Jun 4, 2024

AlanGriffiths enabled auto-merge June 4, 2024 08:38

AlanGriffiths disabled auto-merge June 6, 2024 10:02

Merge remote-tracking branch 'refs/remotes/origin/main' into docs/2.1…

33dcdea

…6_regressions

AlanGriffiths enabled auto-merge June 6, 2024 10:05

AlanGriffiths added this pull request to the merge queue Jun 6, 2024

Merged via the queue into main with commit b25b49d Jun 6, 2024
23 of 25 checks passed

AlanGriffiths deleted the docs/2.16_regressions branch June 6, 2024 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: augmenting the test plan with X11 clients #3397

docs: augmenting the test plan with X11 clients #3397

mattkae commented May 23, 2024 •

edited

Loading

codecov bot commented May 23, 2024 •

edited

Loading

AlanGriffiths left a comment

AlanGriffiths May 24, 2024

AlanGriffiths May 24, 2024

mattkae May 24, 2024

AlanGriffiths May 24, 2024

mattkae May 24, 2024

AlanGriffiths May 24, 2024

AlanGriffiths May 24, 2024

mattkae May 24, 2024

AlanGriffiths May 24, 2024

AlanGriffiths May 24, 2024

mattkae May 24, 2024

Saviq commented May 24, 2024

AlanGriffiths commented May 24, 2024

mattkae commented May 24, 2024

RAOF May 27, 2024

mattkae May 28, 2024

RAOF May 27, 2024

RAOF May 27, 2024

mattkae May 28, 2024

RAOF left a comment

AlanGriffiths left a comment

		6. (X11) Test that `mattermost` can be opened and resized. This was first encountered in
		[#2509](https://github.com/canonical/mir/issues/2509). To test:

		2. Set orientation of your outputs in `$FILENAME` [normal, left, right, inverted]
		3. Confirm that the output has the correctly rotated content

	3. Enter `Ctrl + Alt + Backspace`
	3. "Send Key" `Ctrl + Alt + Backspace`


		- Confirm that the keyboard works after you have disconnected and reconnected it. This
		issue was first encountere in [#3149](https://github.com/canonical/mir/issues/3149).

	issue was first encountere in [#3149](https://github.com/canonical/mir/issues/3149).
	issue was first encountered in [#3149](https://github.com/canonical/mir/issues/3149).

docs: augmenting the test plan with X11 clients #3397

docs: augmenting the test plan with X11 clients #3397

Conversation

mattkae commented May 23, 2024 • edited Loading

What's new?

Tickets Created

codecov bot commented May 23, 2024 • edited Loading

Codecov Report

AlanGriffiths left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saviq commented May 24, 2024

AlanGriffiths commented May 24, 2024

mattkae commented May 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RAOF left a comment

Choose a reason for hiding this comment

AlanGriffiths left a comment

Choose a reason for hiding this comment

mattkae commented May 23, 2024 •

edited

Loading

codecov bot commented May 23, 2024 •

edited

Loading