Notification e-mail sketches

[spbnick]

I've spend the past few weeks trying to bring the new notification system to a state where we can produce a notification message which is short and sweet, but also summarizes results correctly. I got some feedback on the last meeting and tried to accommodate it. Below are some results for you to give additional feedback to.

However, please remember it is more important to reach out to developers, get their feedback, and be able to iterate quickly, rather than make it perfect the first time (which is not really possible). So, I'll try to address your requests and comments, but would rather move on sooner.

Anyway, here go the samples (note the links don't work yet).

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=f78d76e72a4671ea52d12752d92077788b4f5d50&var-patchset_hash=

OVERVIEW

     Revision: ✅ PASS
       Builds: ✅ PASS
        Tests: ❌ FAIL

REVISION

    Status
        ✅ PASS
    Commit
        name: v5.12-rc2-338-gf78d76e72a46
        hash: f78d76e72a4671ea52d12752d92077788b4f5d50
    Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
    By
        arm, redhat, syzbot, tuxsuite

BUILDS

    Status
                 ✅ 679
    Architectures
            arm  ✅ 274
          arm64  ✅  80
         x86_64  ✅  77
           mips  ✅  69
        powerpc  ✅  48
          riscv  ✅  30
           i386  ✅  28
           s390  ✅  21
             sh  ✅  18
            arc  ✅  10
         parisc  ✅   9
          sparc  ✅   9
        aarch64  ✅   2
        ppc64le  ✅   2
          s390x  ✅   2
    By
        arm, redhat, syzbot, tuxsuite

TESTS

    Status
        ❌ 6  ✅ 27  🚧❌ 3  🚧✅ 2
    Failures
        ❌ ltp
        ❌ kselftest-build
        ❌ redhat_cifs_connectathon
        ❌ redhat_geneve
        ❌ redhat_gre
        ...
    By
        arm, redhat, syzbot

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=f78d76e72a4671ea52d12752d92077788b4f5d50&var-patchset_hash=

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is faulty.
    ✅ PASS     - Passed. Tested code is correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7a7fd0de4a9804299793e564a555a49c1fc924cb&var-patchset_hash=

OVERVIEW

     Revision: ✅ PASS
       Builds: ❌ FAIL
        Tests: ❌ FAIL

REVISION

    Status
        ✅ PASS
    Commit
        name: v5.12-rc1-19-g7a7fd0de4a98
        hash: 7a7fd0de4a9804299793e564a555a49c1fc924cb
    Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
    By
        kernelci, redhat, syzbot, tuxsuite

BUILDS

    Status
                 ❌ 1  ✅ 858
    Architectures
            arm  ❌ 1  ✅ 331
          arm64  ➖    ✅ 116
         x86_64  ➖    ✅ 115
           mips  ➖    ✅  85
        powerpc  ➖    ✅  62
          riscv  ➖    ✅  36
           i386  ➖    ✅  32
           s390  ➖    ✅  29
             sh  ➖    ✅  18
            arc  ➖    ✅  10
          sparc  ➖    ✅   9
         parisc  ➖    ✅   9
        aarch64  ➖    ✅   2
        ppc64le  ➖    ✅   2
          s390x  ➖    ✅   2
    Failures
        ❌ 1  arm allnoconfig
    By
        kernelci, redhat, syzbot, tuxsuite

TESTS

    Status
        ❌ 15  ✅ 27  🚧❌ 12  🚧✅ 7  🚧⏩ 3
    Failures
        ❌ baseline
        ❌ ltp-crypto
        ❌ baseline-nfs
        ❌ usb
        ❌ sleep
        ...
    By
        kernelci, redhat, syzbot

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7a7fd0de4a9804299793e564a555a49c1fc924cb&var-patchset_hash=

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is faulty.
    ✅ PASS     - Passed. Tested code is correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7882bede530f1c1f87a98a492c0d2e759945c138&var-patchset_hash=

OVERVIEW

     Revision: ✅ PASS
       Builds: ❌ FAIL

REVISION

    Status
        ✅ PASS
    Commit
        name: v4.19.177-248-g7882bede530f
        hash: 7882bede530f1c1f87a98a492c0d2e759945c138
    Checked out from
        https://gitlab.com/Linaro/lkft/mirrors/stable/linux-stable-rc
    By
        tuxsuite

BUILDS

    Status
                ❌ 108  ✅ 36
    Architectures
           arm  ❌  45  ✅ 24
          mips  ❌  27  ✅  5
         arm64  ❌  12  ➖   
        x86_64  ❌   6  ✅  3
         sparc  ❌   6  ✅  3
          s390  ❌   6  ✅  1
          i386  ❌   6  ➖   
    Failures
        ❌ 6  mips tinyconfig
        ❌ 6  mips allnoconfig
        ❌ 6  arm64 allnoconfig
        ❌ 6  arm64 tinyconfig
        ❌ 3  arm footbridge_defconfig
        ...
    By
        tuxsuite

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7882bede530f1c1f87a98a492c0d2e759945c138&var-patchset_hash=

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is faulty.
    ✅ PASS     - Passed. Tested code is correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.

[spbnick]

@khilman also asked if we could have %fail shown in the notifications. I gave it a thought and tried various combinations. I wasn't able to arrive at a really good solution, but maybe you could have some ideas.

Basically, if we want to keep sorting the statuses horizontally from most to least important (e.g. first show build failures, then successes, then unknown statuses), then the least confusing thing is either to just replace result counts with percentages, or put the percentages into each column.

I.e. this:

    Status
                ❌ 108  ✅ 36
    Architectures
           arm  ❌  45  ✅ 24
          mips  ❌  27  ✅  5
         arm64  ❌  12  ➖   
        x86_64  ❌   6  ✅  3
         sparc  ❌   6  ✅  3
          s390  ❌   6  ✅  1
          i386  ❌   6  ➖   

Could be replaced either with this:

    Status
                ❌  85%  ✅  25%
    Architectures  
           arm  ❌  65%  ✅  35%
          mips  ❌  84%  ✅  16%
         arm64  ❌ 100%  ➖   
        x86_64  ❌  67%  ✅  33%
         sparc  ❌  67%  ✅  33%
          s390  ❌  86%  ✅  14%
          i386  ❌ 100%  ➖   

Or something like this:

    Status
                ❌ 108 ( 85%)  ✅ 36 ( 25%)
    Architectures                    
           arm  ❌  45 ( 65%)  ✅ 24 ( 35%)
          mips  ❌  27 ( 84%)  ✅  5 ( 16%)
         arm64  ❌  12 (100%)  ➖    
        x86_64  ❌   6 ( 67%)  ✅  3 ( 33%)
         sparc  ❌   6 ( 67%)  ✅  3 ( 33%)
          s390  ❌   6 ( 86%)  ✅  1 ( 14%)
          i386  ❌   6 (100%)  ➖

The former seems a bit lacking in information (i.e. an architecture with just one build which failed would look the same as one with a hundred builds which all failed), but perhaps it's OK. The latter, though, looks positively crowded, even if we manage to reduce punctuation somehow, and if there are any unfinished builds, another column of numbers would make this still harder to comprehend.

One thought I had is to use emojis or special Unicode characters to somehow represent percentages. I searched for a while, and this is the best I was able to find. Judge for yourself (GitHub font might not work well here, try pasting this into your e-mail client or terminal):

    Status
                ❌ 108 ◕  ✅ 36 ◔
    Architectures
           arm  ❌  45 ◕  ✅ 24 ◔
          mips  ❌  27 ◕  ✅  5 ◔
         arm64  ❌  12 ●  ➖    ○
        x86_64  ❌   6 ◕  ✅  3 ◔
         sparc  ❌   6 ◕  ✅  3 ◔
          s390  ❌   6 ◕  ✅  1 ◔
          i386  ❌   6 ●  ➖    ○

Another option, using shaded blocks:

    Status
                ❌ 108 ▓  ✅ 36 ░
    Architectures
           arm  ❌  45 ▓  ✅ 24 ░
          mips  ❌  27 ▓  ✅  5 ░
         arm64  ❌  12 █  ➖
        x86_64  ❌   6 ▓  ✅  3 ░
         sparc  ❌   6 ▓  ✅  3 ░
          s390  ❌   6 ▓  ✅  1 ░
          i386  ❌   6 █  ➖    

Finally, one last option is using phases of the moon (yep) to quickly and (hopefully) unambiguously show how much good and bad stuff there is:

    Status
               🌒  ❌ 108  ✅ 36
    Architectures         
           arm 🌒  ❌  45  ✅ 24
          mips 🌒  ❌  27  ✅  5
         arm64 🌑  ❌  12  ➖ 
        x86_64 🌒  ❌   6  ✅  3
         sparc 🌒  ❌   6  ✅  3
          s390 🌒  ❌   6  ✅  1
          i386 🌑  ❌   6  ➖ 

[mh21]

I fondly remember the weather icons of Jenkins 😂

[spbnick]

Yeaaah, right, there are those too 😂, but their meaning was always a little foggy to me 😁.

[spbnick]

Get it, "foggy"? 🙈

[spbnick]

Hmm, I think I actually like the moon phases the best.

[khilman]

IMO the shaded blocks or phases of moon are not very intuitive. We could experiment with those on a web UI, but for email reports, I think we should stick to the simple count plus percentage in parens example above.

[spbnick]

Also, should I abbreviate the list of architectures too? It's getting a little long, I think :D

[khilman]

What does pass/fail for a revision mean?

[spbnick]

Whether patches applied or not, if any.

[khilman]

That's not very obvious to me, so I don't think that is clear for casual users.
e.g. it doesn't really have any meaning for an unpatched kernel, correct? So maybe it should be left out of email reports unless there's a patched kernel, and then be shown as "patches applied: pass/fail".

[spbnick]

Yeah, agreed, we can hide the status for non-patched revisions. I'll think more on how to represent that status for patched ones.

[spbnick]

Alright, taking a snapshot of the discussion so far, here and on meetings, and proceeding with something like the below samples. Let's leave further refinement for later.

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=219d54332a09e8d8741c1e1982f5eae56099de85&var-patchset_hash=6979ed11e9afd456f4a031a7983a175d36bc0b4b4b45aff0000a80abb15c2ce4

OVERVIEW

      Patches: ✅ PASS
       Builds: ✅ PASS

REVISION

    Status
        ✅ PASS
    Commit
        hash: 219d54332a09e8d8741c1e1982f5eae56099de85
    Checked out from
        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
    With 112 patches applied
        https://raw.githubusercontent.com/GKernelCI/linux-patches/5.4/1000_linux-5.4.1.patch
        https://raw.githubusercontent.com/GKernelCI/linux-patches/5.4/1001_linux-5.4.2.patch
        https://raw.githubusercontent.com/GKernelCI/linux-patches/5.4/1002_linux-5.4.3.patch
        https://raw.githubusercontent.com/GKernelCI/linux-patches/5.4/1003_linux-5.4.4.patch
        https://raw.githubusercontent.com/GKernelCI/linux-patches/5.4/1004_linux-5.4.5.patch
        ...
    By
        gkernelci

BUILDS

    Status
               ✅ 5
    Architectures
        amd64  ✅ 1
        arm64  ✅ 1
          arm  ✅ 1
        ppc64  ✅ 1
        sparc  ✅ 1
    By
        gkernelci

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=219d54332a09e8d8741c1e1982f5eae56099de85&var-patchset_hash=6979ed11e9afd456f4a031a7983a175d36bc0b4b4b45aff0000a80abb15c2ce4

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is likely faulty.
    ✅ PASS     - Passed. Tested code is likely correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7882bede530f1c1f87a98a492c0d2e759945c138&var-patchset_hash=

OVERVIEW

       Builds: ❌ FAIL

REVISION

    Status
        ✅ PASS
    Commit
        name: v4.19.177-248-g7882bede530f
        hash: 7882bede530f1c1f87a98a492c0d2e759945c138
    Checked out from
        https://gitlab.com/Linaro/lkft/mirrors/stable/linux-stable-rc
    By
        tuxsuite

BUILDS

    Status
                ❌ 108  ✅ 36
    Architectures
           arm  ❌  45  ✅ 24
          mips  ❌  27  ✅  5
         arm64  ❌  12  ➖   
        x86_64  ❌   6  ✅  3
         sparc  ❌   6  ✅  3
          s390  ❌   6  ✅  1
          i386  ❌   6  ➖   
    Failures
        ❌ 6  mips tinyconfig
        ❌ 6  mips allnoconfig
        ❌ 6  arm64 allnoconfig
        ❌ 6  arm64 tinyconfig
        ❌ 3  arm footbridge_defconfig
        ...
    By
        tuxsuite

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7882bede530f1c1f87a98a492c0d2e759945c138&var-patchset_hash=

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is likely faulty.
    ✅ PASS     - Passed. Tested code is likely correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.

Below is the summary of results Kernel CI database has recorded
for this revision so far. See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7a7fd0de4a9804299793e564a555a49c1fc924cb&var-patchset_hash=

OVERVIEW

       Builds: ❌ FAIL
        Tests: ❌ FAIL

REVISION

    Status
        ✅ PASS
    Commit
        name: v5.12-rc1-19-g7a7fd0de4a98
        hash: 7a7fd0de4a9804299793e564a555a49c1fc924cb
    Checked out from
        https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
    By
        kernelci, redhat, syzbot, tuxsuite

BUILDS

    Status
                 ❌ 1  ✅ 858
    Architectures
            arm  ❌ 1  ✅ 331
          arm64  ➖    ✅ 116
         x86_64  ➖    ✅ 115
           mips  ➖    ✅  85
        powerpc  ➖    ✅  62
          riscv  ➖    ✅  36
           i386  ➖    ✅  32
           s390  ➖    ✅  29
             sh  ➖    ✅  18
            arc  ➖    ✅  10
          sparc  ➖    ✅   9
         parisc  ➖    ✅   9
        aarch64  ➖    ✅   2
        ppc64le  ➖    ✅   2
          s390x  ➖    ✅   2
    Failures
        ❌ 1  arm allnoconfig
    By
        kernelci, redhat, syzbot, tuxsuite

TESTS

    Status
        ❌ 15  ✅ 27  🚧❌ 12  🚧✅ 7  🚧⏩ 3
    Failures
        ❌ baseline
        ❌ ltp-crypto
        ❌ baseline-nfs
        ❌ usb
        ❌ sleep
        ...
    By
        kernelci, redhat, syzbot

See complete and up-to-date report at:

    https://staging.kernelci.org:3000/d/checkout/checkout?orgId=1&var-git_commit_hash=7a7fd0de4a9804299793e564a555a49c1fc924cb&var-patchset_hash=

LEGEND

    💥 ERROR    - Aborted. Test, tested code, or both might be faulty.
    ❌ FAIL     - Failed. Tested code is likely faulty.
    ✅ PASS     - Passed. Tested code is likely correct.
    🆗 DONE     - Finished. Status of tested code is unknown.
    ⏩ SKIP     - Skipped. Planned, but didn't execute.
    ❓ UNKNOWN  - In progress, or status unknown.

    🚧 WAIVED   - Waived result. Test is too new or shows known failures.

    ➖ BLANK    - No data, zero.