[Feature] SPIKE Prime UI

[laurensvalk]

While most of the hubs have just one status light, SPIKE Prime and MINDSTORMS Inventor have more buttons and a light grid. This issue is to define their user interface (UI).

UI is a strict superset of the UI for other hubs

• The status light works the same
• The center button works the same as the button on the other hubs
• Pybricks code works the same

Use the light grid to convey the same or additional information such as:

• Program is running or stopped
• Program slot, since SPIKE Prime can have more than one.

User control of light grid

• The user can display arbitrary things on the grid.
• If they do, the UI is not visible until the program ends.

About program slot

• Use the left and right button to select. A single pixel indicates which is selected.
• Slot remains visible while program is running and while stopped.
• 5 slots might be enough. Does anyone ever use more? You can always make an extra menu in your script.
• Pybricks Code won't have to know about slots. It just sends the script; the hub saves it at the current slot.

The above could look something like this:

[laurensvalk]

Here is a simple script that mimics this behavior. This will give you a sense of what this might look like.

If you try this, any input is appreciated.

To exit this test script, press Bluetooth and Center button simultaneously.

from pybricks.hubs import InventorHub
from pybricks.parameters import Color, Button
from pybricks.tools import wait, StopWatch
from pybricks.geometry import Matrix

hub = InventorHub()
hub.light.off()

slot = 0
running = False

X = 100
STOP = Matrix([
    (0, X, X, X, 0),
    (0, X, X, X, 0),
    (0, X, X, X, 0),
    (0, 0, 0, 0, 0),
    (0, 0, 0, 0, 0),
])

watch = StopWatch()
def update_animation():
    cycle = ((watch.time() // 40) * 9) % 256
    slot_row = [(100 if slot == i else 0) for i in range(5)]
    slot_image = Matrix([[0, 0, 0, 0, 0]] * 4 + [slot_row])

    b = [0] * 8
    for i in range(8):
        offset = (cycle + i * 32) % 256
        b[i] = 0 if offset > 200 else (offset if offset < 100 else 200 - offset)
    
    run_image = Matrix([
        (0, b[0], b[1], b[2], 0),
        (0, b[7], 0,    b[3], 0),
        (0, b[6], b[5], b[4], 0),
        (0, 0, 0,       0,    0),
        (0, 0, 0,       0,    0),
    ])
    hub.display.icon(slot_image + (run_image if running else STOP))


while True:

    try:
    
        pressed = ()
        while not any(pressed):
            pressed = hub.buttons.pressed()
            update_animation()
        
        hub.display.pixel(4, slot, 0)
        if not running and Button.LEFT in pressed:
            slot = max(slot - 1, 0)
        elif not running and Button.RIGHT in pressed:
            slot = min(slot + 1, 4)
            
        hub.display.pixel(4, slot)
            
        while any(hub.buttons.pressed()):
            update_animation()
    except SystemExit:
        if Button.BLUETOOTH in pressed:
            raise SystemExit
        running = not running
        if running:
            print("Starting program:", slot)
        else:
            print("Stopped")

[laurensvalk]

This is mostly done, except for multi-program slots. We won't be adding multiple slots for the time being, so we can close this.

If we have multi-file projects as in #189, then users could make their own interactive menus to make "slots".

[laurensvalk]

We won't be adding multiple slots for the time being, so we can close this.

I think it is now time to re-open and start completing this issue, four years on! 😄

To start things off, here is a video of the UI suggested at the time. You can try it for yourself using the script above to get a real feel if you like. What do you think? @afarago, @ggramlich, @DrTom

Completely different ideas are welcome too.

spikeui.mp4

It is a fairly minimalist UI, but it has some advantages:

• You can see everything in one glance.
• You can still see your program choice after you start.
• Fast. No fancy scroll animations.
• You can operate this with your eyes closed, or with the hub hidden. (Since it doesn't wrap around.)
• Now you all know why we have that strange 3x3 square 😄 This was the plan all along...

Potential downside: "just 5 slots":

• But do you really need more out of the box? You can still make your own fancy menus as you can now.
• We could extend this idea up to 10 slots. So 6--10 will just have the colors inverted.

Pybricks Code UI implications:

• Potentially none. We could just always download it to the selected slot, or at least start that way.
• Users might be used to selecting the slot per program on the PC, but I have found this confusing at times as well. It means you have to mentally keep the two in sync, or accidentally override something. So there's something nice about keeping this super minimal.
• In either variant, we can display the hub state visually when it is connected.

[afarago]

Love it! This is how it should be! Simple, elegant, straightforward and verh handy if Pybricks make it to multiple programs download.

I would default to slot 1 (0) and as a first line comment add to the program which slot it should download to. Naming wise I would use 1 based indexing.

---

(Maybe i's too early for this discussion, yet I just add it here)
I would maybe differentiate somehow between user and built-in programs.
Keep 5 slots for user ptograms and 5 slots for built in programs.

Switch trigger could be long press left button, visual could be blinking program selection pixel.

[laurensvalk]

Thank you!

Yes, I agree that the user exposed index should be 1 based. And if we are creative about it visually, maybe it doesn't need any number.

Question for coaches, is 5 slots enough, also for beginning teams?

Keep 5 slots for user ptograms and 5 slots for built in programs.

Other than port view (/ heart program), do any of the builtin programs need to be triggered from the hub? If it's the only one, perhaps left+right could be used start it. Or attempting to scroll to the 6th program could start it, so it is easier to discover.

[laurensvalk]

Tying into #1716, I think this proposed UI for the light matrix combines well with having the hub status light off entirely, just like in the video. Blue is then exclusively reserved for the Bluetooth light in various different states (in this video, it is already connected, so solid).

[laurensvalk]

Implementing this was easier than I thought. I'll have a look at combining it with #1716 so it gets in a reasonable state for review before publishing this.

[afarago]

If the beginner teams could wish they would ask for a function to control the state of the "program counter" programatically.

Goal: e.g. to step to the next mission automatically after running the current round successfully. Similar to hub_menu.

Proposal

import ui_menu from pybricks.tools

idx = ui_menu.current_program()
ui_menu.current_program(min(5,idx+1))

[laurensvalk]

If the beginner teams could wish they would ask for a function to control the state of the "program counter" programatically.

Team's wishes are our command 🧞 ! This should be quite easy to implement. We'll just need to carefully document what the existing hub_menu is for, since it can still be useful.

[laurensvalk]

Here is some preliminary progress of doing the above at the firmware level, along with #1716 to move bluetooth state to the bluetooth light. The program slot is made part of the "idle UI" (formerly the "square"), so that it gently fades in and out on boot and power-off.

Also includes demo of starting port view.

Not yet decided about the blue animations on the main button in this case. In this demo it is hidden, but that is not necessarily how it will be.

spikeui2.mp4

[afarago]

It is awesome!

Came to think of the user interaction flow; why do we need the heart program hub "activation"?

The user activates "port view" from the IDE by e.g. clicking the connected hub name field at bottom right.

---

Though in the future it would be awesome to add it in reverse as well: user can activate the port view (and dialog/subview pops up) when pressing this combo on the hub.

[laurensvalk]

why do we need the heart program hub "activation"?

It would be nice to have it not just send out telemetry but also be useful on its own. Then you'd also want to be able to start it without a computer.

For example, sometimes I would just like an easy way to see the reflection value, or maybe test drive a motor. While the main menu (built into the firmware) is very simple on purpose, I think we could get pretty creative with the builtin Port View program in Python. If we do, it could get a more generic name like Port Control.

Once started, perhaps you could use the arrow keys to activate other modes... Not just for display, but also RC mode, for example: Then it will automatically look for the Powered Up remote or Xbox Controller to control all the ports.

[laurensvalk]

So from a UI perspective, this single "interactive hub program" is recognizable as a softly fading icon, as in the video above.

To enter, press < and > simultaneously. To exit, press them again...

Use < or > to go to the next or previous mode.

Default mode

• Send telemetry to PC, if connected
• Toggle through ports A, B, C, D, E, F using a button to see one sensor value on 5x5 display.

⬛🟨⬛🟨⬛
🟨🟨🟨🟨🟨
🟨🟨🟨🟨🟨
⬛🟨🟨🟨⬛
⬛⬛🟨⬛⬛

Play mode

• Scans for a remote (perhaps both types)
• Used to test motors or drive vehicles.
• Chosen functionality within this mode could be indicated using colors. Maybe toggle through the ports using the center button on the remote.

⬛🟨⬛⬛⬛
⬛🟨🟨⬛⬛
⬛🟨🟨🟨⬛
⬛🟨🟨⬛⬛
⬛🟨⬛⬛⬛

co-op mode

I feel like we could do something creative out of the box with hub to hub communication, though I'm not sure exactly what this might do. Maybe make sound when a nearby hub in the same mode is detected. (Teachers might object 😄 )

🟨🟨🟨🟨🟨
⬛⬛⬛⬛🟨
🟨🟨🟨⬛🟨
⬛⬛🟨⬛🟨
🟨⬛🟨⬛🟨

[laurensvalk]

We might also want to have the ability to delete one or all program slots. This is often asked for with the stock firmware.

[laurensvalk]

The program only starts when you release the center button. It has always been that way in Pybricks, so that your robot performance isn't affected by the user still holding down the button, throwing the robot off course.

I can't really explain it, but with the light off this waiting on release feels a bit like a "delay".

Maybe we can combine some visual indication that the button was pressed prior to starting, perhaps combined with David's suggestion in #1716 (comment).