Friction Log: Post-Demuxed 2021

[dylanjha]

This is basically a brain dump of things we ran into when using media-chrome for Demuxed 2021. We can open up sub-discussions for the major pieces, but this is a sort of grab-bag of all kinds of things I ran into.

For context, this is what the VOD player looks like on mobile + desktop. The Live player was the same but missing a bunch of UI elements that don't make sense on live (time range, seek forward/back playback rate, things like that)

We used this branch of @cjpillsbury 's fork

Not mobile friendly

• The default design examples and tools were clearly not intended to be used on mobile.
• Starting with <media-control-bar>, the mobile styling is busted. With any sensible amount of controls, I quickly run out of space. The control bar defaults to using flex wrap on controls, which is find, but clearly feels like a band-aid to jimmy-rig a desktop-first solution into a mobile viewport
• Other players in the wild that work well on mobile (Netflix, Youtube, etc) all have a concept of overlaying controls on the media element. media-chrome did not give me any utilities for doing so, nor provided any examples
• The <media-time-range> (seek bar, scrub bar, progress bar, whatever you want to call it) on mobile does not work while the media is playing. It's unusable. This is probably the most significant bug. I believe the underlying reason is because the media element is triggering updates and the element is an input type="range" so it gets those updated values and keeps re-setting it's internal state. There have been discussions around other pros/cons to using input type="range" (very few players in the wild that we looked at choose to go this route)

Styling in general

• Using custom slot= for icons is beautiful. That felt really natural and easy
• The great part about styling media-chrome is the ability to control each individual element and show/hide/resize with CSS. My markup (psuedo-code) ended up looking a bit like this in React code (below)
• The default of <media-controller> being display: inline-flex; flex-direction: column-reverse felt wrong, I had to work around that default immediately.

<media-controller>
  <mux-video />
  <div className="large-play">
      {/* this was positioned relative and overlaid the entire player when in the paused state
         it was really easy to hide the play button when the video was playing with pure css - that was :cool::
   
         .large-play media-play-button {
           display: none;
         }
         .large-play media-play-button[media-paused] {
           display: block;
         }
   */ }
    <LargePlayButton /> // this was a `<media-play-button>` with a custom slot=
  </div>
  <div className="mobile-time-range">
      {/* awesome that I can have this custom class with a media-query to show/hide at different breakpoints
           also awesome to conditionally render DOM elements easily in react world. This felt very natrual
     */ }
    {!isLive && <media-time-range />}
    {!isLive && <media-time-display show-duration />}
    <div className="control-bar-wrapper">
       <media-control-bar>
          {/* bunch of control bar things, some media-* elements, and some custom react components */ }
        </media-control-bar>
    </div>
  </div>
</media-controller>

Container sizing

• Need aspect ratio work: Experimenting with an aspect-ratio attribute #66 , in every implementation of media-chrome I end up having to do something like this:

1. put an outside container with padding-bottom % to get the aspect ratio that I'm going for and use background image blur hash CSS.
2. style media-controller with: position: relative; width: 100%; height: 100%; top: 0; left: 0 so it takes up the whole container. This feels like some boilerplate that can be cleaned up.

Missing cc captions button

• We had to add this on a fork
• The cc captions button we added was basic, all it did is enable/disable the showing of the captions track on the media element, we did not take over the presentation of captions

Missing airplay and chromecast controls

• We added airplay at our application level, wasn't able to add chromecast in time
• It was really cool that within <media-control-bar> we can add our own custom react components intertwined with the <media-* UI elements. That felt like a superpower and not something that is easily done in other players

{hasAirplay && <AirPlayButton onClick={() => muxVideoRef.current.webkitShowPlaybackTargetPicker()} />}

Icon sizing is wonky when using custom slots

• The icons felt a bit wonky, they didn't always take up the full space of the container and was difficult to get sizing right. I had to do things like this:

media-play-button {
  width: 40px;
  align-items: center;
  justify-content: center;
}
media-play-button[media-paused] {
  --media-button-icon-width: 20px;
}
media-play-button {
  --media-button-icon-width: 11px;
}

Loading indicator

• This didn't exist, so we added it on the fork. It worked really well, the application had to control the showing/hiding of that, which was fairly easy, but media-controller should be able to detect the loading state of the media element. The React code we had for that was pretty trivial, we could call the checkLoading callback in a bunch of places like timeupdate, canplay, waiting, stalled and we didn't run into any bugs with this.

  const checkLoading = useCallback(() => {
    if (muxVideoRef.current?.paused) {
      setIsLoading(false);
    } else if (muxVideoRef.current?.readyState >= 3) {
      setIsLoading(false);
    } else {
      setIsLoading(true);
    }
  }, [muxVideoRef.current]);

Missing documentation

• every UI element and the slot=s (I had to dig through source code to find this)
• all the --media-- CSS vars (I had to dig through source code to find this)

a11y

• Worked well from my testing. I was able to control everything with the keyboard and get voiceover functionality working 👍🏼

keyboard shortcuts

• I found myself wanting keyboard shortcuts. Mainly just forward / backward arrows for the skip back/skip forward 30 seconds and spacebar to play/pause,

working with

• media-chrome worked great with <mux-video> -- very happy with that combo 💯

125 lines of custom CSS

The biggest takeaway for me was the amount of custom CSS I had to write to even get a functional player on mobile. In total I had 125 lines of custom CSS. That feels like way too much.

---

Immediate Action Items:

1. Re-frame our general mindset to be mobile-first. With all our examples, every default CSS rule, every architecture decision, let's evaluate it from a mobile-first mindset. It's so easy to overlook this because we develop it all on desktop, but so many use cases in the wild require great mobile experiences and if we don't nail that then we can't win. Going from mobile to desktop is much easier than going the other way around. Relevant work being done over here: Proposal: media-interaction-feedback element #71 and Improv/styling rework #74
2. My experience around styling difficulties like aspect ratio, etc also aligns with what we have heard from early adopters so that clearly feels like the biggest blocker to getting more adoption right now. (mobile is a big part of this)

---

Overall, I'm really excited with what we have here. And it feels like knocking out some of these bigger foundational blockers around positioning, styling and mobile will really make this a breeze to work with. 💖

[mmcc]

For the icon sizing comment, when I was looking at those early on I did notice that it didn't seem like they'd been "icon-ified" or whatever. I think I'd probably want to document an expectation of icons that have a height of 1em so size can just be controlled via font-size of the control-bar. That also requires some intentional design when folks are making these icons, but it's made my life much easier in the past.

[cjpillsbury]

I took a slightly different route in my PR. I'd recommend we use viewBox and not have any explicit sizes "baked in to" our svg nodes, using external sizing (including our css vars assigned to the svg) to determine the specific size of the icon, which will also maintain aspect ratio by default when scaling. For an explainer of this approach, see: https://css-tricks.com/scale-svg/

Note that this doesn't mean we couldn't switch our width/height vars to use em units instead of px units by default, though you'll want to keep in mind that the icon isn't the full size of the button (minimally it's svg+padding).

I do like the idea of preferencing height over width with our button icons, since that's most frequently going to be the axis that we want to constrain (and then let width simply be computed based on e.g. the svg's intrinsic dimensions/aspect ratio).

[cjpillsbury]

hoping my draft/WIP PR #74 will help with some of these mobile/styling issues

[cjpillsbury]

Adding this:
There were a number of env-specific checks that needed to be added that we'll want to make sure make it in sooner rather than later. The ones that immediately come to mind:

• mobile safari (maybe just iOS): needed to handle entering/exiting full screen + controls show/hide.
  • We didn't do this, but we should also add something for https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#attr-playsinline
• Firefox: needed to hide PIP button (this could be done using feature support/API detection rather than user agent)

[cjpillsbury]

Also @dylanjha we should do a high level diff of our demuxed branch vs main at some point to make sure we don't loose effort, even if we don't solve the issues in the exact same way. This will likely need to include some auditing of what we solved at the app layer.

[dylanjha]

One important commit that fixed the bug with react-dom is this one:

b6b57b6

[cjpillsbury]

yeah we had some initialization bugs as well that I saw on last diff/compare (the volume was one of them)

[dylanjha]

Okay, I have now gone through re-building the demuxed player based on the work @cjpillsbury did in this branch #74

Note that the cc button and loading component were 2 things added on the fork and do not exist in the style fixes branch, we'll work on getting those added.

Mobile player

✅ Considerably better UX (compare this screenshot to the screenshot above). Main improvement is the controls overlaid using the new slots
✅ slot="top-controls" was handy for moving the mute, playback rate, pip and fullscreen to the top
✅ slot ="controls-overlay" was handy for putting the play button + backward/forward seekers in the middle
✅ <media-control-bar> was used for the time range + time display

Note that I still could have achieved this functionality with the old implementation, but the fact that media-chrome didn't have a default path for doing so meant that path had a lot of friction

Desktop player (no change)

✅ was able to leverage slot="controls-overlay" for the large play button instead of doing that manually

---

Application code diff

Here is the diff in a gist file: https://gist.github.com/dylanjha/0add32352bf6575a50e618695830392e#file-media-chrome-style-fixes-diff

Here is a PR to the compare (private link): https://github.com/demuxed/live.2021.demuxed.com/compare/main...dj/style-fixes

Stats:

❯ g df --stat main components/
 components/media-chrome.js  | 179 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------------------------------
 components/player-loader.js |   6 +++---
 2 files changed, 78 insertions(+), 107 deletions(-)

The amount of custom CSS code I had went from 125 lines to 81 lines (35% decrease), not bad. I don't think "minimize the number lines of custom CSS" is necessarily a north star metric, but it is an indicator, and this iteration felt a lot better.

I'll move the discussion to the PR for some more detailed feedback about the individual line by line changes.

[heff]

Great friction log! I agree in general with the directions outlined.

The control bar defaults to using flex wrap on controls, which is find, but clearly feels like a band-aid to jimmy-rig a desktop-first solution into a mobile viewport

lol, ouch. Makes sense. Using flex wrap actually firstly made it possible to put the progress range above the other controls with just ordering of the elements and 100% width on the range. (see the gif demo on the repo) The narrow mobile layout is really just unsolved.
Can we grab a screen shot of what the controls look like in this case?

The (seek bar, scrub bar, progress bar, whatever you want to call it) on mobile does not work while the media is playing. It's unusable.

By this we mean the progress handle doesn't move when you drag, despite it appearing to work otherwise, is that right?

Using custom slot= for icons is beautiful. That felt really natural and easy

I got a question in my talk about using non-svgs. Did you use SVGs for this? We should put together an example either way.

The default of being display: inline-flex; flex-direction: column-reverse felt wrong

Makes sense. Christian and I had a conversation about making a slot for the control bar. Breaking change, but we should do it.

.large-play media-play-button[media-paused]

That's awesome, and also, did you need to do that? Any child of media-container should be hidden when not paused. Unless you were overriding how that worked.

It was really cool that within we can add our own custom react components intertwined

That's cool and I hadn't thought of that. We should promote that somehow.

Icon sizing is wonky

Yeah, we need someone to dig into how to make this easy. It gets complicated with icons and SVGs that bring their own padding. Sounds like there's already some ideas.

How did you feel about creating two different control bars for mobile vs. desktop, from "Not an issue" to "too much duplication"?

[dylanjha]

lol, ouch. Makes sense.

Sorry, was that too harsh?? 😞

Did you use SVGs for this? We should put together an example either way.

Yes, used SVGs, and @cjpillsbury has a "Demuxed" theme in progress

Any child of media-container should be hidden when not paused. Unless you were overriding how that worked.

I think I needed to do that, I think I overrode that behavior but would have to revisit to understand exactly why.

How did you feel about creating two different control bars for mobile vs. desktop, from "Not an issue" to "too much duplication"?

Not an issue, honestly. The UI is so different for mobile/touch interfaces vs. desktop/mouse interfaces that it felt natural to build both of them independently.

In fact, it felt way nicer writing 2 different sets of controls and just showing/hiding the whole thing with media queries compared to other players where I'm trying to shove all of them into a single control bar that tries to be responsive.