Video Stories and video blocks don't have a thumbnail image in previews

[pollyplummer]

Expected behavior

I would expect that the Video story posts would have a thumbnail shown in the post, like the image stories, but it looks like it just has a placeholder image. I created the story by uploading 1 video and publishing it without any changes. See attached image.

This was tested on WordPress for Android version 16.5-RC-1, Pixel 3 phone, Android version 11.

[designsimply]

Thank you for the report @pollyplummer!

I tested the steps you provided and found that the thumbnail works for me if I view it in a browser but the video thumbnail does not work if I view the post in a Preview or a Webview (via View Site), as mentioned on the original report. 👍

Steps to reproduce:

1. Go to My Site > + > Story post.
2. Select a video.
3. Tap the checkmark at top right.
4. Add a title and tap the "Publish Now" button.
5. Go to My Sites > View Site.
6. Observe that the video poster image (or video thumbnail) is not visible and shows an awkward placeholder image.
7. Tap the world icon (near bottom right) to open the same view in a web browser.
8. Observe that video thumbnail works correctly in a web browser.

Also:

1. Go to My Site > + > Blog post.
2. Add a video block and insert a video from the device.
3. Add a title and publish the post.
4. Go to My Sites > View Site.
5. Observe that the video poster image (or video thumbnail) is not visible and shows an awkward placeholder image.
6. Tap the world icon (near bottom right) to open the same view in a web browser.
7. Observe that video thumbnail works correctly in a web browser.

Result: video thumbnails for story and blog posts do not show up in Previews or Webviews.

Story post:

View Site > Webview for a Story Post	Firefox	Chrome

^{Tested with WPAndroid 16.5-rc-2 on Pixel 3 Android 11.}

Blog post with a video block:

View Site > Webview for a Blog Post	Firefox	Chrome

^{Tested with WPAndroid 16.5-rc-2 on Pixel 3 Android 11.}

[designsimply]

@planarvoid is it too late to include this in the Media Picker Consolidation project by chance? I know it's a stretch… 🤔 but thought I would ask just in case!

[designsimply]

Decided to add this to the list of issues to consider for Groundskeeping (our maintenance rotations) starting Feb 8 and targeting v16.8.

[planarvoid]

@designsimply the media picker consolidation project's current phase is done and this is something not related to it anyway (it's not in the media picker). I'd probably consider it as part of the Stories project? It's either stories or gutenberg but I guess Groundskeeping is fine 👍

[designsimply]

Thanks! This may also be a duplicate of #13589. After seeing that issue, I think this one may be more appropriate for Groundskeeping and I'll try to circle back to check on it in our next few rotations.

[aforcier]

This should be a little bit improved once this PR lands in WP.com and Jetpack: Automattic/jetpack#19176. (I was fixing a separate, video loading indicator issue for the block and realized it may have an effect on this issue.)

Only for stories, and only when VideoPress is enabled, the preview should show correctly in the in-app preview (once the PR has landed). This was done by setting the poster element of the video tag in the story (which is available if VideoPress is enabled).

In other cases I'm not sure what could be done since it seems like the WebView doesn't preload videos by default, and we don't have a poster available for non-VideoPress videos. Perhaps there's a setting we can choose to tweak in the in-app WebView for this. Then improving the preview in Gutenberg would be a separate issue, I don't have any insight into that one.

[mzorz]

I've only managed to scratch the surface on this one and wondered if we haven't tried this before, maybe we could try the suggestion in this [SO comment]( Add #t=1 after src in html file <source src="xyz.mp4#t=1" type="video/mp4">), wdyt?

[aforcier]

I gave the #t=1 workaround for Stories a try per @mzorz's request. It kind of works, with many caveats:

• The video src seems to be modified by WordPress by a filter when we're not using VideoPress, so that the attribute on the loaded page looks like: src=blob:https://mywebsite.com/feabf007-f441-4333-bbb8-32654ec91c27. Changing the URL to append the timestamp argument within the block doesn't work, because it gets replaced. I'm not sure how to work around that (I did all the rest of my testing by modifying the html when inspecting the webview from Chrome).
• Setting #t=1 on the video src also affects where playback starts when playing the story. However this isn't too bad, since using t=0.1, t=0.01 seems to work too, so we can reduce how much of the start of the video we shave off.
• When the timestamp argument is set, the poster attribute seems to be ignored. So we'd need to add conditional logic to use it correctly on VideoPress sites.
• I'm not very well versed in how video loading/blobs/partial loading works, but I see three partial content requests being made when the timestamp argument is set, vs only 1 previously. That seems a bit worrying.
• Related to the above point: I can't find much good documentation on what the timestamp argument makes mobile webviews (or really any browser) do. The cause for this issue was that by default those don't load videos without manual user interaction, for bandwidth reasons. Without clear documentation I'm not sure why the workaround works, or if it's even a good idea/if we're subverting intended behavior.
• I'm generally hesitant to make this hacky change to the block, especially since regular web and mobile browsers don't experience this issue. At the very least we'd want to make sure we understand what this is doing on various browsers before implementing it.

Given the above, and that all of this can only apply to the story block and not the video block (since that one is in core and we can't easily modify its behavior), I still recommend that we look for a solution in the WebView configuration itself rather than changing the block rendering. This would have the added benefit of solving the problem for video blocks too - and essentially makes this issue a duplicate of #13589 as Sheri thought.

Note also that all of this only affects Stories on non-VideoPress Jetpack sites, since VideoPress is always used for videos uploaded to WordPress.com. So it doesn't have the impact is seems to on the surface, since the poster attribute change was made.

I did, however, suggest to @mzorz that he take a look at whether we can improve the Story block in mobile Gutenberg to respect the poster attribute when that's set. This would mirror the change I made that fixed the preview for webviews, and would mean that video previews work correctly for story blocks throughout the app as long as VideoPress is enabled.

[mzorz]

The video src seems to be modified by WordPress by a filter when we're not using VideoPress, so that the attribute on the loaded page looks like: src=blob:https://mywebsite.com/feabf007-f441-4333-bbb8-32654ec91c27. Changing the URL to append the timestamp argument within the block doesn't work, because it gets replaced. I'm not sure how to work around that (I did all the rest of my testing by modifying the html when inspecting the webview from Chrome).

ah, even if it worked, this is a bummer and must be overcomplicating things. We'd have to check where the URL gets replaced (API perhaps?)

Setting #t=1 on the video src also affects where playback starts when playing the story. However this isn't too bad, since using t=0.1, t=0.01 seems to work too, so we can reduce how much of the start of the video we shave off.

Agreed, this was a known side behavior with the hack (it's actually a hack after all)

When the timestamp argument is set, the poster attribute seems to be ignored. So we'd need to add conditional logic to use it correctly on VideoPress sites.

👎

I'm not very well versed in how video loading/blobs/partial loading works, but I see three partial content requests being made when the timestamp argument is set, vs only 1 previously. That seems a bit worrying.

Yeah as far as I understand some of the video needs to be downloaded so then the browser can take a frame and present it as a poster. How it calculates how much of it needs to be downloaded is unknown to me but agreed that indicating a timestamp as opposed to a frame reference would likely make the codec need more frames to decode and extract 1 frame at the actual given timestamp.

But then this is also important, from this link:

For this feature to work, your server must support range requests and that capability must be enabled. Most servers enable range requests by default. Because some hosting services turn them off, you should confirm that range requests are available for using fragments on your site.

So basically in the case of self-hosted sites with jetpack this would be yet another hurdle to overcome (and not sure everyone can set this up, given many self-hosted WordPress sites are being hosted on shared servers)

Related to the above point: I can't find much good documentation on what the timestamp argument makes mobile webviews (or really any browser) do. The cause for this issue was that by default those don't load videos without manual user interaction, for bandwidth reasons. Without clear documentation I'm not sure why the workaround works, or if it's even a good idea/if we're subverting intended behavior

Agreed. Tried to look for some documentation backing this up myself and I also couldn't find much. The HTML5 spec doesn't even mention it EDIT: the right spect to look into this is media fragments., especially see the following paragraphs in section 7.4 Media Fragment Servers:

Reasonable Clipping: Temporal clipping needs to be as close as reasonably possible to what the media fragment specified, and not omit any requested data. "Reasonably close" means the nearest compression entity to the requested fragment that completely contains the requested fragment. For temporal fragments, this means that if a request is made for http://www.example.org/video.ogv#t=60,100, but the closest decodable range is t=58,102 because this is where a packet boundary lies for audio and video, then it will be this range that is returned. The user agent is then capable of displaying only the requested subpart and should also just do that. For some container formats this is a non-issue, because the container format allows specification of logical begin and end.

and

Reasonable byte ranges: If a single temporal range request would result in a disproportionally large number of byte ranges it may be better for the server to return a redirect to the query form of the media fragment. This situation could happen if the underlying media file is organized in a strange way.
so I wonder whether this is an implementation on the Chrome side alone that may affect other browsers or pass unnoticed for other browsers, or they may even change the behavior unadvertedly any time in the future.

so I wonder whether this is an implementation on the Chrome side alone that may affect other browsers or pass unnoticed for other browsers, or they may even change the behavior unadvertedly any time in the future.

It definitely is up to the implementation to deal with different sources (video files) and come up with the best solution for the requested timestamp, so the output may end up varying and having different results being experienced for different users.

I'm generally hesitant to make this hacky change to the block, especially since regular web and mobile browsers don't experience this issue. At the very least we'd want to make sure we understand what this is doing on various browsers before implementing it.

I still recommend that we look for a solution in the WebView configuration itself rather than changing the block rendering. This would have the added benefit of solving the problem for video blocks too - and essentially makes this issue a duplicate of #13589 as Sheri thought.

This makes a lot of sense. 👍

I did, however, suggest to @mzorz that he take a look at whether we can improve the Story block in mobile Gutenberg to respect the poster attribute when that's set. This would mirror the change I made that fixed the preview for webviews, and would mean that video previews work correctly for story blocks throughout the app as long as VideoPress is enabled.

Will look into that next week 👍

[mzorz]

Note that I edited the last comment to include the reference to the media fragments HTML5 spec.

[thehenrybyrd]

I was still able to reproduce this in WP Android 20.4-rc-2, on a Samsung Galaxy S21.
@mzorz it looks like it's been a while since you worked on this; do you think this is a reasonable sized fix for groundskeeping, or would it be better as a project?