Embedded video not playing offline

[Jaifroid]

This may still be related to #122, and I'm not sure if #135 may solve this, but I am struggling to find examples of embedded YouTube (or other) video that actually work when the browser/machine is fully offline. For example, on this page: http://localhost:8080/www.ready.gov_es_2022-07/A/www.ready.gov/disability-toolkit there are several embedded videos, but if the local machine is in aeroplane mode, none of them will play.

It might also be good to clarify which videos should be in a Zimit ZIM, and which should be available only as links to external content. If a site relies a lot on YouTube for serving videos that are clearly its own content, should these be available in the ZIM? I'm not sure that the distinction can be made simply on whether a video is embedded in a widget on a page, because many sites will also provide direct links to videos for accessibility reasons, and will not necessarily embed them all (plus embedded videos are often just glorified links to external content in any case).

[Jaifroid]

I see some work was done on this in webrecorder/browsertrix-crawler#4, so we should await #135 in order to real-world test the new code.

[rgaudin]

Just to be clear, YT (and other popular platforms) videos should work. That's a core zimit feature which lent us towards using this technology.
AFAIK, it's mostly broken at the moment and we are awaiting @ikreymer on this. Actually, we should prepare some simple-yet-real recipes so it's easier to diagnose, debug and fix. I believe last iteration had some use case work on Chrome but not FF. If you have interest in this, you're welcome 😉

Regarding your second remark, I don't think it's a scraper matter at the moment. ZIM “creators” should decide this per-ZIM and apply the necessary include/exclude options on the recipe. I don't think anybody invested enough effort into an actual zimit recipe to have an educated opinion on whether those are sufficient.

[ikreymer]

Embedded videos should have been fixed via openzim/warc2zim#83, if its broken again, that's unfortunately possible as platforms change these systems all the time. Is there an exact repro?

[ikreymer]

Just did a quick test of that one page - the 4 embedded youtube videos are working, tested in Chrome/Firefox/Safari.
Used latest zimit from main, which uses latest browsertrix-crawler 0.7.0-beta.3, and tested with kiwix-serve / kiwix-tools 3.2.0
Here's the zim file:
https://drive.google.com/file/d/1DmphukPo6bf7wEygbkpCMXeGeIbafkSp/view?usp=sharing

[Jaifroid]

@ikreymer Thank you. I have downloaded and tested your ZIM, and corroborate that the four embedded videos are now included in the ZIM. They play correctely with Kiwix Serve (which I tested via Kiwix Desktop).

So, for this issue to be fixed, is it enough to wait for #135? Or do we need to incorporate further updates before testing and releasing?

[Jaifroid]

@ikreymer While I have your attention, could I quickly ask whether the "fuzzy matching" you mention in openzim/warc2zim#80 is still the way in which you match the request for a video and the response? I have had great difficulty till now finding information in common between requests for video and the response, even when examining the headers for redirects. However, till now I have mostly been testing "broken" archives, where a video blob is in the archive, but Kiwix Serve has not been able to link it to a video embed either. I need to take a closer look at this working ZIM to understand the video matching process. Presumably "fuzzy" is being used as a loose description for a partial but determinate match, and there is in fact a one-to-one relationship via some piece of information in the querystring between a request and the video blob response.

[rgaudin]

@ikreymer Thank you. I have downloaded and tested your ZIM, and corroborate that the four embedded videos are now included in the ZIM. They play correctely with Kiwix Serve (which I tested via Kiwix Desktop).

I confirm as well, this is good news.

So, for this issue to be fixed, is it enough to wait for #135? Or do we need to incorporate further updates before testing and releasing?

Both zimfarm and youzimit are using master so any new ZIM we create is using it. I believe we'll just run a round of tests and make a release.