YouTube: Some fields not found when run from translation-server

[geofferb]

When parsing a YouTube URL, the server attempts to use the YouTube translator, but ultimately skips it because it fails to read the title with the below error:

(1)(+0000000): Translation using YouTube failed

(1)(+0000000): Error: No title specified for item

    Error: No title specified for item
        at Object._itemDone (C:\zotero\modules\translate\src\translation\translate.js:615:32)
        at Object._itemDone (C:\zotero\src\translation\sandboxManager.js:96:17)
        at Object.complete (C:\zotero\modules\translate\src\translation\translate.js:1965:26)
        at scrape (eval at <anonymous> (C:\zotero\src\translation\sandboxManager.js:70:4), <anonymous>:148:7)
        at doWeb (eval at <anonymous> (C:\zotero\src\translation\sandboxManager.js:70:4), <anonymous>:75:3)
        at Zotero.Translate.Web.rest (C:\zotero\modules\translate\src\translation\translate.js:1346:49)
        at C:\zotero\modules\translate\src\translation\translate.js:1331:39
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

These lines in the translator are responsible for parsing the title:

translators/YouTube.js

Lines 106 to 108 in 7837c42

	item.title = text(doc, '#info-contents h1.title') // Desktop
	|| text(doc, '#title')
	|| text(doc, '.slim-video-information-title'); // Mobile

The titles are read in properly in both in Scaffold and the Zotero desktop client, so this seems to be an issue with the translation server.

[dstillman]

The YouTube translator uses scraping for some fields because of this:

translators/YouTube.js

Lines 101 to 103 in 7837c42

	/* YouTube won't update the meta tags for the user,
	* if they open e.g. a suggested video in the same tab.
	* Thus we scrape them from screen instead.

But when run from translation-server we only have the JSON-LD. So maybe we need to check whether the page is actually built and fall back to JSON-LD for all fields if not.

[geofferb]

@dstillman Upon further testing, I've found that the translation-server can only get the useless BreadcrumbList JSON-LD and not the VideoObject one the translator uses. However, it seems to get <meta> tags fine; could those be used as a fallback?

Also, is there a way to see the HTML retrieved by translation-server? It would make debugging this much easier.

[dstillman]

Probably, but we would have to make sure it was only using the meta tags for over-the-wire saves and never using them for Connector-based saves, due to the problem explained in the comment. So we'd have to make sure we were detecting that properly.

Also, is there a way to see the HTML retrieved by translation-server?

Yes, you can add Zotero.debug(doc.documentElement.innerHTML) or similar. (But it's likely just the same as you'd get with curl.)