Multiple JSON page requests made before falling back to watch page

[benkaiser]

Created a comment on the relevant line here: 60cdb41#r44919269

Basically it appears that the token logic is setting a promise to the cache rather than the actual value now. As a result I'm seeing ytdl make a bunch of json requests all with "reload now" responses before finally falling back to the watch page.

It does eventually resolve with the right information, but it's causing the performance to be sub-par (~several seconds for getInfo).

Here is a network trace with the current logic:

First request:

Second request:

And by updating getIdentityToken to this: (basically don't use cookieCache.getOrSet, actually make it a promise and cookieCache.set in the later part of the promise):

const getIdentityToken = async (id, options, key, throwIfNotFound) => {
  let page = await getHTMLWatchPageBody(id, options);
  let match = page.match(/(["'])ID_TOKEN\1[:,]\s?"([^"]+)"/);
  if (!match && throwIfNotFound) {
    throw new UnrecoverableError('Cookie header used in request, but unable to find YouTube identity token');
  }
  exports.cookieCache.set(key, match && match[2]);
  return match && match[2];
};

First request:

Second request:

[fent]

the json endpoint returning {"reload":"now"} started happening about 1 to 2 weeks ago. it's not related to the cache returning a promise. the identity token request happens after the json request, when it previously wasn't needed since we'd get a good reply on the first request

[benkaiser]

@fent surely we aren't intending on passing a header with [object Promise] as the value though?

[github-actions]

🎉 This issue has been resolved in version 4.1.7 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[benkaiser]

Hey @fent, I think we need to revisit. I'm still seeing promises being sent in the x-youtube-identity-token.

Would you be open to my submitting a pull-request to fix?

[TimeForANinja]

fent is currently taking a bit of a break
but if you manage to fix it a PR is always welcome 😉
it's a good start for a discussion