streamingData not found

[SrRapero720]

I am getting trying to get video data using the doc but is throwing me a BadPageException

Videos tested:
e3l1piy1pn8
https://www.youtube.com/watch?v=GmG4X9PGOXs
https://www.youtube.com/watch?v=5UW3oWn60D0

Well i tested more videos and always i get the same error
Gist: https://gist.github.com/SrRapero720/ecf46d27d76383ca3ecf098c9af8fe74

Aditional Context:
The package isn't the original, because i shadow the package in the jar. because i am working on a minecraft mod doing rendering stuff. and other mods implements this dependency but with old versions and exporting the same package cause crashes.

code used:

    @Nullable
    public static String getYoutubeRawURL(String videoUrl) throws MalformedURLException {
        // I tested with both cases, using videoID and videoUrl. nothing works
        var videoID = isAValidYoutubeURL(videoUrl) ? getVideoID(new URL(videoUrl)).get("v").get(0) : DEFAULT_URL;
//        var videoID = videoUrl;
        WaterFrames.LOGGER.info(videoID);
        synchronized (CACHE) {
            var hit = CACHE.get(videoID);
            if (hit != null && ((hit.url != null && System.currentTimeMillis() - hit.time < 1000 * 60 * 10) || (hit.url == null && System.currentTimeMillis() - hit.time < 1000 * 60))) return hit.url();
            var mediaInfo = youtubeDownloader.getVideoInfo(new RequestVideoInfo(videoID).callback(new YoutubeCallback<VideoInfo>() {
                @Override
                public void onFinished(VideoInfo data) {
                    String url = data.bestVideoWithAudioFormat().url();
                    WaterFrames.LOGGER.info(url);
                }

                @Override
                public void onError(Throwable throwable) {
                    WaterFrames.LOGGER.error("failed", throwable);
                }
            }));
            var video = mediaInfo.data();

            if (mediaInfo.error() != null) throw new RuntimeException(mediaInfo.error());


            String url = video.bestVideoWithAudioFormat().url();
            hit = new URLCacheEntry(System.currentTimeMillis(), url);
            CACHE.put(videoID, hit);
            return hit.url;
        }
    }

[sealedtx]

just tested your videos, I got successfull results sometimes, but also errors sometimes

error is caused by youtube error response:

    "status": "ERROR",
    "reason": "This video is unavailable",
    "errorScreen": {
      "playerErrorMessageRenderer": {
        "subreason": {
          "runs": [
            {
              "text": "Watch on the latest version of YouTube.",

it happens when youtube-downloader tries to spoof Android client:
https://github.com/sealedtx/java-youtube-downloader/blob/master/src/main/java/com/github/kiulian/downloader/parser/ParserImpl.java#L63

I may assume that this workaround uses google API v1 which is already deprecated and youtube may respond with errors for old clients to force them use newer version. Still not clear why something it responds with actual expected video data... And this happens not for all videos.

As a workaround for you I can suggest two ways:

1. simply retry same request until it succeed (in my case it is like 50/50 success/errors)
2. remove spoofing part parseVideoAndroid and try only with parseVideoWeb (if you already repackaged jar you can safely modify it yourself)

let me know if it works for you and I will implement better fallback procedure for this failed android client

you can also investigate similar issues and look for a solution in other youtube-related projects like https://github.com/ytdl-org/youtube-dl

[SrRapero720]

sadly I can't modify the code directly, I just did some JarInJar to make it work (and rename the package using shadowjar)

In any case, using a tool called mixin that modifies the bytecode at runtime I was able to eliminate that line of code and now the error is different.

[17:42:09] [WATERCoRE-45-65181655849003/ERROR] [waterframes/]: Failed waterframes to load:
17:42:09.056
game
java.lang.RuntimeException: me.srrapero720.github.kiulian.downloader.YoutubeException$BadPageException: streamingData and videoDetails not found
17:42:09.056
game
at me.srrapero720.waterframes.ytdl.YoutubeUtil.getYoutubeRawURL(YoutubeUtil.java:64) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:classloading}
17:42:09.056
game
at me.srrapero720.waterframes.display.MediaDisplay.lambda$new$1(MediaDisplay.java:124) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:classloading}
17:42:09.057
game
at me.srrapero720.watercore.api.thread.ThreadUtil.lambda$threadTry$1(ThreadUtil.java:21) ~[WATERCoRE-1.4.0-BETA20.jar%2365!/:1.4.0-BETA20] {re:classloading}
17:42:09.057
game
at me.srrapero720.watercore.api.thread.ThreadUtil.lambda$threadTryArgument$3(ThreadUtil.java:34) ~[WATERCoRE-1.4.0-BETA20.jar%2365!/:1.4.0-BETA20] {re:classloading}
17:42:09.057
game
at java.lang.Thread.run(Thread.java:833) [?:?] {}
17:42:09.057
game
Caused by: me.srrapero720.github.kiulian.downloader.YoutubeException$BadPageException: streamingData and videoDetails not found
17:42:09.057
game
at me.srrapero720.github.kiulian.downloader.parser.ParserImpl.parseVideoWeb(ParserImpl.java:153) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:mixin,re:classloading,pl:mixin:APP:waterframes.mixin.json:YoutubeDownloaderMixin,pl:mixin:A}
17:42:09.057
game
at me.srrapero720.github.kiulian.downloader.parser.ParserImpl.parseVideo(ParserImpl.java:65) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:mixin,re:classloading,pl:mixin:APP:waterframes.mixin.json:YoutubeDownloaderMixin,pl:mixin:A}
17:42:09.057
game
at me.srrapero720.github.kiulian.downloader.parser.ParserImpl.parseVideo(ParserImpl.java:53) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:mixin,re:classloading,pl:mixin:APP:waterframes.mixin.json:YoutubeDownloaderMixin,pl:mixin:A}
17:42:09.057
game
at me.srrapero720.github.kiulian.downloader.YoutubeDownloader.getVideoInfo(YoutubeDownloader.java:56) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:classloading}
17:42:09.057
game
at me.srrapero720.waterframes.ytdl.YoutubeUtil.getYoutubeRawURL(YoutubeUtil.java:50) ~[WATERFrAMES-1.1.0-ALPHA8.jar%2366!/:1.1.0-ALPHA8] {re:classloading}
17:42:09.057
game
... 4 more

the only change i do is redirect to send a null value and skip to parseVideoWeb

[sealedtx]

@SrRapero720 try this version:
com.github.sealedtx:java-youtube-downloader:ab074a8ccc
I created a branch where I created priority mechanism for spoofing clients and also pass extra informatio YoutubeException

Try it and dump additional information and attach it here, it will help to analyze why there is no streamingData in response.

YoutubeDownloader downloader = new YoutubeDownloader();
// by default priority is ["android", "web"]
// set only WEB client, in this case it will skip ANDROID which causes failures
downloader.getConfig().setClientsPriority(Collections.singletonList(Config.WEB_CLIENT)); 
        
            Response<VideoInfo> response = downloader.getVideoInfo(new RequestVideoInfo("GmG4X9PGOXs"));
            if (!response.ok()) {
                if (response.error() instanceof YoutubeException) {
                    YoutubeException ytE = (YoutubeException) response.error();
                    String additionalData = ytE.getAdditionalData();
                    String step = ytE.getStep();
                    String message = ytE.getMessage();
                    System.out.println(message);
                    System.out.println(step);
                    System.out.println(additionalData);
                }
            }

[YeudaBy]

Hey.
I have the same bug but with all the videos i tried.
I tried to skip the spoofing part as you said, and it worked but the video's downloading is very slow.
I would love to an update for a stable solution, thank you very much!

[WaldemarrGr]

https://video.google.com/timedtext?hl=en&type=list&v=mwiHAukbWjM - is it normal behavior that it generates such a link?

[sealedtx]

@YeudaBy probably youtube changed something in android client and now current version fails to retrieve video info
To be honest I have no time and motivation to investigate the problem. If someone can provide details how this was fixed in similar libraries like youtube-dl, I may try to implement it in java-youtube-downloader.

@WaldemarrGr I assume that your screenshot is related to subtitles? Please create separate issue, it is different error

[SrRapero720]

fent/node-ytdl-core#1216
this could help

[Sterta]

i have opened a pull request with a fix

[Sterta]

After some more investigation in the problem today, i found a better sollution, with no limitation in the video quality, i will update the pull request.

[Sterta]

now it uses ANDROID_TESTSUITE as client, as far as i could test it, their is no limitation in quality

[sealedtx]

guys I just drafted new release 3.2.3 with fix, let me know if issue still apears.
great thanks to @Sterta for investigation and fixing!