Viki Http Error 404 again

[zackmark29]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.07.28
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '-F', 'https://www.viki.com/videos/91001v-dream-high-episode-1', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[viki] Logging in
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpq2whuxta\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpq2whuxta\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Description

Yesterday in the morning. I can login and get 720p/1080p resolution.
But suddenly the error is back until now.
Could anyone also experiencing?

[RobinD42]

I'm seeing this now too, after successfully downloading from viki.com a couple weeks ago. @dstftw, can you provide the issue number this is a duplicate of so we can follow and/or comment on that issue?

[zackmark29]

I'm seeing this now too, after successfully downloading from viki.com a couple weeks ago. @dstftw, can you provide the issue number this is a duplicate of so we can follow and/or comment on that issue?

I don't know why he's always closing this post without telling the duplicate link and problem

[zackmark29]

I'm seeing this now too, after successfully downloading from viki.com a couple weeks ago. @dstftw, can you provide the issue number this is a duplicate of so we can follow and/or comment on that issue?

I think the problem from viki now is the APP_SECRET was changed again that's why we can't login

[RobinD42]

It might be #25374, I guess. At first glance it seemed a little different than this one, but with a closer look it looks like the same exception traceback.

[zackmark29]

It might be #25374, I guess. At first glance it seemed a little different than this one, but with a closer look it looks like the same exception traceback.

Yeah maybe they already working on it. Because the issue is already visible for collaborators

[blackjack4494]

I am working on a fix but keep experiencing the same strange bug. If I test my code with requests everything works totally fine. However as soon as I implement that to to viki extractor I keep getting the same 404 Error.

Honestly I am not exactly sure why. But could it be some HTTPS related problem? Tho this seems strange since my modified soundcloud extractor works totally fine with HTTPS.

Also you have to change app version to '100000a'.

Check my gist if you want to try it yourself

If you gonna enter this

https://api.viki.io/v4/sessions.json?app=100000a

you will get an actual response with json returning a 404. This should be at least downloadable when using _download_json but it isn't so my assumption it isn't using the https url
but rather this

http://api.viki.io/v4/sessions.json?app=100000a

will give you 404 (since there is no http endpoint) obviously.

[RobinD42]

What version of Python are you using? Since you're seeing a difference in behavior depending on using requests or not, then it could actually be dependent on the version of Python if requests is not being used by default. IIRC, the default SSL support in more recent versions of Python is stricter about certificate verification and may fail to validate in some common but slightly insecure situations. Requests handles the more secure cert verification and will avoid the problems. Or you can go back to Python 3.6 which was before the security changes.

Also, the patch in #25374 does fix the problem for me, if I'm using Py3.6.

[blackjack4494]

@RobinD42 I tried with 2.7, 3.5 and 3.7 there is no difference. Right now I will simply implement the login using urllib and see how it goes.

I would prefer if there would be native urllib3 or requests support

[zackmark29]

I am working on a fix but keep experiencing the same strange bug. If I test my code with requests everything works totally fine. However as soon as I implement that to to viki extractor I keep getting the same 404 Error.

Honestly I am not exactly sure why. But could it be some HTTPS related problem? Tho this seems strange since my modified soundcloud extractor works totally fine with HTTPS.

Also you have to change app version to '100000a'.

Check my gist if you want to try it yourself

If you gonna enter this

https://api.viki.io/v4/sessions.json?app=100000a

you will get an actual response with json returning a 404. This should be at least downloadable when using _download_json but it isn't so my assumption it isn't using the https url
but rather this

http://api.viki.io/v4/sessions.json?app=100000a

will give you 404 (since there is no http endpoint) obviously.

Do you think the problem is from APP_SECRET?

[zackmark29]

I tested my C# code and I'm able to login and get the token
but when I tried to download the json I'm getting the error "Response status code does not indicate success: 400 (Bad Request)."
Here's what I used
https://api.viki.io/v4/
APP: 100000a

[zackmark29]

@blackjack4494 could you tell me if this is a correct api?

https://api.viki.io/v4/videos/91001v/streams.json?app=100000a&t=1597700092&site=www.viki.com&token=h-yPux43pRp4Wj-T3sTKanIq_2u0014239459uti00k8hlyP000_djAy&sig=650eccca93e6b512103e74b1220b9398689a9ecb

this is where I'm getting the error

[blackjack4494]

@zackmark29 should work with python3+
You can also use other methods by un/commenting. The standard is that you type everything in terminal so password won't be exposed.

No clue how your c# code looks like. If you can show me somehow I could help you out.

[blackjack4494]

brief update. I managed to get it working using urllib directly. However only login and downloading of json including subtitles do work. Unfortunately the video streams json won't download as of yet (400 Bad request) I could fix that if I would know the exact problem but error handling in youtube-dl is not deep enough or I haven't figured it out yet.

All I know is that the token is invalid when it tries to fetch the video streams json.

Okay nevermind I figured out what the problem is whatsoever. It's pretty messy and complicated. If you want to download the stream json the api still wants the old APP with 5 in it. Login and everything else wants the 0 in it...

[zackmark29]

brief update. I managed to get it working using urllib directly. However only login and downloading of json including subtitles do work. Unfortunately the video streams json won't download as of yet (400 Bad request) I could fix that if I would know the exact problem but error handling in youtube-dl is not deep enough or I haven't figured it out yet.

All I know is that the token is invalid when it tries to fetch the video streams json.

Okay nevermind I figured out what the problem is whatsoever. It's pretty messy and complicated. If you want to download the stream json the api still wants the old APP with 5 in it. Login and everything else wants the 0 in it...

yeah the only problem actually is the streams.json and other API's are working.

Do you mean the 100005a? is working with the streams.json?

Edit: I tried and successfully login with app 100005a. but not yet tried with the streams.json

[zackmark29]

FINALLY! I got it and it worked! Thanks @blackjack4494 for the information.

{"1080p":{"https":{"url":"https://v4.viki.io/91001v/91001v_high_1080p_9c961d_1908070957.mp4?e=1597710457&h=ab9265b85e84c9322d03954f496d51ce","seek":"ms","cdn":"limelight","id":"94944608","bumper_duration":0}},"360p":{"https":{"url":"https://v4.viki.io/91001v/91001v_high_360p_d146ae_1908070957.mp4?e=1597710457&h=7ee518cae11b540b7f637a14c457791d","seek":"ms","cdn":"limelight","id":"94944602","bumper_duration":0}},"480p":{"https":{"url":"https://v4.viki.io/91001v/91001v_high_480p_b1d67f_1908070957.mp4?e=1597710457&h=c16485e881f066487f0f1d06258841c8","seek":"ms","cdn":"limelight","id":"94944617","bumper_duration":0}},"720p":{"https":{"url":"https://v4.viki.io/91001v/91001v_high_720p_a116d5_1908070957.mp4?e=1597710457&h=740fd3b633530d5facb3007fd6aa0710","seek":"ms","cdn":"limelight","id":"94944611","bumper_duration":0}},"mpd":{"http":{"url":"https://manifest-viki.viki.io/v1/91001v/limelight/domain_4/mpd/normal/viki/high/mpdhd_mob/non_kcp/na/manifest.mpd?app=100005a&tag=mpdhd_mob:high:non_kcp:sourceVIKI:mpd:cached&h_timestamp=1597706857&u=14239459u&h_request_id=2a201040ac65cb32afb460f6124e6949/14762986993352435333&h_country=ph&h_host=hyperion-5f8cb47669-966jq&h_data_center=sng&sig=36f973f796be6b57a126b9caa95a7bc7fb22d78f","cdn":"limelight","drms":[],"id":"94944622","bumper_duration":0}}}

Here's the fixed from my c# code so that this can also applied to youtube-dl

API_URL = "https://api.viki.io/v4/";
APP_5 = "app=100005a";
APP_VERSION = "2.2.5.1428709186";
FULL API: https://api.viki.io/v4/videos/91001v/streams.json?app=100005a&t=1597707072&site=www.viki.com&token=A4maekf_jy2svG9sDQkkD8wL_2u0014239459uti00k8hnvb005_djAy&sig=7bbb262715b84907d18afe6bff23b69ce665e18b

I also included the "x-viki-app-ver" and "x-viki-as-id" to my login request headers
because if I will not include them, I'm getting the error. This is the solution by @tarhan from this issue: #25374

[blackjack4494]

@zackmark29
maybe you could show your c# code. Alternatively if you don't want to show it publicly you can send me an email (check my profile) or invite me to your private repo. I am just curious how it looks like.

I am just super confused and upset that for some SUPER ODD reason windows version works fine but as soon as I try on various ubuntu (also raspberry pi) it just won't work and I have no clue at all...

[zackmark29]

@zackmark29
maybe you could show your c# code. Alternatively if you don't want to show it publicly you can send me an email (check my profile) or invite me to your private repo. I am just curious how it looks like.

I am just super confused and upset that for some SUPER ODD reason windows version works fine but as soon as I try on various ubuntu (also raspberry pi) it just won't work and I have no clue at all...

Sure no problem. Here's the code for httpclient login request

` public static HttpClient ApiClient { get; set; }
public static HttpClientHandler ClientHandler { get; set; }

    public static void ApiRequest()
    {
        ClientHandler = new HttpClientHandler()
        {
            UseProxy = PROXY.IsUseProxy ? true : false, 
            Proxy = PROXY.IsUseProxy ? new WebProxy($"{PROXY.ProxyHost}:{PROXY.ProxyPort}") : null,
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
            ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }
        };
        {
            ApiClient = new HttpClient(ClientHandler);
            AddAllHeaders(ApiClient);
            ApiClient.DefaultRequestHeaders.Add("x-viki-app-ver", "4.0.53"); 
            ApiClient.DefaultRequestHeaders.Add("x-viki-as-id", "100005a");
        }
    }

    public static void AddAllHeaders(HttpClient client)
    {;
        client.DefaultRequestHeaders.Accept.Clear();
        client.DefaultRequestHeaders.Add("Method", "GET");
        client.DefaultRequestHeaders.UserAgent.TryParseAdd(UA);
        client.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue { NoCache = true, NoStore = true, MustRevalidate = true };
        client.DefaultRequestHeaders.Add("pragma", "no-cache");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
        client.DefaultRequestHeaders.ConnectionClose = true;
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    }`

and here's the completed streams api:
https://api.viki.io/v4/videos/91001v/streams.json?app=100005a&t=1597707072&site=www.viki.com&token=A4maekf_jy2svG9sDQkkD8wL_2u0014239459uti00k8hnvb005_djAy&sig=7bbb262715b84907d18afe6bff23b69ce665e18b

Let me know if you need other code

[blackjack4494]

@zackmark29 thanks :)
it didn't help me directly but it satisfied my curiosity haha. Also try to drop x-viki-as-id if you experience any differences.
I made a PR #26372 which should finally fix the error for all platforms. Give it a try and @RobinD42 too if you want :)
Am glad about some response.

[zackmark29]

@zackmark29 thanks :)
it didn't help me directly but it satisfied my curiosity haha. Also try to drop x-viki-as-id if you experience any differences.
I made a PR #26372 which should finally fix the error for all platforms. Give it a try and @RobinD42 too if you want :)
Am glad about some response.

haha it worked without the Id. Thanks!
I'll try your youtube-dl fixed later.