Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CBC works nicely with new login stuff, but somehow only grabs every other episode? #24493

Open
mshiels opened this issue Mar 27, 2020 · 7 comments
Open

Comments

@mshiels
Copy link

@mshiels mshiels commented Mar 27, 2020

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dl version 2020.03.24
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-i', '-w', '--geo-bypass', '--hls-prefer-nati
ve', '--hls-use-mpegts', '--username', 'PRIVATE', '--password', 'PRIVATE', '--do
wnload-archive', 'DownloadedCBC.txt', '--cookies', 'GetForCBCCookies.txt', '--ou
tput', 'Y:\\Diggstown\\Season %(season_number)d\\Diggstown - s%(season_number)03
de%(episode_number)03d - %(title)s.mp4', 'https://watch.cbc.ca/season/diggstown/
season-2/01a585a1-2c7d-47a3-9562-7186aac979f1']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.03.24
[debug] Git HEAD: 049c0486b
[debug] Python version 3.7.4 (CPython) - Windows-2012ServerR2-6.3.9600-SP0
[debug] exe versions: ffmpeg N-92396-g55e021f39b, ffprobe N-92396-g55e021f39b
[debug] Proxy map: {}
[debug] Using fake IP 99.242.112.231 (CA) as X-Forwarded-For.
[cbc.ca:watch] 01a585a1-2c7d-47a3-9562-7186aac979f1: Downloading XML
[cbc.ca:watch] 01a585a1-2c7d-47a3-9562-7186aac979f1: Downloading XML
[download] Downloading playlist: Season 2
[cbc.ca:watch] playlist Season 2: Collected 5 video ids (downloading 5 of them)
[download] Downloading video 1 of 5
[debug] Using fake IP 99.238.119.123 (CA) as X-Forwarded-For.
[cbc.ca:watch:video] ec751a3d-eefd-412a-8c42-9b95b54b0c1f: Downloading XML
[cbc.ca:watch:video] ec751a3d-eefd-412a-8c42-9b95b54b0c1f: Downloading XML
[cbc.ca:watch:video] ec751a3d-eefd-412a-8c42-9b95b54b0c1f: Downloading m3u8 info
rmation
[debug] Default format spec: bestvideo+bestaudio/best
[download] Trailer has already been recorded in archive
[download] Downloading video 2 of 5
[download] Vince Hu has already been recorded in archive
[download] Downloading video 3 of 5
[cbc.ca:watch:video] 25cd2ed5-7bd0-4b2e-8bf5-cf55884f7c85: Downloading XML
[cbc.ca:watch:video] 25cd2ed5-7bd0-4b2e-8bf5-cf55884f7c85: Downloading XML
[cbc.ca:watch:video] 25cd2ed5-7bd0-4b2e-8bf5-cf55884f7c85: Downloading m3u8 info
rmation
[debug] Default format spec: bestvideo+bestaudio/best
[download] Cheryl Syllilboy has already been recorded in archive
[download] Downloading video 4 of 5
[download] Willy MacIsaac Redux has already been recorded in archive
[download] Downloading video 5 of 5
[cbc.ca:watch:video] fc1e7a90-0d23-4e15-9abd-c6dbe8317bbe: Downloading XML
[cbc.ca:watch:video] fc1e7a90-0d23-4e15-9abd-c6dbe8317bbe: Downloading XML
[cbc.ca:watch:video] fc1e7a90-0d23-4e15-9abd-c6dbe8317bbe: Downloading m3u8 info
rmation
[debug] Default format spec: bestvideo+bestaudio/best
[download] Tanya Ferreira has already been recorded in archive
[download] Finished downloading playlist: Season 2

Description

NOW for sure I have already downloaded the 0/2/4 episodes, so it's not trying again, but 1, 3 didn't download at all.

Well not quite sure how to explain this but I think you can tell from the above dump, that's it doing some sort of weird matching where it only really attempts to download every second episode. So in this case the trailer, then 2, then 4'. It's weird and I tried to look at it, but was unable to figure it out. At least your new CBC stuff works, while my older version has become broken now. But in my older version I somehow did fix this but being brain dead can't seem to figure out where I had fixed the issue.

@ilike2burnthing
Copy link

@ilike2burnthing ilike2burnthing commented Mar 27, 2020

Tried removing '--download-archive', 'DownloadedCBC.txt'?

@mshiels
Copy link
Author

@mshiels mshiels commented Mar 27, 2020

@ilike2burnthing
Copy link

@ilike2burnthing ilike2burnthing commented Mar 27, 2020

Single episode download successful using:
youtube-dl.exe --newline -i -o "D:\Downloads\%(title)s.%(ext)s" --ignore-config --hls-prefer-native --add-metadata -u PRIVATE -p PRIVATE -v "https://gem.cbc.ca/media/diggstown/season-2/episode-1/38e815a-0123eac1556"

Playlist download successful using:
youtube-dl.exe --newline -i -v -o "D:\Downloads\%(title)s.%(ext)s" --ignore-config --hls-prefer-native --add-metadata "https://gem.cbc.ca/season/diggstown/season-2/01a585a1-2c7d-47a3-9562-7186aac979f1"
(these particular videos don't require an account, so I removed the login details this time)

On my first attempt the videos all started to download but then eventually failed. Only the audio was complete. The second time round they were fine, so I'm putting it down to a network issue on my end.

Try that second one, obviously replacing the output location, but use a different folder than you used before.

If that doesn't work, try just this to rule out any other complications:
youtube-dl.exe -v -o "[YOUR OUTPUT LOCATION HERE]" "https://gem.cbc.ca/season/diggstown/season-2/01a585a1-2c7d-47a3-9562-7186aac979f1"

If you're still having no luck, give the author a ping - #19160

@mshiels
Copy link
Author

@mshiels mshiels commented Mar 29, 2020

Well after diffing my code more closely I noticed a change and after a lot of debug print statements I think I figured it out. If you enable the saving of the downloaded IDs, the code is actually mucking around internally and you end up with iD of the second video being saved after the 1st video is downloaded. Turn the option off and of course you get all files then, but no record of them being downloaded.

Tracked down to the following piece of code - literally screwing things up

rss = xpath_element(result,'rss')
if rss:
info.update(self._parse_rss_feed(rss)['entries')[0[)

it basically is somehow ending up with the information for the next video and stomping all over the valid information. I just put a

rss = None

before the check and things go back to properly downloading.

I assume something is wrong with the api - but I could not pin down what it was.

BUT FOR SURE a nasty bug!

@mshiels
Copy link
Author

@mshiels mshiels commented Mar 29, 2020

And more information if it helps.

Before the rss/update stuff the info looks something like this for say downloading Wild Bill using the following URL ...

INFO DUMP
{
'id': 'cf064132-cca5-446e-b1bb-a05f76ca3ab6',
'title': 'cf064132-cca5-446e-b1bb-a05f76ca3ab6',
'formats': [
{
'format_id': 'AACHEv1-2_0-English',
'url': 'https://cbcvott-a.akamaihd.net/p/38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/segments/CBC_WILD_BILL_S01E00_a5/prog_index.m3u8',
'manifest_url': 'https://cbcvott-a.akamaihd.net/p//38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/CBC_WILD_BILL_S01E00-v2-25549487.m3u8?cljwt.eyJhbGciOiJIUzI1NiIsImV4cCI6MTU4ODA0NzQ5Njg4NSwidHlwIjoiSldUIiwiaWF0IjoxNTg1NDU1MjUyOTM0LCJ2ZXJzaW9uIjoiVjEifQ.eyJjbGp3dC5wYXlsb2FkTWFwIjoick8wQUJYVnlBQUpiUXF6ekYvZ0dDRlRnQWdBQWVIQUFBQU9nN2lCMnVGSUJadHR4YWxPTjRMYVBLOHNiS0hWc1BaUEV2eGlBNEtLaHhzd1lJdDhDc1lTUU9WWnpJd1Uxd3ZVWDdtNDVzTnZvclpvM2pJeEE4anhhU0FZWEFvQVBXUkpQOU1xT3hNQUlFMkpsbnRGaWNjREtSTkVJVklzcGswVmpGMDJUVXNKMU05Ukp4NHlDT1VNVklhRjFzWGU0VDE0dVljaGZLMjllejNQaTBHU3Nrb0VkanZkN3pUVU9uVFRRQTlnOUYwcHB6SjNhVFBhTUU2YW5Id3VCMnMvaGtBdFBucVgrZzdUVzRrRFJkQ010OEIvQVNzQ2haUFE4TE5Jeld5RTAwZTFldnEvZmJGMU1lczJqTUhpMFl5SmRZeERlQzhFbW9FZXZUclVFZ0dzZ2o3NmRhR1JuZm1tdW1YRlhqU2VvenhMdWdWL0VXQWhibXFiKzA0OW5lWWt1Wk41bU1kWVdEbi9HcEhDbTBnOEw5NGhhT1M0VkIxcW91bCs4OFJ1SzAzZWs0UkxUeDhENnJXMGtOdERxSVQ0WDNrSFRHd1RPVnR4R1NoTmhsaTFNUXlsQkNxR3FVNXY0NVFOc3ZOUE1ZYXI0ZXpnb3ZPcnhicEF0TXdPY3VobDA3MVhueU5hRlFZREN3VzNaSjZCWDV0bUtqWXhiTVdPeUNOSXFtdEJPaXNtYUtPaFFoRThKa005b0hjN0Q4eVQ5bHNwNjQ2blFFaFZiblBycndHYVAxbDIydGk2YlhKZDY1MkNwVmtEbmVETFRYOWlRbG5yKzVvTHRhZmRVSEpoRDMvdFkxcFBlbFFWek0rc0kzMTY4M3NRS3ZsTWJJR1paMHZFNFZwUVJQRG1rRUpYbFhLQzFOaHRjbWFxRDhubnhmcStkQW9KYTJJbVNPclJHNkdUYnlqSWNtV1dvTkdWSnk4Q2kreEErbXIvOHhudW1oWWFLTHIzK3FaOWhRMk8xTEtRaEEyalZjaGFyUkRWR1VmUVFkRzQ3a2N5NzVybmI3VVduMEZKbkhzWFkvU3dWbytMTWo0TlpiWmkyNmJKZW0rd0ZBZGFKQ2x1ZHFJa2NDblpUVkxDTUU2K2NPTjNCbElVd2ZWL0pVR1ovWDBaTk94aXkreFE5b1JFVmNPOFBLMmc2a3pkSksxemZGcEZXKzJYekRFWFJuaHo5Nk1lOEpqbWhOMGdWMUtUckNQeWtTZ3FqaThob09WVmJOQ1VpZjNMM0lkdTJkdGl2M1habEVCWXVOK3gzZTIrVmN5U1VPQ0xOVVFHTDlIMjlyeVdiYkJjR2RsZ1JkUThSa3ZiQ2VKZFVyRG1JMkgyOGNHRCtCYS9CZjNLWVRTLzhQNlFldmFZMitrbTdrZkhnMFJSMjBtMVM4YnJJNXd0VmV4QkZ4VWZSenZmL1RzeUl6d0N0NG00VVRBSWdjTitqQUszYWM1T0REWktvRTVBcG9zc01oL1Yxb2dKb3NIaEJkcnM4emhkRkxMS3plb01OTlN2bXcxZGtTVjNpMVJKVVJ2WjNkbUFjZlFkRUNlY09yNXBZQXVMM0FmaUhXTC92Ny90bmpoQ2ZuY01IRHZWdVRJdFpEcm5LcnMzN2ZzenlPMGNEa3dwSFJkQlBTay9ualNRRFdZS0xOV0RWU3c9PSIsImlzcyI6ImNsZWFycGxheSJ9.HySaF23Xrh8_6t8TfuYC3xLqiw6BtEG8xyjvU46PiHI',
'language': 'eng',
'ext': 'mp4',
'protocol': 'm3u8',
'preference': None,
'vcodec': 'none',
'acodec': 'aac'
},
{
'format_id': 'AACLC-2_0-English',
'url': 'https://cbcvott-a.akamaihd.net/p/38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/segments/CBC_WILD_BILL_S01E00_a7/prog_index.m3u8',
'manifest_url': 'https://cbcvott-a.akamaihd.net/p//38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/CBC_WILD_BILL_S01E00-v2-25549487.m3u8?

AND you can see it's referring to the S01E00 trailer which is fine

AFTER THE INFO UPDATE

{'id': '38e815a-01242059898', 'title': 'Episode 1', 'formats': [{'format_id': 'AACHEv1-2_0-English', 'url': 'https://cbcvott-a.akamaihd.net/p/38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/segments/CBC_WILD_BILL_S01E00_a5/prog_index.m3u8', 'manifest_url': 'https://cbcvott-a.akamaihd.net/p//38e815a-01253ca9881/CBC_WILD_BILL_S01E00-v2-25549487/CBC_WILD_BILL_S01E00-v2-25549487.m3u8?

Now the update has turned the 'id' into literally the identifier for the 's01e01' episode. I know that since I printed the XML output from the CBC API and had passed it into an XML viewer. But I don't have a copy of it saved sorry to say.

So once you finish the first one it will now use that id to save away into the downloaded record, and kaboom - the next video you want to download will get skipped. So you up with this every other episode situation.

@mshiels
Copy link
Author

@mshiels mshiels commented Mar 29, 2020

And some more information since I did find the XML api output I had captured. Here are some snippets showing the ID that got replaced above, but with the 'next shows' ID

38e815a-01253ca9881 <title>Trailer</title>

and the second shows snippet

38e815a-01242059898 <title>Episode 1</title>

Still unsure since it's been a while since I explored that API - I used to have a good understanding of it back in 2018 when I implemented a basic auth for cbc for my stuff. and I swear I had a way to just browse the XML infomation but alas I didn't keep good notes it seems.

@ilike2burnthing
Copy link

@ilike2burnthing ilike2burnthing commented Mar 29, 2020

Ping @meunierd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.