404 error with tweet video #32007

jcsouz10 · 2023-04-07T21:39:44Z

Checklist

I'm reporting a broken site support issue
I've verified that I'm running youtube-dl version 2021.12.17
I've checked that all provided URLs are alive and playable in a browser
I've checked that all URLs and arguments with special characters are properly quoted or escaped
I've searched the bugtracker for similar bug reports including closed ones
I've read bugs section in FAQ

Verbose log

 ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

I'm trying to download metadata from tweets video and i'm getting 404 error. But yesterday was working very well

The text was updated successfully, but these errors were encountered:

dirkf · 2023-04-07T22:42:29Z

Provide a verbose log. Read the FAQ section BUGS in the Manual if you're not sure how to generate one. Also, review other active issues for examples.

dirkf · 2023-04-07T22:45:00Z

Also, very often 404, even for some API URL rather than the page itself, is the site's way of telling us that the content is no longer available, so do check this again:

[x] I've checked that all provided URLs are alive and playable in a browser

PhobosK · 2023-04-08T10:35:06Z

I can confirm this problem...

But it seems it affects downloading of videos from accounts with explicit content. No matter if the video is explicit by itself, or not on per account bases no videos can be downloaded.
Using the youtube-dl -u command line option and password, do not make any difference.
Videos from other accounts that are not explicit can be downloaded without a problem.

Not sure if the debug log helps but here it is. BTW I chose some not so explicit video (it is an animation) from an explicit account and since not sure how things with such links should be treated/reported here, I masked the link in the log and will post under it the actual user and status, so you can construct the URL:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://twitter.com/XXXXXX/status/XXXXXXXXXXX']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.10 (CPython) - Linux-6.2.9-1-default-x86_64-with-glibc2.37
[debug] exe versions: avconv 12.3, avprobe 12.3, ffmpeg 4.4.3, ffprobe 4.4.3
[debug] Proxy map: {}
[twitter] XXXXXXXXXXX: Downloading guest token
[twitter] XXXXXXXXXXX: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib64/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

The user is MommyLinaaa and the video is in status: 1644617705968791552

jcsouz10 · 2023-04-08T15:44:12Z

Provide a verbose log. Read the FAQ section BUGS in the Manual if you're not sure how to generate one. Also, review other active issues for examples.

Sorry for delay to answer..

My verbose log is exactly the same of Phobosk

PhobosK · 2023-04-08T16:23:36Z

@jcsouz10 - Nope. No solution or workaround...
I am not a developer here :)
I guess @dirkf will chek this and help fixing it... now as he got more specific info on the issue :)

dirkf · 2023-04-08T23:58:56Z

Possibly Twitter is testing cookies against the UA or (worse) TLS fingerprint that was seen when the cookie was issued?

dirkf · 2023-04-09T10:52:53Z

I also received these comments:

setting a cookie with the --cookies cookies.txt option for an account with access to the content doesn't seem to help either.

cookies.txt:
twitter.com FALSE / TRUE 1696528758 auth_token [auth_token_value]
twitter.com FALSE / TRUE 1696528726 guest_id [guest_id_value]

And:

On line youtube_dl/YoutubeDL.py#2303

youtube-dl/youtube_dl/YoutubeDL.py

Line 2303 in 213d1d9

url = req.get_full_url()

|url = req.get_full_url()| is returning
|'https://api.twitter.com/1.1/guest/activate.json'| rather than the original
request url |https://twitter.com/MommyLinaaa/status/1644617705968791552|

In incognito, it seems the post requires age-verification:

As mentioned, i also get 'https://api.twitter.com/1.1/guest/activate.json' with
--cookies cookies.txt option set using an |auth_token| that logs me in either
set manually from the dev console or imported using the cookies.txt file
directly with the EditThisCookie extension (i.e. - I can see the post and not
the age-verification page). However, for some reason it doesn't get past the
age-verification though when using the --cookies cookies.txt option in youtube-dl.

This explains that the problem occurs when yt-dl fetches a page that has some content flag and the page request redirects to some user verification page (age verification?). There is also some trickery where the authorisation token from that page may need some other headers set to be valid.

dirkf · 2023-04-09T11:43:21Z

Probably related: yt-dlp/yt-dlp#5998

dirkf · 2023-04-10T21:19:39Z

Actually related: yt-dlp/yt-dlp#6763

PhobosK · 2023-04-12T04:02:43Z

Use option
--cookies-from-browser $BROWSER_NAME
should fix it

@jasonm23 ,
you are talking about the youtube-dl fork yt-dlp
There is no such option in the currently available stable version of youtube-dl 2021.12.17
So I do not see how this is relevant here !

BTW @dirkf , just as an additional info debugging - since my account has 2FA enabled, I tried with it also, but the result is the same:

[debug] Command-line args: ['-v', '-u', 'PRIVATE', '-2', 'XXXXXX', 'https://twitter.com/XXXXXXXXXXX/status/XXXXX']
...............
...............
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Gokujo · 2023-04-12T12:28:53Z

I tried with cookies but I still get 404 Error

[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.6 (CPython) - Linux-6.2.6-76060206-generic-x86_64-with-glibc2.35
[debug] exe versions: ffmpeg 4.4.2, ffprobe 4.4.2
[debug] Proxy map: {}
[twitter] 1614746772974501896: Downloading guest token
[twitter] 1614746772974501896: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/maximharder/Development/Private/Python/env/botPosterEnv310/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/maximharder/Development/Private/Python/env/botPosterEnv310/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

options that I am giving to downloader:

				ydl_opts = {
					'format': 'bestaudio/best',
					'outtmpl': file_path,
					'noplaylist': True,
					'cookies': TWITTER_COOKIES,
					'verbose': True,
					'continue': True,
					'dump-user-agent': True,
				}

maybe I am doing sth wrong?

dirkf added the clarification-needed label Apr 7, 2023

This comment was marked as resolved.

Sign in to view

dirkf added broken-IE problem with existing site extraction account-needed and removed clarification-needed labels Apr 9, 2023

This comment was marked as duplicate.

Sign in to view

dirkf removed the account-needed label Apr 9, 2023

This comment was marked as off-topic.

Sign in to view

dirkf mentioned this issue May 6, 2023

Twitter #32155

Closed

This comment was marked as off-topic.

Sign in to view

dirkf mentioned this issue May 23, 2023

spankbang 403 Forbidden error #32227

Closed

This comment was marked as duplicate.

Sign in to view

This comment was marked as resolved.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

404 error with tweet video #32007

404 error with tweet video #32007

jcsouz10 commented Apr 7, 2023 •

edited by dirkf

Loading

dirkf commented Apr 7, 2023

dirkf commented Apr 7, 2023

PhobosK commented Apr 8, 2023

jcsouz10 commented Apr 8, 2023

This comment was marked as resolved.

PhobosK commented Apr 8, 2023

dirkf commented Apr 8, 2023

dirkf commented Apr 9, 2023

This comment was marked as duplicate.

dirkf commented Apr 9, 2023

dirkf commented Apr 10, 2023

This comment was marked as off-topic.

PhobosK commented Apr 12, 2023

Gokujo commented Apr 12, 2023

This comment was marked as off-topic.

This comment was marked as duplicate.

This comment was marked as resolved.

404 error with tweet video #32007

404 error with tweet video #32007

Comments

jcsouz10 commented Apr 7, 2023 • edited by dirkf Loading

Checklist

Verbose log

Description

dirkf commented Apr 7, 2023

dirkf commented Apr 7, 2023

PhobosK commented Apr 8, 2023

jcsouz10 commented Apr 8, 2023

This comment was marked as resolved.

PhobosK commented Apr 8, 2023

dirkf commented Apr 8, 2023

dirkf commented Apr 9, 2023

This comment was marked as duplicate.

dirkf commented Apr 9, 2023

dirkf commented Apr 10, 2023

This comment was marked as off-topic.

PhobosK commented Apr 12, 2023

Gokujo commented Apr 12, 2023

This comment was marked as off-topic.

This comment was marked as duplicate.

This comment was marked as resolved.

jcsouz10 commented Apr 7, 2023 •

edited by dirkf

Loading