[video.fc2.com] ERROR: Unable to extract OpenGraph title

[LS12tthappy]

Checklist

• [ √] I'm reporting a broken site support
• [√ ] I've verified that I'm running youtube-dl version 2019.08.13
• [√ ] I've checked that all provided URLs are alive and playable in a browser
• [ √] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [√ ] I've searched the bugtracker for similar issues including closed ones

Verbose log

OS : Ubuntu 18.04
Python --version : Python 2.7.15+
Internet : [ OK√ ]
Test URL: https://video.fc2.com/content/200911145aCpHnxm

• [√ ] I've checked that all provided URLs are alive and playable in a browser

root@ubuntu:~/youtube-dl# youtube-dl -v 'http://video.fc2.com/content/200911145aCpHnxm'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://video.fc2.com/content/200911145aCpHnxm']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.08.13
[debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-55-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 4.0.3-static, ffprobe 4.0.3-static
[debug] Proxy map: {}
[fc2] 200911145aCpHnxm: Downloading webpage
ERROR: Unable to extract OpenGraph title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/fc2.py", line 89, in _real_extract
    title = self._og_search_title(webpage)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 1117, in _og_search_title
    return self._og_search_property('title', html, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 1105, in _og_search_property
    escaped = self._search_regex(og_regexes, html, name, flags=re.DOTALL, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 1005, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract OpenGraph title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

• [√ ] I've verified that I'm running youtube-dl version 2019.08.13

root@ubuntu:~/youtube-dl# youtube-dl --version
2019.08.13

• [√ ] I've checked that all provided URLs are alive and playable in a browser

Test URL:  https://video.fc2.com/content/200911145aCpHnxm

Last week I was able to download the video on video.fc2.com. But today it seem like doesn't work (anymore), giving me the OpenGraph-Error.
Thanks for reading my issue!

[kuchenmitsahne]

I looked into this a bit more and it seems that FC2 has made how they serve up their videos slightly more complex. What I observed is that, once you hit the play button, a GET request is made to

https://video.fc2.com/api/v3/videoplaylist/{video_id}sh=1&fs=0

which returns structured data of this form:

{
  "type": 1,
  "playlist": {
    "nq": "/api/v3/videoplay/{some_id}/2?signature={some_signature}&t={some_t}"
  },
  "playstatus": "/api/v3/videoplaystatus/{some_id}?signature={some_signature}",
  "playtime": "https://api.video.fc2.com/api/v3/videoplaytime/{some_id}?signature={some_signature}",
  "service": null,
  "session": {
    "id": "{some_id}",
    "nq": "{some_nq}"
  }
}

some_id, some_signature, some_t, and some_nq are all generated on a per-request basis. some_t seems to just be the Unix epoch of whenever the response is sent (or request is made?).

From there, another GET request is made using the above information plus a bit extra:

https://video.fc2.com/api/v3/videoplay/{some_id}/2?signature={some_signature}&t={some_t}&{some_nq}=0&pf=Win32&lg=en-US&referrer={actual_referrer_url}

The referrer can be left blank, and I suppose pf and lg will change depending on your system's region settings, although I don't know what effect (if any) it will have on the video you're served.

The response to this GET request will be 302 Found, and the Location will be of the form

https://vip-videorare1000.fc2.com/up/flv/{first_six_chars_of_video_id}/{next_2_chars}/{next_char}/{video_id}.free.mp4?mid={some_mid}

where some_mid is the generated authentication token. From here, the video can be downloaded without any issues.

Actual example using the above video:

curl -X GET 'https://video.fc2.com/api/v3/videoplaylist/200911145aCpHnxm?sh=1&fs=0'

Response:

{
  "type": 1,
  "playlist": {
    "nq": "/api/v3/videoplay/vSgvvlKll9HTHlSTozRTRWz9a9P9vvSPgQ8AnxFk/2?signature=3N6FQ63WQNVWSI40KTY3LCH.T_N8VITMVNR$HIVXWDGBM-2WRN&t=1565818254"
  },
  "playstatus": "/api/v3/videoplaystatus/vSgvvlKll9HTHlSTozRTRWz9a9P9vvSPgQ8AnxFk?signature=3N6FQ63WQNVWSI40KTY3LCH.T_N8VITMVNR$HIVXWDGBM-2WRN",
  "playtime": "https://api.video.fc2.com/api/v3/videoplaytime/vSgvvlKll9HTHlSTozRTRWz9a9P9vvSPgQ8AnxFk?signature=3N6FQ63WQNVWSI40KTY3LCH.T_N8VITMVNR$HIVXWDGBM-2WRN",
  "service": null,
  "session": {
    "id": "vSgvvlKll9HTHlSTozRTRWz9a9P9vvSPgQ8AnxFk",
    "nq": "UyGI4D"
  }
}

curl -sIX GET 'https://video.fc2.com/api/v3/videoplay/BAA101_Z10LwLADmLiqAfsw_aZ1YAZi1DUxb1cIo/2?signature=1KT8JR-4IA01_4255.PVC8BYNMRQSI5GS&t=1565818706&eTVaF8=0&pf=Win32&lg=en-US&referrer='

Response Headers:

HTTP/1.1 302 Found
Server: nginx
Date: Wed, 14 Aug 2019 21:39:24 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
Set-Cookie: PHPSESSID=77ceb9ec59b1cd585b6b3fd47f6c28f4; path=/; domain=.fc2.com; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: https://vip-videorare1000.fc2.com/up/flv/200911/14/5/200911145aCpHnxm.free.mp4?mid=56529eed565a625c7a9c314cd2b97021
X-Whom: 1011
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

Judging by those type and service keys, there's probably some other cases to consider, but this should work for the standard free videos.

[LS12tthappy]

Thank you a lot for detailed explanation :)

[JasoonS]

This may help some people:

videoId=<<PUT THE VIDEO URL HERE>>
params=($(curl -X GET "https://video.fc2.com/api/v3/videoplaylist/${videoUrl}?sh=1&fs=0" | jq --raw-output '.playlist.nq,.session.nq'))
youtube-dl "https://video.fc2.com${params[1]}&${params[2]}=0&pf=Win32&lg=en-US&referer="  

Copy this full example in your terminal to test this:

videoUrl=200911145aCpHnxm
params=($(curl -X GET "https://video.fc2.com/api/v3/videoplaylist/${videoUrl}?sh=1&fs=0" | jq --raw-output '.playlist.nq,.session.nq'))
youtube-dl "https://video.fc2.com${params[1]}&${params[2]}=0&pf=Win32&lg=en-US&referer=" 

The videoId is the very end section of the URL of the video you want to download.

Also grab jq here if you don't have it installed: https://stedolan.github.io/jq/

[kuchenmitsahne]

Thanks, this worked like a charm! The only thing I had to change to get it to work on my machine was changing the indices of params to 0 and 1 since I'm using bash.

Also note that in your first code block, you set a variable called videoId but the variable used in the next line is videoUrl. I'm guessing you intended to use videoId throughout.