Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Facebook's extracted mpd (MPEG-DASH manifest) is missing base url for live stream #11530

Closed
orsonteodoro opened this issue Dec 25, 2016 · 6 comments
Closed

Facebook's extracted mpd (MPEG-DASH manifest) is missing base url for live stream #11530

orsonteodoro opened this issue Dec 25, 2016 · 6 comments

Comments

@orsonteodoro
Copy link

@orsonteodoro orsonteodoro commented Dec 25, 2016 •
edited

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.12.22. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2016.12.22 also confirmed with latest commit or live ebuild.

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • [] Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

youtube-dl -v -o t.out "https://www.facebook.com/YesThatKevinSmith/videos/10153977324206930/"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-o', u't.out', u'https://www.facebook.com/YesThatKevinSmith/videos/10153977324206930/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.12-gentoo_no_offensive_sites (confirmed in latest live 9999 ebuild/commit also)
[debug] Python version 2.7.12 - Linux-4.4.26-gentoo-x86_64-AMD_Phenom-tm-_9850_Quad-Core_Processor-with-gentoo-2.3
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[facebook] 10153977324206930: Downloading webpage
[facebook] 10153977324206930: Downloading webpage
[debug] Invoking downloader on u'https://video.xx.fbcdn.net/hvideo-frc1/v/dash-abr1/10153977324231930.mpd?_nc_rl=AfDeJiUm0OgCdFjV&oh=bbe9ff443988d7923f64d4a3ca1e3638&oe=58614952'
[download] Destination: t.out
[download] 100% of 3.83KiB in 00:00

<?xml version="1.0"?>
<MPD
    type="dynamic"
    xmlns="urn:mpeg:dash:schema:mpd:2011"
    availabilityStartTime="2016-12-24T19:35:41-08:00"
    availabilityEndTime="2016-12-24T19:53:25-08:00"
    publishFrameTime="1021"
    minimumUpdatePeriod="PT1S"
    minBufferTime="PT1S"
    timeShiftBufferDepth="P00Y00M00DT00H00M10.01S"
    suggestedPresentationDelay="PT2S"
    profiles="urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011"
    xmlns:xsi="http://www.w3.org/2011/XMLSchema-instance"
    xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 DASH-MPD.xsd">
  <Period start="PT0S" id="dash">
    <AdaptationSet
        id="1"
        segmentAlignment="true">
      <Representation
          id="live-md_H264"
          mimeType="video/mp4"
          codecs="avc1.4d401e"
          width="360"
          height="640"
          FBQualityClass="sd"
          FBQualityLabel="360p"
          frameRate="29"
          sar="1:1"
          startWithSAP="1"
          bandwidth="500000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-md/10153977324231930-$Time$.m4v"
            initialization="../dash-md/10153977324231930-init.m4v">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
      <Representation
          id="live-ld_H264"
          mimeType="video/mp4"
          codecs="avc1.4d401e"
          width="240"
          height="426"
          FBQualityClass="sd"
          FBQualityLabel="240p"
          frameRate="29"
          sar="1:1"
          startWithSAP="1"
          bandwidth="250000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-ld/10153977324231930-$Time$.m4v"
            initialization="../dash-ld/10153977324231930-init.m4v">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet
        id="2"
        segmentAlignment="true">
      <AudioChannelConfiguration
          schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011"
          value="1"/>
      <Representation
          id="live-md_AAC"
          mimeType="audio/mp4"
          codecs="mp4a.40.2"
          audioSamplingRate="48000"
          startWithSAP="1"
          bandwidth="62000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-md/10153977324231930-$Time$.m4a"
            initialization="../dash-md/10153977324231930-init.m4a">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Description of your issue, suggested solution and other information

The MPD extracted or produced is not adding enough information to the manifest for media players to parse in non-verbose mode. It provides the relative location "../" for media and initialization but not the server/domain or the necessary directories.

There needs to be post-processing of the extracted documents before they are sent out the pipe (to vlc or mplayer) as stdin to another player or dumped to a file. After it downloads the extracted mpd file, it should use an XML editor or module to inject the base URL <BaseURL> node with server address into the MPD manifest exactly under the <MPD> node.

The affected code is in youtube_dl/extractors/facebook.py

Examples of MPD manifest:
https://youtu.be/447-60cdu2s?t=1m27s
https://developer.mozilla.org/en-US/Apps/Fundamentals/Audio_and_video_delivery/Setting_up_adaptive_streaming_media_sources

It should look like:

<?xml version="1.0"?>
<MPD
    type="dynamic"
    xmlns="urn:mpeg:dash:schema:mpd:2011"
    availabilityStartTime="2016-12-24T19:35:41-08:00"
    availabilityEndTime="2016-12-24T19:53:25-08:00"
    publishFrameTime="1021"
    minimumUpdatePeriod="PT1S"
    minBufferTime="PT1S"
    timeShiftBufferDepth="P00Y00M00DT00H00M10.01S"
    suggestedPresentationDelay="PT2S"
    profiles="urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011"
    xmlns:xsi="http://www.w3.org/2011/XMLSchema-instance"
    xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 DASH-MPD.xsd">
  <BaseURL>https://video.xx.fbcdn.net/hvideo-frc1/v</BaseURL>
  <Period start="PT0S" id="dash">
    <AdaptationSet
        id="1"
        segmentAlignment="true">
      <Representation
          id="live-md_H264"
          mimeType="video/mp4"
          codecs="avc1.4d401e"
          width="360"
          height="640"
          FBQualityClass="sd"
          FBQualityLabel="360p"
          frameRate="29"
          sar="1:1"
          startWithSAP="1"
          bandwidth="500000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-md/10153977324231930-$Time$.m4v"
            initialization="../dash-md/10153977324231930-init.m4v">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
      <Representation
          id="live-ld_H264"
          mimeType="video/mp4"
          codecs="avc1.4d401e"
          width="240"
          height="426"
          FBQualityClass="sd"
          FBQualityLabel="240p"
          frameRate="29"
          sar="1:1"
          startWithSAP="1"
          bandwidth="250000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-ld/10153977324231930-$Time$.m4v"
            initialization="../dash-ld/10153977324231930-init.m4v">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
    <AdaptationSet
        id="2"
        segmentAlignment="true">
      <AudioChannelConfiguration
          schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011"
          value="1"/>
      <Representation
          id="live-md_AAC"
          mimeType="audio/mp4"
          codecs="mp4a.40.2"
          audioSamplingRate="48000"
          startWithSAP="1"
          bandwidth="62000">
        <SegmentTemplate
            presentationTimeOffset="0"
            timescale="1000"
            media="../dash-md/10153977324231930-$Time$.m4a"
            initialization="../dash-md/10153977324231930-init.m4a">
          <SegmentTimeline>
             <S t="909942" d="1001"/>
             <S t="910943" d="1001"/>
             <S t="911944" d="1001"/>
             <S t="912945" d="1001"/>
             <S t="913946" d="1001"/>
             <S t="914947" d="1001"/>
             <S t="915948" d="1001"/>
             <S t="916949" d="1001"/>
             <S t="917950" d="1001"/>
             <S t="918951" d="1001"/>
          </SegmentTimeline>
        </SegmentTemplate>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 25, 2016

There needs to be post-processing of the extracted documents before they are sent out the pipe (to vlc or mplayer)

Nope. Instead you should just pass the MPD URL to players instead of the document.
@yan12125 yan12125 closed this Dec 25, 2016
@orsonteodoro
Copy link
Author

@orsonteodoro orsonteodoro commented Dec 25, 2016

The problem is that the mpd url is not disclosed. It is known by youtube-dl internally but not by the user and the mpd file and the media player. The user sends the video url but not the MPD url. People are not reading.
@orsonteodoro
Copy link
Author

@orsonteodoro orsonteodoro commented Dec 25, 2016

I provide this: https://www.facebook.com/YesThatKevinSmith/videos/10153977324206930/

youtube-dl exposes only in debug mode not production code: https://video.xx.fbcdn.net/hvideo-frc1/v/dash-abr1/10153977324231930.mpd?_nc_rl=AfDeJiUm0OgCdFjV&oh=bbe9ff443988d7923f64d4a3ca1e3638&oe=58614952
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 25, 2016

The problem is that the mpd url is not disclosed

You can use -g:

MP4Client $(youtube-dl -g "https://www.facebook.com/appledaily.tw/videos/10155432762962069/")
@orsonteodoro
Copy link
Author

@orsonteodoro orsonteodoro commented Dec 25, 2016

Okay, thanks for clarifying. But, it should be in the FAQ or man page.
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Dec 25, 2016

It's indeed on the man page:

-g, --get-url                    Simulate, quiet but print URL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@orsonteodoro @yan12125
You can’t perform that action at this time.