Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

youtube-dl run from an Automator workflow does not repair malformed AAC bitstream automatically as it does when run from the terminal #15115

Closed
rllguibord opened this issue Dec 29, 2017 · 16 comments
Labels

Comments

@rllguibord
Copy link

@rllguibord rllguibord commented Dec 29, 2017

  • [ X] I've verified and I assure that I'm running youtube-dl 2017.12.28

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

[debug] System config: []
[debug] User config: [u'-o', u'~/Movies/%(title)s.%(ext)s', u'-f', u'mp4']
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.smithsonianmag.com/videos/category/arts-culture/the-photography-of-eudora-welty/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.12.28
[debug] Python version 2.7.10 - Darwin-17.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1
[debug] Proxy map: {}
[generic] the-photography-of-eudora-welty: Requesting header
WARNING: Falling back on generic information extractor.
[generic] the-photography-of-eudora-welty: Downloading webpage
[generic] the-photography-of-eudora-welty: Extracting information
[generic] the-photography-of-eudora-welty: Downloading m3u8 information
[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a46e1df_0x7bca30aa4e4327c23b4fdc026d7e6ead038e35ae'
[download] /Users/rguibord/Movies/The Photography of Eudora Welty.mp4 has already been downloaded
[download] 100% of 15.19MiB
[debug] ffmpeg command line: ffprobe -show_streams 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "/Users/rguibord/Movies/The Photography of Eudora Welty.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4' -c copy -f mp4 '-bsf:a' aac_adtstoasc 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.temp.mp4'

As can be seen from the -v output above, youtube-dl run from the Terminal automatically debugs the downloaded video with ffmpeg and repairs the downloaded video so that it can be played with Quicktime. But when I run youtube-dl through a workflow that I can invoke through Launchbar, it downloads the file but does not detect the "malformed AAC bitstream" nor repair it. This means that I then need to run a separate ffmpeg command on the file to repair it. My Automator workflow looks like this:
Screenshot 2017-12-29 15.05.16.pdf

Why doesn’t youtube-dl detect the malformed bitstream automatically when run this way? and can anyone help me modify the workflow so that it will. Thank you, -rg

@rllguibord rllguibord changed the title youtube-dl run from an Automator workflow does not repair malformed AAC bitstream automatically as it does in the terminal youtube-dl run from an Automator workflow does not repair malformed AAC bitstream automatically as it does when run from the terminal Dec 29, 2017
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Dec 29, 2017

Post the complete youtube-dl log from automator.

@dstftw dstftw closed this Dec 29, 2017
@dstftw dstftw added the incomplete label Dec 29, 2017
@rllguibord
Copy link
Author

@rllguibord rllguibord commented Dec 30, 2017

The full log was visible in the screenshot I uploaded, but here it is:

2017-12-30 01:04:49 +0000: Get Current Webpage from Safari completed
2017-12-30 01:04:49 +0000: Set Value of Variable completed
2017-12-30 01:04:49 +0000: Conversion from URLs to Text completed
2017-12-30 01:04:53 +0000: Run Shell Script completed
2017-12-30 01:04:53 +0000: Workflow completed

Perhaps you could use this as well the results of running the script by adding -v to the shell script command. Here that is:
(
"[generic] the-photography-of-eudora-welty: Requesting header",
"[generic] the-photography-of-eudora-welty: Downloading webpage",
"[generic] the-photography-of-eudora-welty: Extracting information",
"[generic] the-photography-of-eudora-welty: Downloading m3u8 information",
"[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a47105a_0xeeceabc344ac5dbd719a57e93e9e7c97bac4222f'",
"[download] /Users/rguibord/Movies/The Photography of Eudora Welty.mp4 has already been downloaded",
"
[download] 100% of 15.19MiB"
)

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Dec 30, 2017

That's not a complete log. There is no output written to stderr.

@rllguibord
Copy link
Author

@rllguibord rllguibord commented Dec 30, 2017

Thank you for your reply. I see your point. Would anything be written to stderr if no error was detected. That is what I thought was happening here, that the error in the file was detected, and then corrected, when running youtube-dl in the Terminal, but not detected, perhaps not scanned for, when running youtube-dl as a shell script within Automator.

If I am wrong about this, how would I obtain a more complete log?
-rg

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Dec 31, 2017

Change the Automator workflow to add -v to the youtube-dl command line.

(Also, paste the text of the log output here in a comment, don't use a screenshot.)

@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 2, 2018

Hi John, thank you for your input. You are right that I did not show the use of -v in the Automator workflow script even though I had implemented it before my earlier posting the output. The workflow now looks like this:
#!/bin/bash
/usr/local/bin/youtube-dl -v $1

Unfortunately,the output looks the same as as in my previous post except that this time the full download takes place since I deleted the earlier downloaded file. Here is the full output, still without the [debug] lines that follow the completion of the download when I run the script in terminal.

(
"[generic] the-photography-of-eudora-welty: Requesting header",
"[generic] the-photography-of-eudora-welty: Downloading webpage",
"[generic] the-photography-of-eudora-welty: Extracting information",
"[generic] the-photography-of-eudora-welty: Downloading m3u8 information",
"[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a4969a7_0x4f7be8a1c981298fe7c429343f7ca33fc2a50041'",
"[hlsnative] Downloading m3u8 manifest",
"[hlsnative] Total fragments: 60",
"[download] Destination: /Users/rguibord/Movies/The Photography of Eudora Welty.mp4",
"
[download] 0.0% of ~5.70MiB at 29.07KiB/s ETA 14:38
[download] 0.1% of ~5.70MiB at 85.80KiB/s ETA 04:53
[download] 0.1% of ~5.70MiB at 194.45KiB/s ETA 02:06
[download] 0.3% of ~5.70MiB at 255.39KiB/s ETA 01:07
.............
[download] 99.9% of ~16.40MiB at 916.69KiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 1.00MiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 1.00MiB/s ETA 00:00
[download] 100% of 16.40MiB in 00:20"
)

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Jan 2, 2018

Looks like you're only getting stdout and not stderr. Try:

#!/bin/bash
/usr/local/bin/youtube-dl -v $1 2>&1
@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 2, 2018

Ah, thank you. Here is the output now. FYI, I do have FFMPEG installed, and it runs if I run youtube-dl from the terminal.

(
"[debug] System config: []",
"[debug] User config: [u'-o', u'~/Movies/%(title)s.%(ext)s', u'-f', u'mp4']",
"[debug] Custom config: []",
"[debug] Command-line args: [u'-v', u'https://www.smithsonianmag.com/arts-culture/eudora-welty-as-photographer-117044298/']",
"[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII",
"[debug] youtube-dl version 2017.12.28",
"[debug] Python version 2.7.10 - Darwin-17.3.0-x86_64-i386-64bit",
"[debug] exe versions: none",
"[debug] Proxy map: {}",
"[generic] eudora-welty-as-photographer-117044298: Requesting header",
"WARNING: Falling back on generic information extractor.",
"[generic] eudora-welty-as-photographer-117044298: Downloading webpage",
"[generic] eudora-welty-as-photographer-117044298: Extracting information",
"[generic] eudora-welty-as-photographer-117044298: Downloading m3u8 information",
"[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a4bf2d7_0x03518317715249b4eaeaa76ce4a2876f3ad9969e'",
"[hlsnative] Downloading m3u8 manifest",
"[hlsnative] Total fragments: 60",
"[download] Destination: /Users/rguibord/Movies/The Photography of Eudora Welty.mp4",
"
[download] 0.0% of ~5.70MiB at 101.26KiB/s ETA 39:27
[download] 0.1% of ~5.70MiB at 155.87KiB/s ETA 13:27
[download] 0.1% of ~5.70MiB at 285.32KiB/s ETA 05:50
[download] 0.3% of ~5.70MiB at 133.42KiB/s ETA 03:17
[download] 0.5% of ~5.70MiB at 184.62KiB/s ETA 01:45
..............................
[download] 99.9% of ~16.40MiB at 110.04KiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 97.32KiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 97.32KiB/s ETA 00:00
[download] 100% of 16.40MiB in 03:37",
"WARNING: eudora-welty-as-photographer-117044298: malformed AAC bitstream detected. Install ffmpeg or avconv to fix this automatically."
)

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Jan 2, 2018

This means ffmpeg is not your PATH as seen by Automator.
Open Terminal.app and check for the location of ffmpeg:

bash$ type -ap ffmpeg

If the answer is, say, /usr/local/bin, add that to your PATH in the script:

#!/bin/bash
export PATH=$PATH:/usr/local/bin
/usr/local/bin/youtube-dl -v $1 2>&1
@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 2, 2018

Oh my God! That is fantastic. Thank you! The output is below, and more importantly the mp4 file is fixed and playable.

Two questions, and I understand if you tell me merely that I need to study shell scripting more. If that is your recommendation, where might I start?

First, why is there a need to use the export Path command when youtube-dl was already being called in that directory?

Second, what does "2>&1" mean in this script?

Thank you again. Here is the output.

(
"[debug] System config: []",
"[debug] User config: [u'-o', u'~/Movies/%(title)s.%(ext)s', u'-f', u'mp4']",
"[debug] Custom config: []",
"[debug] Command-line args: [u'-v', u'https://www.smithsonianmag.com/arts-culture/eudora-welty-as-photographer-117044298/']",
"[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII",
"[debug] youtube-dl version 2017.12.28",
"[debug] Python version 2.7.10 - Darwin-17.3.0-x86_64-i386-64bit",
"[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1",
"[debug] Proxy map: {}",
"[generic] eudora-welty-as-photographer-117044298: Requesting header",
"WARNING: Falling back on generic information extractor.",
"[generic] eudora-welty-as-photographer-117044298: Downloading webpage",
"[generic] eudora-welty-as-photographer-117044298: Extracting information",
"[generic] eudora-welty-as-photographer-117044298: Downloading m3u8 information",
"[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a4c4267_0x3c78de1dd20e6d8282f0af18ced87cb520491a19'",
"[hlsnative] Downloading m3u8 manifest",
"[hlsnative] Total fragments: 60",
"[download] Destination: /Users/rguibord/Movies/The Photography of Eudora Welty.mp4",
"

[download] 0.0% of ~5.70MiB at 101.26KiB/s ETA 39:27
[download] 0.1% of ~5.70MiB at 155.87KiB/s ETA 13:27
[download] 0.1% of ~5.70MiB at 285.32KiB/s ETA 05:50
[download] 0.3% of ~5.70MiB at 133.42KiB/s ETA 03:17
[download] 0.5% of ~5.70MiB at 184.62KiB/s ETA 01:45
. . . . . . . . . . . . . . . .

[download] 99.9% of ~16.40MiB at 605.65KiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 753.12KiB/s ETA 00:00
[download] 100.0% of ~16.40MiB at 753.12KiB/s ETA 00:00
[download] 100% of 16.40MiB in 00:03",
"[debug] ffmpeg command line: ffprobe -show_streams 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4'",
"[ffmpeg] Fixing malformed AAC bitstream in "/Users/rguibord/Movies/The Photography of Eudora Welty.mp4"",
"[debug] ffmpeg command line: ffmpeg -y -i 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4' -c copy -f mp4 '-bsf:a' aac_adtstoasc 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.temp.mp4'"
)

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Jan 3, 2018

Two questions, and I understand if you tell me merely that I need to study shell scripting more. If that is your recommendation, where might I start?

Unfortunately much of this is really the weird interaction between Automator and shell scripting. I don't have a good answer for you. It's certainly not a youtube-dl bug and this is definitely the wrong place.

First, why is there a need to use the export Path command when youtube-dl was already being called in that directory?

The PATH defines where the script looks to find other programs. I don't know where you have youtube-dl installed and where you have ffmpeg installed, and you didn't say. But if, as you suggest, both ffmpeg and youtube-dl are installed in /usr/local/bin ("youtube-dl was already being called in that directory") then it is puzzling. Perhaps you have youtube-dl installed in multiple places? What does type -ap youtube-dl report in the Terminal, and for that matter, what did type -ap ffmpeg report? Edit: And what is PATH currently set to?: echo $PATH

Second, what does "2>&1" mean in this script?

It tells the shell to redirect the output of standard error (fd2) to standard output (fd1). This is, apparently, necessary because Automator captures the output of standard output but not standard error.

@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 13, 2018

John, Thank you again. My Automator service works now as desired.

I have youtube-dl and ffmpeg installed in the same directory, and both in only one directory. I am using Homebrew to manage both.
type -ap youtube-dl
/usr/local/bin/youtube-dl

type -ap ffmpeg
/usr/local/bin/ffmpeg

echo $PATH
/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Jan 14, 2018

Doesn't make much sense, but apparently something is different in the shell spawned by Automator. So I suppose you should, for diagnostic purposes only, replace the Automator script with:

#!/bin/bash
type -ap ffmpeg youtube-dl
export PATH=$PATH:/usr/local/bin
/usr/local/bin/youtube-dl -v $1 2>&1

and see what it reports. Or we could just go home.

@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 14, 2018

I am game to try it, and here is the output within Automator of the script you suggested above:

(
"[debug] System config: []",
"[debug] User config: [u'-o', u'~/Movies/%(title)s.%(ext)s', u'-f', u'mp4']",
"[debug] Custom config: []",
"[debug] Command-line args: [u'-v', u'https://www.smithsonianmag.com/videos/category/history/the-photography-of-eudora-welty/']",
"[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII",
"[debug] youtube-dl version 2017.12.28",
"[debug] Python version 2.7.10 - Darwin-17.3.0-x86_64-i386-64bit",
"[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1",
"[debug] Proxy map: {}",
"[generic] the-photography-of-eudora-welty: Requesting header",
"WARNING: Falling back on generic information extractor.",
"[generic] the-photography-of-eudora-welty: Downloading webpage",
"[generic] the-photography-of-eudora-welty: Extracting information",
"[generic] the-photography-of-eudora-welty: Downloading m3u8 information",
"[debug] Invoking downloader on u'https://videos-f.jwpsrv.com/content/conversions/adUEQeXN/videos/KE0LykmE-30499008.mp4.m3u8?token=0_5a5b9ffb_0xdc2ec90c8fac7689e95687ecf8ed79e52ae43320'",
"[hlsnative] Downloading m3u8 manifest",
"[hlsnative] Total fragments: 60",
"[download] Destination: /Users/rguibord/Movies/The Photography of Eudora Welty.mp4",
"
[download] 0.0% of ~5.70MiB at 21.70KiB/s ETA 15:43
. . . . . . . . . . . . . . . . . . . . .
[download] 100% of 16.40MiB in 00:15",
"[debug] ffmpeg command line: ffprobe -show_streams 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4'",
"[ffmpeg] Fixing malformed AAC bitstream in "/Users/rguibord/Movies/The Photography of Eudora Welty.mp4"",
"[debug] ffmpeg command line: ffmpeg -y -i 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.mp4' -c copy -f mp4 '-bsf:a' aac_adtstoasc 'file:/Users/rguibord/Movies/The Photography of Eudora Welty.temp.mp4'"
)

@johnhawkinson
Copy link
Contributor

@johnhawkinson johnhawkinson commented Jan 14, 2018

It should have reported two paths prior to the youtube-dl output that you pasted. Did you start copying at the very very top? Maybe add an echo $PATH in there. (Before where you set PATH). We don't the youtube-dl output.

@rllguibord
Copy link
Author

@rllguibord rllguibord commented Jan 14, 2018

I added the echo $PATH. Here is the whole script I am copying from Automator followed by the output.
I think you meant "We don't need the youtube-dl output." Homebrew put symlinks in /usr/local/bin/ and installs the applications in their own directories. Could that be causing the unexpected results you are seeing?

#!/bin/bash
type -ap ffmpeg youtube-dl
echo $PATH
export PATH=$PATH:/usr/local/bin
/usr/local/bin/youtube-dl -v $1 2>&1

(
"/usr/bin:/bin:/usr/sbin:/sbin",
"[debug] System config: []",
"[debug] User config: [u'-o', u'~/Movies/%(title)s.%(ext)s', u'-f', u'mp4']",
"[debug] Custom config: []",
"[debug] Command-line args: [u'-v', u'https://www.smithsonianmag.com/videos/category/history/the-photography-of-eudora-welty/']",
"[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII",
"[debug] youtube-dl version 2018.01.14",
"[debug] Python version 2.7.10 (CPython) - Darwin-17.3.0-x86_64-i386-64bit",
"[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1",
"[debug] Proxy map: {}",
"[generic] the-photography-of-eudora-welty: Requesting header",
"WARNING: Falling back on generic information extractor.",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.