--download archive on OSX 10.9.4 chokes on fnctl.flock

[jringloff]

I am a total hack (I don't know python from lisp) but it appears that the locking method used by youtube-dl for the --download-archive option don't jive with OSX 10.9.4

imac:john(60:1060)$ youtube-dl -vicwn --download-archive archive "http://www.youtube.com/playlist id here/videos"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-vicwn', '--download-archive', 'archive', 'http://www.youtube.com/playlist id here/videos']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.08.10
[debug] Python version 2.7.6 - Darwin-13.3.0-x86_64-i386-64bit
[debug] Proxy map: {}
[download] Downloading playlist: playlist id here
[youtube:user] playlist id here: Downloading video ids from 1 to 51
[youtube:user] playlist playlist id here: Downloading 4 videos
[download] Downloading video #1 of 4
ERROR: [Errno 45] Operation not supported
Traceback (most recent call last):
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 527, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 638, in process_ie_result
reason = self._match_entry(entry)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 484, in _match_entry
if self.in_download_archive(info_dict):
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1123, in in_download_archive
with locked_file(fn, 'r', encoding='utf-8') as archive_file:
File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1130, in enter
_lock_file(self.f, exclusive)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1115, in _lock_file
fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)
IOError: [Errno 45] Operation not supported

In utils.py changing from a fcntl.lockf to a flock seems to the trick:

from

else:
    import fcntl

    def _lock_file(f, exclusive):
        fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)

    def _unlock_file(f):
        fcntl.lockf(f, fcntl.LOCK_UN)

to

else:
    import fcntl

    def _lock_file(f, exclusive):
        fcntl.flock(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)

    def _unlock_file(f):
        fcntl.flock(f, fcntl.LOCK_UN)

[jringloff]

Of course this might break some other non-windows flavors. so I am sure more logic needs to be applied.

[phihag]

Thank you very much for the detailed report!

I don't know OSX development, but this seems weird since lockf is actually documented on OSX. But I'm not emotionally attached to it, and flock should emulate where it's not available anyways, so I've just switch to flock.

[jringloff]

I could be missing a library or have something broken. But I thought I would bug it for validation. I will also update my mac ports and see if that has an impact using the version I was using. lockf seems the better option, so while flock may be a fix, it may not be the best fix, particularly if it turns out to be environmental (just my system).