[postprocessor] Add retries to file rename

[ggstrader]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

• I'm reporting a bug unrelated to a specific site
• I've verified that I'm running yt-dlp version 2022.09.01 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

I'm using yt-dlp as a drop-in replacement for youtube-dl simply for it's speed. The command:

youtube-dl.exe -o 'E:\[MY-PATH]\%(playlist)s\%(title)s.%(ext)s' -f 'bestaudio[ext=m4a]' --ignore-errors --continue --no-overwrites --add-metadata https://youtube.com/playlist?list=PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP

has no errors. But with yt-dlp:

yt-dlp.exe -vU -o 'E:\[MY-PATH]\%(playlist)s\%(title)s.%(ext)s' -f 'bestaudio[ext=m4a]' --ignore-errors --continue --no-overwrites --add-metadata https://youtube.com/playlist?list=PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP

each file downloads fine, but errors out when renaming the part file:

ERROR: [WinError 5] Access is denied: 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a' -> 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a'
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1459, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1556, in __extract_info
  File "yt_dlp\YoutubeDL.py", line 1614, in process_ie_result
  File "yt_dlp\YoutubeDL.py", line 2712, in process_video_result
  File "yt_dlp\YoutubeDL.py", line 3247, in process_info
  File "yt_dlp\YoutubeDL.py", line 3424, in post_process
  File "yt_dlp\YoutubeDL.py", line 3406, in run_all_pps
  File "yt_dlp\YoutubeDL.py", line 3385, in run_pp
  File "yt_dlp\postprocessor\common.py", line 24, in run
  File "yt_dlp\postprocessor\common.py", line 129, in wrapper
  File "yt_dlp\postprocessor\ffmpeg.py", line 706, in run
PermissionError: [WinError 5] Access is denied: 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a' -> 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a'

It looks like the try_rename function has a minor difference from the same function in youtube-dl. I'm not aware of why that might be causing this issue, since it seems that if os.replace was causing issues then os.rename should as well. Based on what little related info I could find online I've tried:

• updating python from 3.7 to 3.10
• checking that yt-dlp is up to date
• running powershell as administrator
• disabling windows defender, since one person mentioned anti-virus software being the cause
• made sure directory structure isn't read-only
• using --no-part which doesn't appear to change anything. It still downloads a temp part file and tries to rename

I also note that it's not reliable which songs from any playlist have this error. Sometimes the renaming works, sometimes it doesn't... about 1 out of every 5.

I don't have a ton of time to sink into this so I'm wondering if anyone else on Windows (10) can reproduce this or if it's unique to me.

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', '-o', 'E:\\[MY-PATH]\\%(playlist)s\\%(title)s.%(ext)s', '-f', 'bestaudio[ext=m4a]', '--ignore-errors', '--continue', '--no-overwrites', '--add-metadata', '--concurrent-fragments', '4', '--windows-filename', '--file-access-retries', '10', '--fragment-retries', '10', 'https://youtube.com/playlist?list=PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d6] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 2021-04-25-git-d98884be41-full_build-www.gyan.dev (setts), ffprobe 2021-04-25-git-d98884be41-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1670 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.09.01, Current version: 2022.09.01
yt-dlp is up to date (2022.09.01)
[debug] [youtube:tab] Extracting URL: https://youtube.com/playlist?list=PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP
[youtube:tab] PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP: Downloading webpage
[debug] [youtube:tab] Final URL: https://www.youtube.com/playlist?list=PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP
[youtube:tab] PLsmhANoEf1qUifoSdCHh0vQ2PrZGOIbsP: Downloading API JSON with unavailable videos
[download] Downloading playlist: All CoD zombies songs
[youtube:tab] Playlist All CoD zombies songs: Downloading 61 videos of 61
...
[download] Downloading video 2 of 61
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=MAGYDBcJke8
[youtube] MAGYDBcJke8: Downloading webpage
[youtube] MAGYDBcJke8: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] MAGYDBcJke8: Downloading 1 format(s): 140
[debug] Invoking http downloader on "https://rr4---sn-naj7yn76.googlevideo.com/videoplayback?expire=1663679029&ei=1WUpY6tE4Zjb-A-n7pfgDA&ip=2601%3A681%3A4403%3A4a50%3Ad53%3Af8bf%3Af649%3A4ef2&id=o-AND0F2Wq7jC5fkj5YE2vJCo8oyOM2zd2c2NCNmKCJ7PC&itag=140&source=youtube&requiressl=yes&mh=Jr&mm=31%2C26&mn=sn-naj7yn76%2Csn-a5msenle&ms=au%2Conr&mv=m&mvi=4&pl=36&initcwndbps=2355000&spc=yR2vpxV-FqF6wkV0QOSQEoPg1CeAH24&vprv=1&svpuc=1&mime=audio%2Fmp4&gir=yes&clen=4710902&dur=291.038&lmt=1577147185960294&mt=1663657036&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&rbqsm=fr&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhANNzaI0Xmglfnb8Kmg43XEbQmkS8BBGNnH1LOwdJIT1hAiA4sMlUrVxeEoDUGIDEqlPO2cRBYyPOwYxw0nsTdDOnHQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAJs_wbXgNQ_0L3APhPB3_sGmof0mpLjmTs4jZt3VkB4DAiBN6qgbOYVXQSE-Fi0WLFGZ1nXK_yV2jc_uRWihFMwfUA%3D%3D"
[download] Destination: E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a
[download] 100% of 4.49MiB in 00:00 at 8.97MiB/s
[FixupM4a] Correcting container of "E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a" -map 0 -dn -ignore_unknown -c copy -f mp4 -movflags "+faststart" "file:E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a"
[Metadata] Adding metadata to "E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a" -map 0 -dn -ignore_unknown -vn -acodec copy -write_id3v1 1 -metadata "title=Shi No Numa Song - The One - Elena Siegman - Lyrics [Official]" -metadata "date=20101112" -metadata "description=Every zombie song https://www.youtube.com/playlist?list=PLE4CE8488145CA2A5

►Instagram: https://bit.ly/KSherwoodOpsInstagram
►Discord Server:  https://bit.ly/KSherwoodOpsDiscord
►Twitter: https://bit.ly/KSherwoodOpsTwitter
►Snapchat: https://bit.ly/KSherwoodOpsSnapchat
►Facebook: https://bit.ly/KSherwoodOpsFacebook
►Twitch: https://bit.ly/KSherwoodOpsTwitch
►Guitar tabs: https://bit.ly/KSherwoodOpsGuitarTabs

Thumbnail:  https://twitter.com/MrPentek" -metadata "synopsis=Every zombie song https://www.youtube.com/playlist?list=PLE4CE8488145CA2A5

►Instagram: https://bit.ly/KSherwoodOpsInstagram
►Discord Server:  https://bit.ly/KSherwoodOpsDiscord
►Twitter: https://bit.ly/KSherwoodOpsTwitter
►Snapchat: https://bit.ly/KSherwoodOpsSnapchat
►Facebook: https://bit.ly/KSherwoodOpsFacebook
►Twitch: https://bit.ly/KSherwoodOpsTwitch
►Guitar tabs: https://bit.ly/KSherwoodOpsGuitarTabs

Thumbnail:  https://twitter.com/MrPentek" -metadata "purl=https://www.youtube.com/watch?v=MAGYDBcJke8" -metadata "comment=https://www.youtube.com/watch?v=MAGYDBcJke8" -metadata "artist=ksherwoodops" -movflags "+faststart" "file:E:\[MY-PATH]\All CoD zombies songs\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a"
ERROR: [WinError 5] Access is denied: 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a' -> 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a'
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1459, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1556, in __extract_info
  File "yt_dlp\YoutubeDL.py", line 1614, in process_ie_result
  File "yt_dlp\YoutubeDL.py", line 2712, in process_video_result
  File "yt_dlp\YoutubeDL.py", line 3247, in process_info
  File "yt_dlp\YoutubeDL.py", line 3424, in post_process
  File "yt_dlp\YoutubeDL.py", line 3406, in run_all_pps
  File "yt_dlp\YoutubeDL.py", line 3385, in run_pp
  File "yt_dlp\postprocessor\common.py", line 24, in run
  File "yt_dlp\postprocessor\common.py", line 129, in wrapper
  File "yt_dlp\postprocessor\ffmpeg.py", line 706, in run
PermissionError: [WinError 5] Access is denied: 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].temp.m4a' -> 'E:\\[MY-PATH]\\All CoD zombies songs\\Shi No Numa Song - The One - Elena Siegman - Lyrics [Official].m4a'

[ggstrader]

I was ending up with a situation where I'd have a song.m4a and song.temp.m4a, where (seemingly randomly) one of the 2 would have metadata and the other wouldn't... that's the main reason for posting this issue as I couldn't very easily just delete all of the temp files or I'd lose metadata.

Oddly enough, running the command twice for a playlist corrects any of the problem files - deleting the temp files and adding metadata to the others.

I'm personally satisfied with that, so this can be closed as far as I'm concerned. This still seems like a potential bug so I'll let someone else close it when they've had the chance to look at it.

[pukkandan]

Are you sure you don't have another program that is locking the files? Some AVs and backup programs may do this

[ggstrader]

Are you sure you don't have another program that is locking the files? Some AVs and backup programs may do this

Pretty sure, like I said I disabled Windows Defender. I'd imagine running it the second time would fail in the same way if that were the case

[pukkandan]

Well, something's blocking the file from being accessed, and that has nothing to do with yt-dlp. The best I can do is maybe add a retry to the renames

[Jackabomb]

I have the same issue. I see this consistently any time I use --add-metadata inside my Win10 VM using network-mapped folders.
Unlike @ggstrader, I see this issue consistently and merely re-running the command does not fix it.
I don't see this when accessing a location on a local disk, either within the VM or on my native machine.

It might not be the worst idea to implement some kind of timed retry.
Apparently, os.replace() on windows is known to require a few attempts sometimes. That Stack Overflow question is a goldmine, because it actually gets to the heart of the problem. A web search for "python os.replace WinError 5" turns up a bunch of results (here's one) for pretty much all the basic file IO operation in the os module, with people having intermittent errors on all kinds of operations. Most responders get distracted by the permissions side of things and then scratch their head when the OP comes back and says "I fixed it! I just needed a short delay!" because they never got to the heart of the problem.

edit
In case the SO link ever dies, the gist of the problem is that os.replace on Windows uses MoveFileEx, which is not guaranteed to be atomic (and, in practice, is not atomic especially on networked drives).

[MarkKoz]

I experienced the same issue with network-mapped folders. My workaround has been to use a temporary path that is on a normal disk instead. For example, --paths "temp:$env:TEMP". This works alright assuming your other disk has enough space to store the temporary files. This may be a concern with larger files or when doing downloads in parallel.

[pukkandan]

To devs: Look into #8088 also, when implementing this

[dirkf]

...os.replace on Windows uses MoveFileEx,...

As well as os.replace(), there is now also pathlib.Path.replace(), but at least in CPython 3.12 this just wraps os.replace(). Thus, all the Python library APIs for updating a file eventually come down to the internal_rename() function here, which applies renameat() or rename(), or on Windows MoveFileEx(), without much in the way of additional checks or processing; this hasn't significantly changed since Py3.3 -- prior versions used only rename()/MoveFile() with no os.replace().

The problem with retrying is that there's no reason to believe that the inaccessible destination will become accessible within the timescale of a download operation (ie, without delays in the order of hours). However this may be something that works in practice but not in theory, with a judicious choice of default delay.

Documentation suggests that it's not just the destination file that must not be open but also any parent directory, even unto the root of the share, which would explain why this problem is both prevalent and difficult to debug.