Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSD friendly file downloading #23606

Closed
mrx23dot opened this issue Jan 2, 2020 · 3 comments
Closed

SSD friendly file downloading #23606

mrx23dot opened this issue Jan 2, 2020 · 3 comments

Comments

@mrx23dot
Copy link

@mrx23dot mrx23dot commented Jan 2, 2020

  • I'm reporting a feature request
  • I've verified that I'm running youtube-dl version 2020.01.01
  • I've searched the bugtracker for similar feature requests including closed ones

Description

Downloading a simple video file eg: youtube-dl -f "hls-480p" xvideos_site_56MB_vid_mp4
Creates many temp files on HDD during download:

x.mp4.part-Frag55.part
x.mp4.part-Frag56.part files of 0-1KB
x.mp4.ytdl
x.mp4.part

SSDs have limited lifespans when writing to them, so please add a feature to use more RAM instead of temporary HDD/SSD writes. Eg. --memory 128MB, would wait for 128MB incoming data then write to the final file, without unnecessary additional writes.

Imagine an SSD in a download server, or SD in RaspberryPi, or even in HDD it causes fragmentation.
See Transmission's file cache handling as an example.

@mrx23dot mrx23dot added the request label Jan 2, 2020
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jan 2, 2020

Download files to RAM disk.

@dstftw dstftw closed this Jan 2, 2020
@dstftw dstftw added the out-of-scope label Jan 2, 2020
@reallyuniquename
Copy link

@reallyuniquename reallyuniquename commented Jul 14, 2020

@dstftw

Download files to RAM disk.

That requires a lot of RAM since some files are huge, I'm talking about dozens of gigabytes per file.

Wouldn't it be possible to download current fragment to memory and then write it to file instead of abusing hardware with unnecessary writes? Or at least add an option to specify directory where youtube-dl downloads fragments to.

@reallyuniquename
Copy link

@reallyuniquename reallyuniquename commented Aug 26, 2020

I've been revisiting this issue, it turns out ffmpeg doesn't trash my drive with thousands of files so one can use --hls-prefer-ffmpeg although I'm not sure about the amount of writes. Maybe buffer size for ffmpeg could be tweaked as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.