"Temporary Folder" feature redesign

[glassez]

The problem is that everyone sees (or wants) in this feature something different, and the behavior, which does not fit into his vision, he feels wrong. There are some issues with this feature (e.g. #5503, #7113, #7137). Each offers to fix it on his manner, without thinking about its overall logic. Lately I often think about how to change it. I propose to discuss "desired" implementation of "Temporary folder" feature in this topic.

You can speak here how you can see this feature. I want a detailed description, possibly with some sort of algorithms of its activity in some circumstances.

The main problems that should be addressed are:

1. Torrent- or File-based temporary folder usage;
2. Behavior in case of torrents without root folder.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[glassez]

@sledgehammer999, @qbittorrent/frequent-contributors, let's discuss it!

[Gelunox]

I would suggest a default cache folder behaviour where

• those weird temporary ID folders aren't created (or removed in line with rule below)
• individual files of torrents are automatically moved to their completed save path
• subsequently if all files in a (sub)folder have been completed and moved, the folder is removed from the cache folder. (this could be implemented by marking that folder as completed and moving it as well, thereby not making any difference in treatment of files and folders)
• if "recheck torrents on completion" is checked, check a file once it completes before moving it.

an argument could be made to make all of the above optional (i.e. add settings items for them).

I've also read some suggestions where people ask that they can set save paths for individual files. For that use-case a per file/folder based override could be implemented that either saves directly or moves from cache depending on if the cache folder is enabled.

File tracking & force-rechecking for torrents would have the application check both paths for each file. In case of it existing in both places a prompt could be shown or an automated (saved) action could be performed (i.e. keep newest, keep biggest, keep oldest, etc.).

If this can get implemented in such a way that folders and files are not differentiated from each other this should also work for torrents without a root folder.

[donselmo]

• The temp ID doesn't bother me, but it make sense, to use the normal names.
• Torrent or file based copy/recheck could be tricky, it can be good for large files, but bad for many small files. I suggest a sub option, where the user can decide which is better for him. Or it can be autodecided depends of the current torrent file size & numbers, like if average file size is < 10MiB and more than 100 files we have. But these are just numbers I just can't test wich is bad for disk IO.
• An option to every torrents to 'do not use the temp folder' wich is a great help for limited size temp folders like ramdisks/SSDs.

• if "recheck torrents on completion" is checked, check a file once it completes before moving it.

I agreee

[glassez]

An option to every torrents to 'do not use the temp folder' wich is a great help for limited size temp folders like ramdisks/SSDs.

I'm not sure I understand what you mean... Option to reset the default value from settings during torrent addition (ie. I set "Use temp folder" in settings but I can disable it for some torrents in "Add torrent" dialog)?

[glassez]

And yet... a little to limit your imagination, I will say that we technically can't recheck some particular file, but the entire torrent only.

[donselmo]

I'm not sure I understand what you mean... Option to reset the default value from settings during torrent addition (ie. I set "Use temp folder" in settings but I can disable it for some torrents in "Add torrent" dialog)?

More precisely a right click option, like download in sequential order, but your idea is not bad either.

And yet... a little to limit your imagination, I will say that we technically can't recheck some particular file, but the entire torrent only.

Ah, yes sorry, that was my bad, torrents use hash crc. But this will go back us to my first problem.
The current behaviour is like:
download complete go to complete state -> if use temp then move files to dest folder -> if recheck then force recheck -> if pieces missing go to download state /then the whole torrent copy back to temp folder/.
If my don't use temp folder option is feasible - it's a good word? maybe possible is better? - then the new behaviour is like:
download complete go to complete state -> if use temp then move files to dest folder & set torrent don't use temp folder true -> if recheck then force recheck -> if pieces missing go to download state.
This will download missing pieces without copy and the file based copy is working to. Well it's uncompatible the check a file once it completes before moving it but I can live with this. Or this option - the file based copy - set the behaviour, if set true then work like above, if not, then simply change the sequence to recheck -> move.

[glassez]

download complete go to complete state -> if use temp then move files to dest folder & set torrent don't use temp folder true -> if recheck then force recheck -> if pieces missing go to download state.

This looks like a suitable option. But it has one logical problem. Let's say I add multifiles torrent and I uncheck some files. Then it plays like you suggest: download complete go to complete state -> if use temp then move files to dest folder & set torrent don't use temp folder true -> if recheck then force recheck -> if pieces missing go to download state. It's ok. But if I want to load some missing files later and check it? Torrent will return again in the unfinished state. How to be in this case?

[donselmo]

Hm, good question. It's a rare case wich needed check the temp folder and recheck options both and a partial completed torrent to.
I think the only suitable solution is a sub option like reuse temp folder. So the new option tree is like:
Use temp folder
sub opt: use file copy /if checked, files move it's destination folder right after download, if not, files move only when the whole torrent complete/
sub opt: reuse temp folder for partial torrents /if checked, the torrent copied to the temp folder and then start download new files, if not, the download start the current folder (This option is working only partially downloaded torrents; ie. you unchecked some files to not download it, but later you reconsider to download it after all.)/
(Or so, sorry my english is not good enough.)

Basically this option is decide to remove the don't use temp folder option for a partial torrent. Then the torrent is go to download state.
So in the end still 2 sequence exist no need a third.
If file based copy true:
download complete go to complete state -> move files to dest folder & set torrent don't use temp folder true -> if recheck then force recheck -> if pieces missing go to download state
If not:
download complete go to complete state -> if recheck then force recheck -> if pieces missing go to download state -> if use temp then move files to dest folder & set torrent don't use temp folder true

[glassez]

Torrent or file based copy/recheck could be tricky, it can be good for large files, but bad for many small files. I suggest a sub option, where the user can decide which is better for him. Or it can be autodecided depends of the current torrent file size & numbers, like if average file size is < 10MiB and more than 100 files we have. But these are just numbers I just can't test wich is bad for disk IO.

I don't see a fundamental difference in the relation to disk I/O.
One of the main issues for me, is it really necessary to have two ways (both file-based and torrent-based) or just one.

[donselmo]

In my experience, copy many small files slow down HDD I/O, probably the creation of hundreds of new entries in the FAT table. But it's FAT32/NTFS, I dont work under linux/mac. But yes, no need to overthinking it, if it's to slow that way, the user can simply switch off.
And, no, no need two way, it's just a comfort option ie. recheck is much faster on ramdisk/SSD than HDD so it make sense to use that way, but not a nessecity. And the user still can use the manual move: download to ramdisk/SSD without temp option and move complete torrents manually to HDD.
It would be great if it works both ways, but yes, I know I'm not the one who need to code it.

[ThePOO]

I found a simple solution to the problem. I abandoned qBittorrent and adopted Deluge. With Deluge my folder stays nice and clean. I loved qBittorrent and would come back if it cleaned up after itself, reliably.

[sledgehammer999]

I found a simple solution to the problem. I abandoned qBittorrent and adopted Deluge. With Deluge my folder stays nice and clean. I loved qBittorrent and would come back if it cleaned up after itself, reliably.

Joke's on you. In the last week 3 patches were merged. One fixing the bug about not removing the temporary folders, and the others using the torrents' original names instead of the "IDs".

@glassez is this thread still valid?

[Nerva72]

Yeah but I can hardly blame ThePOO -- this folder nonsense was ignored for far too long by qBittorrent.

[glassez]

this folder nonsense

Please do not claim your opinion as the only true.

@glassez is this thread still valid?

Yes. I'd like to hear more opinions about the following:

Torrent- or File-based temporary folder usage;

Currently we have Torrent-based temporary folder feature. IMO, it has some drawbacks. I'm thinking to change it to File-based feature. Later I will present here an example algorithm how it could work.

[sledgehammer999]

Currently we have Torrent-based temporary folder feature. IMO, it has some drawbacks. I'm thinking to change it to File-based feature.

You want to move individual files to their final destination?

[ThePOO]

Thanks. I will be re-loading qBittorrent and testing shortly. Thanks, again.

…

On 8/5/2017 12:27 PM, sledgehammer999 wrote: I found a simple solution to the problem. I abandoned qBittorrent and adopted Deluge. With Deluge my folder stays nice and clean. I loved qBittorrent and would come back if it cleaned up after itself, reliably. Jokes on you. In the last week 3 patches were merged. One fixing the bug about not removing the temporary folders, and the others using the torrents' original names instead of the "IDs". @glassez <https://github.com/glassez> is this thread still valid? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7164 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJV4peYlaFPLVVBD7MfMvalWwDGRza8sks5sVMIGgaJpZM4OiQh2>.

--

[sledgehammer999]

Thanks. I will be re-loading qBittorrent and testing shortly.

The patch fixing the bug where the folders weren't removed is in 3.3.15. The others aren't included yet in a release.

[ThePOO]

Great! A bunch of my friends will be happy to hear it's fixed and will be waiting on my testing --- they are chomping at the bit to get back into qBitorrent. Thank you for all your efforts to support qBittorrent --- YAY!!!

…

On 8/5/2017 3:37 PM, sledgehammer999 wrote: Thanks. I will be re-loading qBittorrent and testing shortly. The patch fixing the bug where the folders weren't removed is in 3.3.15. The others aren't included yet in a release. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7164 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJV4pfDsdWrICztC9HTbSV0LFsjuFDAJks5sVO7CgaJpZM4OiQh2>.

[glassez]

You want to move individual files to their final destination?

Yes. Unless there are some strong objections.

[Nerva72]

I'm not sure I'm in favor or object to that, particularly because I've never used a torrent client with that feature. But one thing I can think of is it might get annoying to see incomplete torrents popping up in my "Completed" folder, and then have to go check qBT to see if it has actually finished or if all it did was download the read.me file. If I want to sample the finished parts of a collection or series, I would be perfectly capable of doing that if it weren't for the slew of randomly named folders in my "Downloading" folder making it a hassle.

[glassez]

I'm not going to change it until I get approval here.
A possibly, I'll just make some patches for the problems of the current approach.

[holycrepe]

Can we add an option to use unique subfolders as in 3.3.x?
One use case is issues with different torrents having the same name, in which case both torrents would be downloaded to the same temporary folder.

[glassez]

The more I delve into it (I have been thinking about this for quite some time), the more I'm leaning towards the idea that the best solution is to be rid of the "Temporary folder" approach.
IMO, best and clear (as from the point of view of interface, and from the point of view of implementation) will be to use "Move completed torrent to" approach. What do I mean?
Users who want to have unfinished and finished torrents in different folders set "Save path" to their "incomplete" folder and "Move completed torrent to" to their "completed" folder.
At first glance we only swapped a few things. But really it will bring a number of advantages:

1. We are greatly simplifying the implementation design and get rid of the current hard maintainable "paint in ass" code.
2. Users can manage their "incomplete" folder the same way as the "completed" one (even using Auto Torrent Management feature).

@sledgehammer999, what do you think about?

[LordNyriox]

@glassez:

use "Move completed torrent to" approach.

Now that sounds much more sensible—something I might actually use.

I currently do exactly what you describe with my torrents—through externally moving their files, then resetting the affected torrent for that new location. Having qBittorrent do the same thing in a way that is not so annoying—that I can get behind.

[LordNyriox]

@glassez, @sledgehammer999: One thing, however, is that it might be nice to see a per-torrent option for this.

At the initial torrent options menu, you would then have the "Download to" drop-down box. Beneath that a "Move files when complete" check-box—which enables a "Move to" drop-down box.

Then each torrent could be moved to a different completed folder.

What do you think?

[glassez]

@LordNyriox, when I say

Users can manage their "incomplete" folder the same way as the "completed" one (even using Auto Torrent Management feature).

I mean this behavior as well (since we have this behavior for regular/complited save folder).

[glassez]

Also we can solve "semi-downloaded torrent" issue by introducing appropriate option (however, I can't think of its name). I mean, should we consider as completed only fully downloaded torrents or torrents that have completed downloading of all selected files.

[LordNyriox]

@glassez:

I mean this behavior as well (since we have this behavior for regular/complited save folder).

I was under the impression that the current implementation of the "temp folder" feature does not support using a different "incomplete" directory for each torrent—which was the behavior I was trying to describe.

EDIT: * facepalm *

I mean, should we consider as completed only fully downloaded torrents or torrents that have completed downloading of all selected files.

I believe that is ultimately dependent on whether qBittorrent is expected to continue downloading un-selected files after all selected files have been fully downloaded.

There is also a corner case of what happens if someone selects an un-selected file after the torrent is already completed.

Avoiding potential glitches like that, is why I generally avoid letting qBittorrent automate my post-downloading actions.

[glassez]

There is also a corner case of what happens if someone selects an un-selected file after the torrent is already completed.

Current behavior is when torrent finishes downloading of all selected files it become complete, but if you select an un-selected file after the torrent is already completed it become incomplete again.
In a new feature I want to have the following behavior:

1. Moving files from the "incomplete" folder occurs only once.
2. User can decide himself when it should occur: when the torrent completes the download of selected files or all files.

Use cases:

1. I select some files to download but I want to download other files later.
2. I select some files to download and I don't need the rest of the files.

[LordNyriox]

@glassez:

User can decide himself when it should occur

+1

Now, regarding my request for per-torrent incomplete folder…?

EDIT: * facepalm *

[glassez]

Now, regarding my request for per-torrent incomplete folder…?

Yes. This is a consequence of my previous answer:

Users can manage their "incomplete" folder the same way as the "completed" one (even using Auto Torrent Management feature).

User can override default "regular" save folder in Add New Torrent Dialog so he will be able to override "incomplete" folder as well using discussed feature.

[sledgehammer999]

@glassez I agree that it would be far simpler code-wise and probably logic-wise(from user POV) to have "move to folder after completion" functionality.
Per torrent setting is a plus.

As for what we should do when user selects another file while the torrent is already completed and moved: Having it as an option is an approach but it sounds a bit bloaty. Does anyone know what uTorrent does? (pretty much everyone here takes utorrent as de facto standard)

[donselmo]

If I remember correctly, utorrent just start download the missing files in the current location. So if a partially selected torrent downloaded and moved the "complete folder" it's start download there, if select an incomplete torrent, then it's just add the files, and download there, and the whole torrent move the "complete folder", when the download finised.

[sledgehammer999]

I could be wrong but here is my take: From a user POV the "move-after-completion" is a one time event, a command. I don't think he expects his files to be moved back when he selects another one for download.

And from a code point of view it is simpler to implement. After we do the move, we no longer need to remember the previous location. Especially if it is a per-torrent setting.

[glassez]

I could be wrong but here is my take: From a user POV the "move-after-completion" is a one time event, a command. I don't think he expects his files to be moved back when he selects another one for download.

Yes. As I previously mentioned, typically, the user knows what he wants from this torrent so typical u cases are:

1. User selects some files to download but he wants to download other files later ("Move fully downloaded torrent only" option should be set).
2. User selects some files to download and he doesn't need the rest of the files.

(Exceptional case) User selects some files to download and he isn't sure he needed or not the remaining files (or he just changed his mind and decided to download any files once the torrent has already been moved to the "completed" folder. I don't think he expects his files to be moved back. Otherwise he can manage save folder for this torrent manually.

[glassez]

In fact, this feature is limited to setting the "moving" trigger to be activated once torrent finished downloading (either all or just selected files depending on user choice).

And from a code point of view it is simpler to implement.

Yes. This will give us a simple, reliable and maintainable code.

[glassez]

@sledgehammer999, as I understand it, you approve "move-after-completion" feature?

[Gelunox]

Might I suggest adding to the "move-after-completion" the option to have this work for individual files within a torrent as well?

Could be done by moving the file & then setting the priority to "do not download" or, with a little more work, adding a special status to mark it as moved so that when the entire torrent is finished qBT knows that it can rediscover it in the moved-to folder.

[glassez]

Could be done by moving the file & then setting the priority to "do not download" or, with a little more work adding a special status to mark it as moved so that when the entire torrent is finished qBT knows that it can rediscover it in the moved folder.

You mean not seeding completed files until the entire torrent is complete? How do you see the sense in that? You just want not to waste the place in "incomplete" folder (e.g. on limited size disk)?
Actually, the per-file save path management is more complex job than it seems at first glance. Maybe I'll think of something after the main part of feature will be completed.

[sledgehammer999]

Actually, the per-file save path management is more complex job than it seems at first glance. Maybe I'll think of something after the main part of feature will be completed.

I think so(about complexity).
Also I am worried that it isn't a good idea to have a torrent span 2 locations. Especially when we "auto manage" it at this point(not in the sense of TMM). Users might find it confusing. Or they may not remember to salvage all files in case of crash, format, migration etc.