Version 459

closes #447, closes #982, closes #875, closes #989, closes #986, closes #858, closes #855, closes #807, closes #790

db/help my db is broke.txt

@@ -127,9 +127,9 @@ Do not delete your original files for now. Just rename them to 'old' and move th
 
 ** repair **
 
-This is a newer command that tries to fix a database file in place. It sometimes works better than clone, but I have had reports of it not fixing all malformations, so you'll want to run integrity_check again and do a clone if needed.
+This command tries to fix a database file in place. It works very very slowly. While it may be able to recover some rows a clones would lose, it cannot fix everything and may leave you with a database that is still malformed, so you'll want to run integrity_check again and do a clone if needed.
 
-It is very important you have a copy of the broken file here, as this edits the file in place.
+It is very important you have a backup copy of the broken file(s) here, as this edits the file in place.
 
 .open client.db
 .recover
@@ -204,7 +204,9 @@ When your recovery is done, your tag counts or siblings may be incorrect. If so
 
 *** if caches was hit ***
 
-If client.caches.db was hit, this is the best scenario, since everything in there is just a mirror of some other data, and can be reprocessed. Run everything under _database->regenerate_ in turn. That'll likely be enough for almost all cases, but if you still get screwy behaviour with the duplicate system or something, let me know and I'll see what I can do.
+If client.caches.db was hit, this is the best scenario, since everything in there is just a mirror of some other data, and can be reprocessed. Run the main jobs under _database->regenerate_ in turn. Read the pre-run descriptions to make sure you aren't doing the same thing twice, since some of the mapping cache regen jobs there are big and do the work of other jobs too. That'll likely be enough for almost all cases, but if you still get screwy behaviour with the duplicate system or something, let me know and I'll see what I can do.
+
+An alternate answer, if your client.caches.db is really truncated/broken, is just to delete the file and boot without it. The client's repair code is now good enough to try to regenerate the entire file on boot. It may give you more post-boot regen instructions.
 
 
 *** the database broke again ***

db/help my media files are broke.txt

@@ -0,0 +1,51 @@
+If you have lost lots of media files due to a mistake or a drive failing, or if you have rolled back to an older backup and now your file structure is out of sync with your database record, there are several things to do.
+
+First off, make sure you are running on good hard drives now. Check the 'help my db is broke' help document for info here. If your .db files were also on a failing disk, you'll want to run through that whole document to make sure your database itself is ok--there's no point trying to fix the file record on a malformed database.
+
+Next, boot into the client. If you lost whole media file directories, or they moved to a new location, your client will throw up a repair dialog during boot to help you find the right locations again. If you have truly lost some directories, it should be able to recreate empty ones for you.
+
+FYI: In hydrus's file storage, there are two sets of 256 sub folders. One is 'fxx' and one is 'txx', where 'f' is for files, 't' is for thumbnails, and 'xx' counts from 00 to ff in hexadecimal. You need all 512 directories located to boot.
+
+Once you are booted, your client may throw up lots of missing thumbnail/file errors. Do not worry so much about these for now, but it would be best to close your larger pages and pause the import pages for now. Don't pause network traffic, since we'll be using that in a minute.
+
+If you needed to remap your file locations during pre-boot repair, then hit _database->migrate database_. It probably still has your old damaged location set as the 'ideal', and the remapped correct locations just a temporary location it currently wants to move away from. Correct this by giving the new locations some weight and removing the old locations. Once you are done, you should only see good locations listed, no desire to move anything, with 'current' and 'ideal' usage at the same percentage.
+
+Next, we have two classes of problem: files in the database but not in storage, and files in storage but not in the database.
+
+
+*** Files that the database thinks are in storage, but they are not ***
+
+These are the cause of the missing file errors. They could have been deleted or damaged, or they might have been imported after you made your file storage backup. The are not on disk any more, but the database doesn't know that, so any time it tries to access where they _were_, it is surprised and throws the error. We need to let it fix all those records.
+
+Hit _database->file maintenance->review_, and then hit the 'add new jobs' tab. Click the 'all media files' button, or just run a search for 'system:everything', and then queue up the job that says 'if file is missing and has URL, try to download, else remove record'. It should be the default job. There are other jobs--feel free to read their descriptions--but you want to deal with the 'missing' suite for now.
+
+Queue up that job for everything and then on the first tab you can hurry all that work along. It may take an hour or more to check all your files. Any of the files with URLs (i.e. anything you once downloaded) will be queued up in a new download page that may get pretty large, so you might like to pause file maintenance under the same _database->file maintenance_ menu for a bit if it gets too big.
+
+With luck, it will have redownloaded many of your missing files. Those it could not will no longer be listed in 'my files' (as if they were deleted) and will not cause 'missing file' errors. All file hashes and any known URLs will be written to your log file and a new folder in your database directory. If you have a manual way to find what could not be redownloaded, you can try re-importing them in future, but otherwise this is the best answer for now. Even though it sucks to lose files, all you can do now is let the database know about it too.
+
+Once the file maintenance is done, and the download page is done, we then need to run 'if file is missing, remove record' on everything once more to deal with the files that had a URL but could not be downloaded successfully.
+
+You do not need to run the 'regenerate thumbnail' jobs--the client does that automatically as needed when you load a file, so it will fill in gaps no problem--but if you lost whole thumbnail folders and some or all are now empty, running this job will speed up future load by doing that work in the background now.
+
+
+*** Files that are in storage, but the database does not know it ***
+
+~If you only had drive errors or you restored a backup that included both file storage and database files made at the same time, you can ignore this step. You probably have a couple of extra orphan files, but everyone does, it isn't a big problem.~
+
+If you restored an older file storage backup to a newer database, these would be files that were deleted after the backup was made. If you restored an older database backup to a newer file storage, then these would be files that were imported after the backup was made. In either case, they are files in your file structure that the database does not know about, and we want to collect them together to A) delete them or B) reimport them.
+
+Run _database->db maintenance->clear orphan files_. Choose a location for the files to go to, and then wait for it to finish. Browse through them to verify what you are looking at, and then either delete them or reimport them.
+
+
+*** Repository Update Files ***
+
+If you had missing files, that may include some update files for the PTR or another repo, which are stored outside the 'system:everything' search, but are still in your file storage. Don't worry about it too much--the next time repo processing happens, if there is a missing or damaged file, it will notice and run an automatic maintenance routine to fix things. It all happens through the same review file maintenance panel if you want to watch it. Once it has cleared records for the missing files, you can resume repo processing and it will redownload the missing files automatically and resume.
+
+Note these files do not have URLs, so if you are an advanced user and try to run this maintenance yourself with the 'and has URL' qualifier, they will not load up in a download page.
+
+
+*** OK ***
+
+You should now be done! If you get weird file counts on autocomplete results or more missing file errors, let me know.
+
+I recommend you take a breath, pour yourself a drink, and make a job for tomorrow to think about your backup routine.

help/changelog.html

@@ -8,6 +8,44 @@
 		<div class="content">
 			<h3 id="changelog"><a href="#changelog">changelog</a></h3>
 			<ul>
+				<li><h3 id="version_459"><a href="#version_459">version 459</a></h3></li>
+				<ul>
+					<li>main highlights:</li>
+					<li>to help debugging from screenshots etc..., the client now puts its version name on every window title, like 'review services - hydrus client 459'. (issue #447)</li>
+					<li>the 'main gui title' option is reset and replaced with 'application display name' this week. it now alters the 'hydrus client' part on every window title. the actual 'main gui title' is now "main" lmao</li>
+					<li>wrote a new help document, 'help my media files are broke' in the db directory. this collects the different recovery routines I have developed while helping users after drive failure or other problems cause many missing files or a desynced database and file storage structure. I will be pointing people to this in future, please feel free to do the same</li>
+					<li>two new file maintenance jobs are added: for 'presence' and 'integrity' checks, you can now do 'if has URL, then try to redownload, else remove record'. this tidily combines the two more specific jobs that are commonly run after a hard drive problem. the 'presence' version is now the default selected job and recommended for most simple situations</li>
+					<li>a new easy-select button on the review file maintenance panel  lets you select all media files</li>
+					<li>I put some more time into the new duplicate filter zoom locking calculation. thank you to users who sent in examples of my code not working well--I have scaled back what it tried to do. now it will tend to heigh-lock for landscape images and width-lock for everything else _unless_ you are currently viewing the default zoom and that roughly fills the canvas on a dimension and doing the default lock would cause the next image to spill over the screen. the 'solution' here hence targets the 'watermark spilled over' problem more specifically and deals with all combinations of landscape/portrait A/B/canvas better. I'd still like to introduce some zoom locking options here for regular browsing, but pinning down what exactly is useful is trickier than I expected</li>
+					<li>the edit tag import options panel now shows 'THIS CURRENTLY GETS NO TAGS' warning red text if it is non default and no tags are set to parse and there are no additional tags</li>
+					<li>the status bar now shows '1,234 files in 20 collections' when you have just collections or just collections selected (previously, it wrongly said '1,234 collections') (issue #807)</li>
+					<li>macOS clients will now show dialog-created menus in a debug dialog unless the new BUGFIX checkbox under 'gui' options page is unchecked. this _should_ help Big Sur users who are unable to interact with menus created in dialogs like manage tags or manage services. I threw this together, so let me know how this works for you! (issue #986, issue #858)</li>
+					<li>the program now waits specifically for currently running subscriptions to stop work and save themselves before moving on with further shutdown tasks. hand in hand with this, subscriptions are now faster at stopping work on client exit, even when they have no popup message (through which some hackery dackery shutdown signals are sent otherwise) (issue #790)</li>
+					<li>physically deleting thousands of files in one go should no longer lock up the file manager and other systems for ages--physical delete is now serialised and processed on a new threaded mainloop, so it doesn't matter how fast the requests come it, it will chunter at a polite speed and take breaks and should not choke other consumers and freeze up other 'things are great, you can start new work' status checks</li>
+					<li>.</li>
+					<li>network job improvements:</li>
+					<li>hydrus network jobs now try to resume incomplete responses (previously they just dumped out and tried again from the beginning). if a server provides less content than it said it would, or it explicitly gives us a partial response, we now resume at that point! should fix dowmnloading of longer videos on 8chan.moe</li>
+					<li>hydrus network jobs now send a range header by default, letting servers return 206 (partial content) if they wish</li>
+					<li>SSL errors (cert verification and similar) are now caught in the network engine separately to generic connection errors. they will not be reattempted, and the failure note will display specific error info</li>
+					<li>refactored some response header parsing code, cleaning up how some variables are initialised</li>
+					<li>greatly improved the job reattempt system, resetting variables more neatly</li>
+					<li>improved some response range and content length calculations</li>
+					<li>.</li>
+					<li>smaller items, mostly bug fixes:</li>
+					<li>fixed a recent typo bug that caused the edit url class dialog to always spawn with 'file url' type set. sorry, this was stupid! (issue #982)</li>
+					<li>the edit url class dialog now sends the 'normalised' url as the example text for the api and referral string converter edit panels</li>
+					<li>fixed the new advanced file deletion 'remember last' checkboxes in _options->files and trash_. they weren't hooked up right, sorry!</li>
+					<li>fixed the tag menu's siblings submenu's copy command where it says 'ideal is "xxx" on: yyy'. despite the correct label, this was sometimes copying a different service's ideal (issue #855)</li>
+					<li>fixed the 'media zooms' text input under _options->media_ not turning off the 'red' invalid mode once its text is again valid</li>
+					<li>when you cancel the 'edit parser' dialog, it shouldn't say 'it looks like you made changes' so much when you didn't make any. the 'has changes' test now ignores some background test data updates that may have happened (issue #875)</li>
+					<li>if a JSON parsing formula is given HTML, the 'cannot parse' error now tries to detect this and present a better error text (issue #888)</li>
+					<li>I _think_ I fixed a problem in the new bytes rendering calculation (where it goes 1018825 to "995KB") where on some unlucky edge-case numbers it could non-determinitively choose different sig figs (e.g. flipping between 994.9KB and 995KB)</li>
+					<li>fixed a couple of file move actions that were unable to move across Windows partitions when the timestamp was before 1980-01-01 (issue #989)</li>
+					<li>mr bones now recognises you are not a new user if you deleted all your files. you can never exit</li>
+					<li>after some testing, it seems like large 'drop table' operations in SQLite sometimes work within seconds but generally take far longer, often working as slow as 10MB/s (and I just talked to a guy for whom it took _days_(!!!). writing a fix to make 'delete service' always run fast for something as large as the PTR will take planning and work, so for now I have attached a warning note to the delete service confirmation dialog</li>
+					<li>updated the file maintenance review panel to newer async code</li>
+					<li>fixed a typo bug in URL export when a file is missing/bad in file maintenance</li>
+				</ul>
 				<li><h3 id="version_458"><a href="#version_458">version 458</a></h3></li>
 				<ul>
 					<li>quality of life:</li>

hydrus/client/ClientController.py

@@ -16,8 +16,8 @@
 from hydrus.core import HydrusData
 from hydrus.core import HydrusExceptions
 from hydrus.core import HydrusGlobals as HG
-from hydrus.core import HydrusPaths
 from hydrus.core import HydrusSerialisable
+from hydrus.core import HydrusTemp
 from hydrus.core import HydrusThreading
 from hydrus.core import HydrusVideoHandling
 from hydrus.core.networking import HydrusNetwork
@@ -110,6 +110,7 @@ def __init__( self, pubsub, *args, **kwargs ):
         self._pubsub = pubsub
 
         self.setApplicationName( 'Hydrus Client' )
+
         self.setApplicationVersion( str( HC.SOFTWARE_VERSION ) )
 
         QC.qInstallMessageHandler( MessageHandler )
@@ -215,7 +216,7 @@ def _InitDB( self ):
 
     def _InitTempDir( self ):
 
-        self.temp_dir = HydrusPaths.GetTempDir()
+        self.temp_dir = HydrusTemp.GetTempDir()
 
 
     def _DestroySplash( self ):
@@ -1136,6 +1137,8 @@ def InitModel( self ):
 
         #
 
+        self.client_files_manager.Start()
+
         self._managers[ 'undo' ] = ClientManagers.UndoManager( self )
 
         self.frame_splash_status.SetSubtext( 'image caches' )

hydrus/client/ClientDownloading.py

@@ -4,8 +4,8 @@
 from hydrus.core import HydrusConstants as HC
 from hydrus.core import HydrusData
 from hydrus.core import HydrusGlobals as HG
-from hydrus.core import HydrusPaths
 from hydrus.core import HydrusSerialisable
+from hydrus.core import HydrusTemp
 from hydrus.core import HydrusThreading
 
 from hydrus.client import ClientConstants as CC
@@ -307,7 +307,7 @@ def MainLoop( self ):
 
                             if file_repository.IsFunctional():
 
-                                ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
+                                ( os_file_handle, temp_path ) = HydrusTemp.GetTempPath()
 
                                 try:
 
@@ -341,7 +341,7 @@ def MainLoop( self ):
 
                                 finally:
 
-                                    HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
+                                    HydrusTemp.CleanUpTempPath( os_file_handle, temp_path )