Skip to content

Commit

Permalink
Version 308
Browse files Browse the repository at this point in the history
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed May 23, 2018
1 parent 7f39f33 commit 17a6569
Show file tree
Hide file tree
Showing 37 changed files with 2,908 additions and 2,364 deletions.
42 changes: 42 additions & 0 deletions help/changelog.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,48 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 308</h3></li>
<ul>
<li>the multiple watcher will now discard new urls if it is already watching them</li>
<li>the multiple watcher will list x/y progress as just 'x' if x==y (making it easier to scan the list)</li>
<li>the multiple watcher now lists a couple of 'total' summary lines on its ui--the top lists total number of watchers and queue progress, the bottom lists the usual '23 successful, 3 deleted' line, but summed for all watchers</li>
<li>the multiple watcher will now warn you if you try to remove the highlit, alive or un-caught-up watchers</li>
<li>the multiple watcher will now resort if the thread subject (or rather, any data in the current sort column) changes (which usually happens right after it is added, when you see it change from from 'unknown subject' to 'mlp is kino')</li>
<li>fixed an issue where multiple watchers were not unscheduling down their update job correctly on page close</li>
<li>the booru selector in the edit subscription panel should now be in the tab traversal order for keyboard/automated focusing tasks</li>
<li>the boorus in that selector are now alphabetised</li>
<li>tag import options namespaces are now alphabetised</li>
<li>removed/renamed pretty much all references to 'thread' in the watcher code and ui presentation, since it can now do a bunch of other stuff. it is now just the 'watcher' and the 'multiple watcher'</li>
<li>deleted a bunch of old static thread watcher and page of images code from the old downloading system</li>
<li>added an experimental 'compact' button to advanced mode users' manage subscriptions panels. this removes urls from the selected subscriptions' caches that are no longer useful, keeping their load/save snappy. this is still in testing--be careful with it!</li>
<li>the hydrus splash screen now has a bare frame caption and will appear in the taskbar--which helps with some alt-tab and 'where the hell did it go?' stuff if you need to enter a password</li>
<li>wrote five 'reasonable defaults' buttons for the 'check timings' options panel for quick entry for different thread/subscription scenarios.</li>
<li>added a checkbox to this panel that will swap the reactive options with a simpler single checkbox</li>
<li>also clarified/fleshed out the help button on this panel</li>
<li>fixed an important source of program instability related to page alive/dead status checking that was inadvertantly talking subtly to the main gui frame even on non ui threads</li>
<li>improved how some 'page is closed but not destroyed' test logic for pages inside a closed-but-not-destroyed notebook</li>
<li>fixed another small place where the db was talking to the main gui object about status bar updates in a potentially unstable way</li>
<li>fixed another small place where the foreground daemons were talking to the main gui frame in a trivial but potentially unstable way</li>
<li>played around with some taglist sizer and layout settings</li>
<li>the gallery and simple download pages are now a little shorter--the pause and cancel buttons are now just to the right of the status texts, rather than on their own row beneath the network job controls.</li>
<li>the various bandwidth-overriding network jobs in the download system--like gallery page downloading--now wait 30s before overriding their bandwidth. hence these jobs will now obey the usual bandwidth rules up to a point</li>
<li>the simple downloader also obeys the usual bandwidth rules for 30s but no longer has a static wait, so it can run much faster in certain situations</li>
<li>network jobs that will override bandwidth in the future will now report that countdown in their status texts</li>
<li>fixed a bug in the old booru code that meant some boorus were superfluously requesting the 0th indexed page of a gallery more frequently than needed in order to reestablish a 'page size' cache. this value is now cached globally and will be replaced by a completely different system in the new gallery downloader</li>
<li>added a decent tooltip to the 'gallery fixed delay' widgets in the options->downloading panel</li>
<li>the autocomplete input should clear itself after a 'broadcast' event a bit quicker and stop some dupe inputs in certain edge cases</li>
<li>the tumblr url class now recognises that tumblr posts can have multiple files, which helps some source url lookup logic</li>
<li>added a url class for artstation file pages</li>
<li>the primary file import url (the one listed in the file import list) will now correctly not associate with the resulting file if its url class is so set</li>
<li>all the import objects now have much lower idle CPU time and thread needs and start in slightly offset times, smoothing out the thread count spikes</li>
<li>all the import objects will now respond quickly to changes to the underlying file import cache (like right-click->try again events)</li>
<li>the new job scheduling system now uses two queues--fast and slow, in order to reduce some resort/insert overhead</li>
<li>a couple more improvements to the new job scheduling system to smooth out spikes</li>
<li>if the temporary path override does not exist, the client will now compain with spammy popup messages and fall back to the default</li>
<li>if the temporary path override does not exist or is not writeable-to on options dialog ok, a veto exception will be raised</li>
<li>refactored the watcher and multiple watcher to their own file, ClientImportWatchers</li>
<li>misc fixes</li>
</ul>
<li><h3>version 307</h3></li>
<ul>
<li>wrote a gelbooru 0.2.5 (which matches gelbooru itself) parser in the new system. it now has some more redundancy and produces md5 hash and source urls</li>
Expand Down
21 changes: 12 additions & 9 deletions include/ClientConstants.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,22 +137,25 @@
FLAGS_CENTER = wx.SizerFlags( 0 ).Border( wx.ALL, 2 ).Center()

FLAGS_EXPAND_PERPENDICULAR = wx.SizerFlags( 0 ).Border( wx.ALL, 2 ).Expand()
FLAGS_EXPAND_BOTH_WAYS = wx.SizerFlags( 2 ).Border( wx.ALL, 2 ).Expand()
FLAGS_EXPAND_DEPTH_ONLY = wx.SizerFlags( 2 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_CENTER_VERTICAL )
FLAGS_EXPAND_BOTH_WAYS = wx.SizerFlags( 5 ).Border( wx.ALL, 2 ).Expand()
FLAGS_EXPAND_DEPTH_ONLY = wx.SizerFlags( 5 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_CENTER_VERTICAL )

FLAGS_SIZER_CENTER = wx.SizerFlags( 2 ).Center()
FLAGS_EXPAND_BOTH_WAYS_POLITE = wx.SizerFlags( 3 ).Border( wx.ALL, 2 ).Expand()
FLAGS_EXPAND_BOTH_WAYS_SHY = wx.SizerFlags( 1 ).Border( wx.ALL, 2 ).Expand()

FLAGS_SIZER_CENTER = wx.SizerFlags( 5 ).Center()

FLAGS_EXPAND_SIZER_PERPENDICULAR = wx.SizerFlags( 0 ).Expand()
FLAGS_EXPAND_SIZER_BOTH_WAYS = wx.SizerFlags( 2 ).Expand()
FLAGS_EXPAND_SIZER_DEPTH_ONLY = wx.SizerFlags( 2 ).Align( wx.ALIGN_CENTER_VERTICAL )
FLAGS_EXPAND_SIZER_BOTH_WAYS = wx.SizerFlags( 5 ).Expand()
FLAGS_EXPAND_SIZER_DEPTH_ONLY = wx.SizerFlags( 5 ).Align( wx.ALIGN_CENTER_VERTICAL )

FLAGS_BUTTON_SIZER = wx.SizerFlags( 0 ).Align( wx.ALIGN_RIGHT )

FLAGS_LONE_BUTTON = wx.SizerFlags( 0 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_RIGHT )

FLAGS_VCENTER = wx.SizerFlags( 0 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_CENTER_VERTICAL )
FLAGS_SIZER_VCENTER = wx.SizerFlags( 0 ).Align( wx.ALIGN_CENTRE_VERTICAL )
FLAGS_VCENTER_EXPAND_DEPTH_ONLY = wx.SizerFlags( 2 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_CENTER_VERTICAL )
FLAGS_VCENTER_EXPAND_DEPTH_ONLY = wx.SizerFlags( 5 ).Border( wx.ALL, 2 ).Align( wx.ALIGN_CENTER_VERTICAL )

DAY = 0
WEEK = 1
Expand Down Expand Up @@ -268,7 +271,7 @@
NETWORK_CONTEXT_DOWNLOADER = 3
NETWORK_CONTEXT_DOWNLOADER_PAGE = 4
NETWORK_CONTEXT_SUBSCRIPTION = 5
NETWORK_CONTEXT_THREAD_WATCHER_PAGE = 6
NETWORK_CONTEXT_WATCHER_PAGE = 6

network_context_type_string_lookup = {}

Expand All @@ -278,7 +281,7 @@
network_context_type_string_lookup[ NETWORK_CONTEXT_DOWNLOADER ] = 'downloader'
network_context_type_string_lookup[ NETWORK_CONTEXT_DOWNLOADER_PAGE ] = 'downloader page'
network_context_type_string_lookup[ NETWORK_CONTEXT_SUBSCRIPTION ] = 'subscription'
network_context_type_string_lookup[ NETWORK_CONTEXT_THREAD_WATCHER_PAGE ] = 'watcher page'
network_context_type_string_lookup[ NETWORK_CONTEXT_WATCHER_PAGE ] = 'watcher page'

network_context_type_description_lookup = {}

Expand All @@ -288,7 +291,7 @@
network_context_type_description_lookup[ NETWORK_CONTEXT_DOWNLOADER ] = 'Network traffic going through a downloader. This is no longer used.'
network_context_type_description_lookup[ NETWORK_CONTEXT_DOWNLOADER_PAGE ] = 'Network traffic going through a single downloader page. This is an ephemeral context--it will not be saved through a client restart. It is useful to throttle individual downloader pages so they give the db and other import pages time to do work.'
network_context_type_description_lookup[ NETWORK_CONTEXT_SUBSCRIPTION ] = 'Network traffic going through a subscription query. Each query gets its own network context, named \'[subscription name]: [query text]\'.'
network_context_type_description_lookup[ NETWORK_CONTEXT_THREAD_WATCHER_PAGE ] = 'Network traffic going through a single watcher page. This is an ephemeral context--it will not be saved through a client restart. It is useful to throttle individual watcher pages so they give the db and other import pages time to do work.'
network_context_type_description_lookup[ NETWORK_CONTEXT_WATCHER_PAGE ] = 'Network traffic going through a single watcher page. This is an ephemeral context--it will not be saved through a client restart. It is useful to throttle individual watcher pages so they give the db and other import pages time to do work.'

PAGE_FILE_COUNT_DISPLAY_ALL = 0
PAGE_FILE_COUNT_DISPLAY_NONE = 1
Expand Down
86 changes: 56 additions & 30 deletions include/ClientController.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ def __init__( self, db_dir, no_daemons, no_wal ):

HC.options = self.options

self._page_key_lock = threading.Lock()

self._alive_page_keys = set()
self._closed_page_keys = set()

self._last_mouse_position = None
self._menu_open = False
self._previously_idle = False
Expand Down Expand Up @@ -109,6 +114,18 @@ def _ReportShutdownDaemonsStatus( self ):
self.pub( 'splash_set_status_subtext', ', '.join( names ) )


def AcquirePageKey( self ):

with self._page_key_lock:

page_key = HydrusData.GenerateKey()

self._alive_page_keys.add( page_key )

return page_key



def CallBlockingToWx( self, func, *args, **kwargs ):

def wx_code( job_key ):
Expand Down Expand Up @@ -173,22 +190,26 @@ def wx_code( job_key ):

def CallLaterWXSafe( self, window, initial_delay, func, *args, **kwargs ):

job_scheduler = self._GetAppropriateJobScheduler( initial_delay )

call = HydrusData.Call( func, *args, **kwargs )

job = ClientThreading.WXAwareJob( self, self._job_scheduler, window, initial_delay, call )
job = ClientThreading.WXAwareJob( self, job_scheduler, window, initial_delay, call )

self._job_scheduler.AddJob( job )
job_scheduler.AddJob( job )

return job


def CallRepeatingWXSafe( self, window, initial_delay, period, func, *args, **kwargs ):

job_scheduler = self._GetAppropriateJobScheduler( period )

call = HydrusData.Call( func, *args, **kwargs )

job = ClientThreading.WXAwareRepeatingJob( self, self._job_scheduler, window, initial_delay, period, call )
job = ClientThreading.WXAwareRepeatingJob( self, job_scheduler, window, initial_delay, period, call )

self._job_scheduler.AddJob( job )
job_scheduler.AddJob( job )

return job

Expand Down Expand Up @@ -252,11 +273,19 @@ def CheckMouseIdle( self ):

if move_knocked_us_out_of_idle:

self.gui.SetStatusBarDirty()
self.pub( 'set_status_bar_dirty' )




def ClosePageKeys( self, page_keys ):

with self._page_key_lock:

self._closed_page_keys.update( page_keys )



def CreateSplash( self ):

try:
Expand Down Expand Up @@ -480,18 +509,6 @@ def GetNewOptions( self ):
return self.new_options


def GoodTimeToDoForegroundWork( self ):

if self.gui:

return not self.gui.CurrentlyBusy()

else:

return True



def InitClientFilesManager( self ):

def wx_code( missing_locations ):
Expand Down Expand Up @@ -861,27 +878,19 @@ def MenuIsOpen( self ):
return self._menu_open


def PageCompletelyDestroyed( self, page_key ):
def PageAlive( self, page_key ):

if self.gui:
with self._page_key_lock:

return self.gui.PageCompletelyDestroyed( page_key )

else:

return True
return page_key in self._alive_page_keys



def PageClosedButNotDestroyed( self, page_key ):

if self.gui:

return self.gui.PageClosedButNotDestroyed( page_key )
with self._page_key_lock:

else:

return False
return page_key in self._closed_page_keys



Expand Down Expand Up @@ -914,6 +923,15 @@ def RefreshServices( self ):
self.services_manager.RefreshServices()


def ReleasePageKey( self, page_key ):

with self._page_key_lock:

self._alive_page_keys.discard( page_key )
self._closed_page_keys.discard( page_key )



def ResetIdleTimer( self ):

self._timestamps[ 'last_user_action' ] = HydrusData.GetNow()
Expand Down Expand Up @@ -1381,6 +1399,14 @@ def THREADWait():



def UnclosePageKeys( self, page_keys ):

with self._page_key_lock:

self._closed_page_keys.difference_update( page_keys )



def WaitUntilViewFree( self ):

self.WaitUntilModelFree()
Expand Down
35 changes: 31 additions & 4 deletions include/ClientDB.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10302,6 +10302,36 @@ def _UpdateDB( self, version ):



if version == 307:

try:

domain_manager = self._GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )

domain_manager.Initialise()

#

domain_manager.OverwriteDefaultURLMatches( ( 'tumblr file page', 'artstation file page' ) )

#

domain_manager.TryToLinkURLMatchesAndParsers()

#

self._SetJSONDump( domain_manager )

except Exception as e:

HydrusData.PrintException( e )

message = 'Trying to update some url classes failed! Please let hydrus dev know!'

self.pub_initial_message( message )



self._controller.pub( 'splash_set_title_text', 'updated db to v' + str( version + 1 ) )

self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
Expand Down Expand Up @@ -10933,10 +10963,7 @@ def pub_service_updates_after_commit( self, service_keys_to_service_updates ):

def publish_status_update( self ):

if self._controller.IsBooted() and self._controller.gui:

self._controller.gui.SetStatusBarDirty()

self._controller.pub( 'set_status_bar_dirty' )


def GetInitialMessages( self ):
Expand Down

0 comments on commit 17a6569

Please sign in to comment.