Skip to content

Commit

Permalink
Version 351
Browse files Browse the repository at this point in the history
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed May 8, 2019
1 parent 058559e commit 35ad43c
Show file tree
Hide file tree
Showing 42 changed files with 1,976 additions and 1,563 deletions.
24 changes: 24 additions & 0 deletions help/changelog.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,30 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 351</h3></li>
<ul>
<li>wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements</li>
<li>the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance</li>
<li>improved some hover window focus display calculations to play with the new always-on-top tech</li>
<li>both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result</li>
<li>core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly</li>
<li>the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it</li>
<li>unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug</li>
<li>gave all image processing a refactor and general cleanup pass, deleted a bunch of old code</li>
<li>wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate</li>
<li>sped up how fast the tag parent structure builds itself</li>
<li>the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner</li>
<li>similar-files metadata generation now discards blank frames more reliably</li>
<li>subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in</li>
<li>made the disk cache more conservative on non-pre-processing calls</li>
<li>cleaned up some file import code, moving responsibility from the file locations manager to the file import object</li>
<li>updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable</li>
<li>I believe I fixed a rare vector for the 'tryendmodal' dialog bug</li>
<li>fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name--duplicate-named objects in this case will now be discarded</li>
<li>unified url_match/url_class code differences to url class everywhere</li>
<li>updated some common db list selection code to use new python string formatting</li>
<li>plenty of misc code cleanup</li>
</ul>
<li><h3>version 350</h3></li>
<ul>
<li>the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long</li>
Expand Down
104 changes: 45 additions & 59 deletions include/ClientCaches.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,10 @@ def BuildSimpleChildrenToParents( pairs ):
continue


if LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ): continue
if parent in simple_children_to_parents and LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):

continue


simple_children_to_parents[ child ].add( parent )

Expand Down Expand Up @@ -185,13 +188,16 @@ def LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):

potential_loop_paths = { parent }

while len( potential_loop_paths.intersection( list(simple_children_to_parents.keys()) ) ) > 0:
while True:

new_potential_loop_paths = set()

for potential_loop_path in potential_loop_paths.intersection( list(simple_children_to_parents.keys()) ):
for potential_loop_path in potential_loop_paths:

new_potential_loop_paths.update( simple_children_to_parents[ potential_loop_path ] )
if potential_loop_path in simple_children_to_parents:

new_potential_loop_paths.update( simple_children_to_parents[ potential_loop_path ] )



potential_loop_paths = new_potential_loop_paths
Expand All @@ -200,10 +206,12 @@ def LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):

return True

elif len( potential_loop_paths ) == 0:

return False



return False

class BitmapManager( object ):

MAX_MEMORY_ALLOWANCE = 512 * 1024 * 1024
Expand Down Expand Up @@ -604,11 +612,13 @@ def _GenerateThumbnailBytes( self, file_path, media ):

bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]

target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )

percentage_in = self._controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )

try:

thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, bounding_dimensions, mime, width, height, duration, num_frames, percentage_in = percentage_in )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in )

except Exception as e:

Expand Down Expand Up @@ -956,6 +966,19 @@ def LocklessAddFileFromBytes( self, hash, mime, file_bytes ):



def AddFile( self, hash, mime, source_path, thumbnail_bytes = None ):

with self._rwlock.write:

self._AddFile( hash, mime, source_path )

if thumbnail_bytes is not None:

self._AddThumbnailFromBytes( hash, thumbnail_bytes )




def AddThumbnailFromBytes( self, hash, thumbnail_bytes ):

with self._rwlock.write:
Expand Down Expand Up @@ -1226,47 +1249,6 @@ def GetMissing( self ):
return self._missing_locations


def ImportFile( self, file_import_job ):

if HG.file_report_mode:

HydrusData.ShowText( 'New file import job!' )


( pre_import_status, hash, note ) = file_import_job.GenerateHashAndStatus()

if file_import_job.IsNewToDB():

file_import_job.GenerateInfo()

file_import_job.CheckIsGoodToImport()

( temp_path, thumbnail ) = file_import_job.GetTempPathAndThumbnail()

mime = file_import_job.GetMime()

with self._rwlock.write:

self._AddFile( hash, mime, temp_path )

if thumbnail is not None:

self._AddThumbnailFromBytes( hash, thumbnail )



( import_status, note ) = self._controller.WriteSynchronous( 'import_file', file_import_job )

else:

import_status = pre_import_status


file_import_job.PubsubContentUpdates()

return ( import_status, hash, note )


def LocklessChangeFileExt( self, hash, old_mime, mime ):

old_path = self._GenerateExpectedFilePath( hash, old_mime )
Expand Down Expand Up @@ -1511,9 +1493,9 @@ def LocklessRegenerateThumbnailIfWrongSize( self, media ):

path = self._GenerateExpectedThumbnailPath( hash )

numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )

( current_width, current_height ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( current_width, current_height ) = HydrusImageHandling.GetResolutionNumPy( numpy_image )

bounding_dimensions = self._controller.options[ 'thumbnail_dimensions' ]

Expand Down Expand Up @@ -3262,7 +3244,7 @@ def _GetThumbnailHydrusBitmap( self, display_media ):

try:

numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )

except Exception as e:

Expand All @@ -3282,7 +3264,7 @@ def _GetThumbnailHydrusBitmap( self, display_media ):

try:

numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )

except Exception as e:

Expand All @@ -3294,7 +3276,7 @@ def _GetThumbnailHydrusBitmap( self, display_media ):



( current_width, current_height ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( current_width, current_height ) = HydrusImageHandling.GetResolutionNumPy( numpy_image )

( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( media_width, media_height ), bounding_dimensions )

Expand All @@ -3319,7 +3301,7 @@ def _GetThumbnailHydrusBitmap( self, display_media ):

# this is _resize_, not _thumbnail_, because we already know the dimensions we want
# and in some edge cases, doing getthumbresolution on existing thumb dimensions results in float/int conversion imprecision and you get 90px/91px regen cycles that never get fixed
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, ( expected_width, expected_height ) )

if locations_manager.IsLocal():

Expand All @@ -3334,11 +3316,11 @@ def _GetThumbnailHydrusBitmap( self, display_media ):

try:

thumbnail_bytes = ClientImageHandling.GenerateBytesFromCV( numpy_image, mime )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesNumPy( numpy_image, mime )

except HydrusExceptions.CantRenderWithCVException:

thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromStaticImagePath( path, ( expected_width, expected_height ), mime )


except:
Expand Down Expand Up @@ -3383,7 +3365,7 @@ def _GetThumbnailHydrusBitmap( self, display_media ):

else:

numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, ( expected_width, expected_height ) )

if locations_manager.IsLocal():

Expand Down Expand Up @@ -3555,9 +3537,13 @@ def Clear( self ):

path = os.path.join( HC.STATIC_DIR, name + '.png' )

numpy_image = ClientImageHandling.GenerateNumpyImage( path, HC.IMAGE_PNG )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, HC.IMAGE_PNG )

numpy_image_resolution = HydrusImageHandling.GetResolutionNumPy( numpy_image )

target_resolution = HydrusImageHandling.GetThumbnailResolution( numpy_image_resolution, bounding_dimensions )

numpy_image = ClientImageHandling.ThumbnailNumpyImage( numpy_image, bounding_dimensions )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )

hydrus_bitmap = ClientRendering.GenerateHydrusBitmapFromNumPyImage( numpy_image )

Expand Down

0 comments on commit 35ad43c

Please sign in to comment.