• client api:
  • fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly
  • fixed the 'add_url' api call's instability
  • the API will now always return JSON on 200. anything else should be presumed to be raw text
  • '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week
  • '/request_access_permissions' now returns JSON
  • '/add_url' now results JSON on success with more info, 403 on failure
  • '/get_url_info' now returns the 'normalised_url' in the response JSON
  • added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url
  • added '/add_files/add_file', which can import a file from a path or bytes
  • added '/add_tags/get_tag_services', which will return info on the client's tag services
  • updated client api help to reflect the above changes and fleshed out the intro a bit
  • fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time
  • updated the client api tests to check the above
  • refactored client api tests to be neater and in their own file
  • .
  • the rest:
  • fixed the page of pages close bug
  • added a downloader for nijie.info to the client defaults (it needs a login)
  • updated danbooru file page parsers to get 'rating' tag
  • added gelbooru 0.1.11 parser for future application
  • fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction
  • added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)
  • wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest '�' replacement characters
  • the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks
  • moved some more old daemons to the new job scheduler, deleted some old daemon code
  • improved some daemon job wake and shutdown code
  • wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code
  • simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update
  • fixed upnp mapping fetching to cope with ipv6 results
  • improved some memory clearing code to deal with some semi-stubborn objects
  • improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine
  • improved menubar stability, both in finding menus and swapping them out
  • if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup
  • fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions
  • fixed two login invalid cookie error handling bugs
  • maybe made some dupe filter searching more stable
  • fixed a py2 datatype issue that made the client unbootable when updating the client from <296
  • the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go
  • slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB
  • added experimental secret discord bugfix dnd mode checkbox
  • improved how html parsing deals with some unexpected bad tag data
  • turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays
  • wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work
  • improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up--this is an ongoing project
  • improved how some text import line splitting works
  • misc fixes
Assets 6

@hydrusnetwork hydrusnetwork released this Feb 6, 2019 · 1 commit to master since this release

  • client api:
  • wrote some ui to handle client api permissions adding and editing
  • wrote a 'catch a permissions request' mini-dialog for external api permissions adding
  • wrote api calls:
  • GET
  • /api_version
  • /request_new_permissions
  • /verify_access_key
  • /add_urls/get_url_info
  • POST
  • /add_urls/add_url
  • and made a new 'client api' help page to describe in detail what these do
  • wrote fairly comprehensive unit tests for the new client api
  • refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses
  • neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors
  • deleted very old prototype file/thumbnail client server fetch code, which was no longer in use
  • .
  • misc and bug fixes:
  • added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or all files
  • the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)
  • the file url downloader now reports 'downloading file' and 'importing url' text status separately
  • fixed a typo bug from last week that was breaking asc/desc ratings service sorting
  • fixed a typo bug from last week that was stopping manage import folders from opening
  • fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel
  • the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)
  • the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension
  • fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters
  • the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)
  • fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well
  • maybe cleaned up some menubar management code
  • the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook
  • fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running
  • improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into
  • improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save
  • improved thread-safety of serialisable objects as they serialise
  • misc improvements and cleanup
Assets 6

@hydrusnetwork hydrusnetwork released this Jan 30, 2019 · 2 commits to master since this release

  • after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate asynchronously, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios should be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc...
  • I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here
  • wildcard search tags now have an explicit '(wildcard search)' after their label
  • taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!
  • added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side
  • some of the file sort options now default to their respective 'biggest' first, see how you like it
  • folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools
  • if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging
  • fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view
  • fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked
  • wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'
  • the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively
  • updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made
  • removed some old experimental crypto code that isn't used any more--pycryptodome is no longer needed to run the program from source
  • improved some misc client service code
  • deleted some old unused code
  • misc cleanup
  • updated to ffmpeg 4.1 on windows release
  • updated to sqlite 3.26 on windows release
  • updated to wxpython 4.0.4 on all platforms
Assets 6

@hydrusnetwork hydrusnetwork released this Jan 24, 2019 · 3 commits to master since this release

  • fixed another couple of unicode encoding problems with the logging and profiling code
  • the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding
  • fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video
  • fixed some 'max size' download file size testing
  • 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)
  • the downloader easy-importer lain image is now clickable to launch a file selection dialog
  • if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you
  • duplicate content merging now applies to pending as well as current tags
  • the duplicate filter now counts pending tags when saying which file has more tags
  • advanced content updates now copy both current and pending tags. the other actions now state what they do more clearly
  • stopped printing long server error text on 304 and 404, where actual response content is uninteresting
  • removed wx import that accidentally came into server space due to 2to3 check
  • improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds
  • fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else
  • improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements
  • did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations
  • reduced db disk cache aggression significantly and added more memory maintenance to the cache population process
  • fixed an issue where subscriptions were not promptly responding to shutdown events
  • fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events
  • added a 'pubsub report mode' debug mode for simpler pubsub review
  • the db is now less redundantly spammy on certain behind the scenes update notifications
  • wrote a first version of the client api manager and permissions handling objects
  • misc fixes, cleaned some shutdown code
Assets 6

@hydrusnetwork hydrusnetwork released this Jan 16, 2019 · 4 commits to master since this release

  • fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)
  • fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases
  • fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)
  • fixed a str vs bytes issue when loading the filename tagging panel
  • fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)
  • improved some json dump deserialisation code
  • fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel
  • improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data
  • fixed some hex character processing for system preds
  • added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)
  • the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation
  • made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update
  • the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system
  • simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method
  • wrote some new help.txt in the db dir about hanging startups
Assets 6

@hydrusnetwork hydrusnetwork released this Jan 10, 2019 · 5 commits to master since this release

The first version of this release had no files as Github's file upload system wasn't working right and I released on Mediafire. It is now working again, and I have added the files.

  • important:
  • hydrus now runs completely and exclusively on python 3!
  • for users who are updating, the client has special install instructions for just this week:
  • if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)
  • if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)
  • if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!
  • if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out
  • please check the release post for more detailed instructions here
  • .
  • semi-important:
  • the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from чики брики lads appreciated
  • I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)
  • although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source
  • I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries
  • .
  • misc bug fixes:
  • fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class
  • removed some popup flicker related to long error messages
  • fixed some parsing testing ui error handling
  • cleared up some bad text ctrl event handling that could sometimes cause a recursive loop
  • listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion
  • cleaned up some bad external process calling code and improved how child process environment is set up
  • finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this
  • .
  • boring details:
  • ran 2to3 to auto-convert what could be done
  • updated environment to python 3
  • went over a whole ton of unicode encoding/decoding manually to update it to python 3
  • removed all the old tobytestring/tounicode calls in favour of new python 3 handling
  • fixed all the file io to do bytes/str as appropriate
  • corrected a bunch of / vs // int/float stuff
  • fixed up twisted, which has some str/bytes stuff going on
  • fixed all the listctrls to deal with column sorting None values amongst ints/strs
  • fixed png export/import, which had some fun bytes/bytearray/int/str issues
  • updated the swf header parsing code to py3 (more str/bytes stuff)
  • misc float/int fixes
  • fixed up some http.cookies handling, which has changed in py3
  • improved some ancient loopback connection code that was only really checking to see if ports were in use
  • cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked
  • numerous misc other refactoring and so on
  • updated the new network engine to now decode non-utf-8 responses correctly based on actual response header
  • removed some old py2 manual http multipart code
  • removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection
  • replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess
  • deleted some more misc old code
  • tightened up some spammy network error reporting
  • converted all /r/n to /n in my environment project, ha ha ha
  • the ui seems to better support rarer unicode characters like 🔊
  • updated some of the install/update/backup help for all this, and some misc other stuff as well
  • fixed misc bugs
Assets 6

@hydrusnetwork hydrusnetwork released this Dec 12, 2018 · 6 commits to master since this release

  • wrote a system:file viewing stats to comprehensively search the new viewing stats--it should also be synced with the exact current values
  • but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!
  • added a database->file viewing stats menu that lets you suspend file view tracking and clear all records permanently
  • mr. bones now welcomes all users under the help menu
  • fixed mr. bones's confusion at those who have yet to board the ride
  • also mr. bones now makes sure to get the latest file viewing stats
  • moved confirm trash/archive from options->gui to options->files and trash
  • moved a bunch of 'pages' related stuff from options->gui to the new options->gui pages
  • added an option to options->gui pages to change the number of session rolling backups
  • subscription popups now provide an x/y query progress string in their popup text
  • the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default
  • fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import
  • the client will now print up to 512KB of server error info to the log (previously 4KB)
  • removed the youtube download prototype--if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client
  • network report mode now shows more network error information
  • gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience
  • wrote a 'logins' section to the bottom of the 'getting started with downloading' help page
  • misc fixes
Assets 7

@hydrusnetwork hydrusnetwork released this Dec 5, 2018 · 7 commits to master since this release

  • added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!
  • you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media
  • viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)
  • you can now sort files by total media views/viewtime!
  • mr. bones's wild ride continues, as well
  • deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!
  • tag siblings refresh is quicker and less memory heavy thanks to this as well
  • furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results
  • the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!
  • tag and rating sort options are now listed as 'tag:' and 'rating:' respectively
  • fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations
  • fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist
  • the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist
  • cleaned up some of the hover_window-canvas interaction code
  • fixed some long-time sperg-out buffer-drawing when changing position in a long video
  • the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate
  • fixed some ancient repository admin code that fetches summary account info given an account key
  • the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors
  • fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation
Assets 7

@hydrusnetwork hydrusnetwork released this Nov 28, 2018 · 8 commits to master since this release

  • the client serialisation system now supports multiple rolling backups!
  • client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from pages->sessions->append session backup, where they are listed by their timestamp!
  • when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations
  • gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload
  • tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch
  • under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively
  • attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process
  • did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
  • wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster
  • export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.
  • fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work
  • when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!
  • the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads
  • adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well
  • fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices
  • fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add
  • the 'scroll thumbs at this rate per tick' option now rounds the pixel result, rather than always _floor_ing it
  • wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places
  • improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service
  • fixed a small typo bug when trying to auto-add url classes and parsers
  • updated an old deprecated checkboxlist call
  • clarified the integrity section of 'help my db is broke.txt'.
  • misc fixes
Assets 7

@hydrusnetwork hydrusnetwork released this Nov 21, 2018 · 9 commits to master since this release

  • added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal
  • 'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies
  • the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this
  • login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)
  • brushed up some of the 'change login script' code--particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection
  • after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong
  • prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes
  • added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!
  • updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org should all be updated right. exhentai.org is likely too difficult to properly support in the current system
  • the different panels where you enter system predicate information now all run on the new panel sizing system--if you have had problems with these, please let me know how they size now!
  • added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens
  • the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files--presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.
  • misc ui improvements
  • updated 'running from source' in help
Assets 7