Skip to content

Commit

Permalink
Version 341
Browse files Browse the repository at this point in the history
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed Feb 27, 2019
1 parent 0f516b2 commit 4b6fc54
Show file tree
Hide file tree
Showing 56 changed files with 1,732 additions and 436 deletions.
44 changes: 43 additions & 1 deletion help/changelog.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,48 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 341</h3></li>
<ul>
<li>client api:</li>
<li>added /add_tags/add_tags, which does several kinds of tag content updates</li>
<li>added /add_tags/clean_tags, which shows how hydrus will handle potential tags</li>
<li>added /add_urls/associate_url, which allows you to associate urls with files</li>
<li>added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)</li>
<li>updated client api version to 2</li>
<li>updated client help and unit tests for the above</li>
<li>added a linked contents to the client api help</li>
<li>improved some server error handling, mostly moving 403s to more correct 400s</li>
<li>improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500</li>
<li>.</li>
<li>the rest:</li>
<li>tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user</li>
<li>when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)</li>
<li>added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details</li>
<li>reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements</li>
<li>the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows</li>
<li>when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8</li>
<li>improved serialisability of some pending tag data</li>
<li>watchers can now hold and pass on fixed pending tag data</li>
<li>gallery log objects can now hold and pass on fixed pending tag data</li>
<li>file import objects can now hold and action fixed pending tag data</li>
<li>hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects</li>
<li>improved some url-import page drop-target-selection logic</li>
<li>improved error reporting when dropping/api-adding urls</li>
<li>adjusted some url import workflow so big 'already in db' download lists should work a bit faster</li>
<li>attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made</li>
<li>tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth</li>
<li>fixed a source of linux CRITICAL logspam related to several common dialogs</li>
<li>improved ui stability on boot when file folders are missing (particularly for linux)</li>
<li>improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better</li>
<li>did some more misc stability improvements, particularly in various boot fail scenarios</li>
<li>completely removed an ancient and janky focus catcher widget from the main gui frame</li>
<li>now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory--hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large</li>
<li>fixed approx bitrate sorting for malformed video files with explicitly '0' duration</li>
<li>daemon_profile_mode now spams some more info about export folders</li>
<li>fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status</li>
<li>updated windows build to cv 4.0</li>
<li>misc refactoring and fixes</li>
</ul>
<li><h3>version 340</h3></li>
<ul>
<li>client api:</li>
Expand All @@ -33,7 +75,7 @@ <h3>changelog</h3>
<li>added gelbooru 0.1.11 parser for future application</li>
<li>fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction</li>
<li>added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)</li>
<li>wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest ' 'replacement characters</li>
<li>wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest '�' replacement characters</li>
<li>the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks</li>
<li>moved some more old daemons to the new job scheduler, deleted some old daemon code</li>
<li>improved some daemon job wake and shutdown code</li>
Expand Down
198 changes: 186 additions & 12 deletions help/client_api.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,36 @@ <h3>Support created by hydrus users:</h3>
<li><a href="https://gitlab.com/cryzed/hydrus-api">https://gitlab.com/cryzed/hydrus-api</a> - A python module that talks to the API.</li>
</ul>
<h3>API</h3>
<p>The API should always return JSON on 200. Otherwise, assume it will return plain text, sometimes a raw traceback. You'll typically get 400 for a missing parameter, 401 or 403 for missing/insufficient access, and 500 for a real deal serverside error.</p>
<p>If the API returns anything on 200, it should always return JSON. Otherwise, assume it will return plain text, sometimes a raw traceback. You'll typically get 400 for a missing parameter, 401 or 403 for missing/insufficient access, and 500 for a real deal serverside error.</p>
<h3>Contents</h3>
<ul>
<li>
<h4>Access Management</h4>
<ul>
<li><a href="#api_version">GET /api_version</a></li>
<li><a href="#request_new_permissions">GET /request_new_permissions</a></li>
<li><a href="#verify_access_key">GET /verify_access_key</a></li>
</ul>
<h4>Adding Files</h4>
<ul>
<li><a href="#add_files_add_file">POST /add_files/add_file</a></li>
</ul>
<h4>Adding Tags</h4>
<ul>
<li><a href="#add_tags_clean_tags">GET /add_tags/clean_tags</a></li>
<li><a href="#add_tags_get_tag_services">GET /add_tags/get_tag_services</a></li>
<li><a href="#add_tags_add_tags">POST /add_tags/add_tags</a></li>
</ul>
<h4>Adding URLs</h4>
<ul>
<li><a href="#add_urls_get_url_files">GET /add_urls/get_url_files</a></li>
<li><a href="#add_urls_get_url_info">GET /add_urls/get_url_info</a></li>
<li><a href="#add_urls_add_url">POST /add_urls/add_url</a></li>
<li><a href="#add_urls_associate_url">POST /add_urls/associate_url</a></li>
</ul>
</ul>
<h3>Access Management</h3>
<div class="apiborder">
<div class="apiborder" id="api_version">
<h3><b>GET /api_version</b></h3>
<p><i>Gets the current API version. I will increment this every time I alter the API.</i></p>
<ul>
Expand All @@ -37,7 +64,7 @@ <h3><b>GET /api_version</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="request_new_permissions">
<h3><b>GET /request_new_permissions</b></h3>
<p><i>Register a new external program with the client. This requires the 'add from api request' mini-dialog under </i>services->review services<i> to be open, otherwise it will 403.</i></p>
<ul>
Expand Down Expand Up @@ -73,7 +100,7 @@ <h3><b>GET /request_new_permissions</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="verify_access_key">
<h3><b>GET /verify_access_key</b></h3>
<p><i>Check your access key is valid.</i></p>
<ul>
Expand All @@ -99,7 +126,7 @@ <h3><b>GET /verify_access_key</b></h3>
</ul>
</div>
<h3>Adding Files</h3>
<div class="apiborder">
<div class="apiborder" id="add_files_add_file">
<h3><b>POST /add_files/add_file</b></h3>
<p><i>Tell the client to import a file.</i></p>
<ul>
Expand Down Expand Up @@ -139,7 +166,44 @@ <h3><b>POST /add_files/add_file</b></h3>
</ul>
</div>
<h3>Adding Tags</h3>
<div class="apiborder">
<div class="apiborder" id="add_tags_clean_tags">
<h3><b>GET /add_tags/clean_tags</b></h3>
<p><i>Ask the client about how it will see certain tags.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
</ul>
</li>
<li><p>Arguments (in percent-encoded JSON):</p></li>
<ul>
<li>tags : (a list of the tags you want cleaned)</li>
</ul>
<li>
<p>Example request:</p>
<pre>Given tags [ " bikini ", "blue eyes", " character : samus aran ", ":)", " ", "", "10", "11", "9", "system:wew", "-flower" ]:</pre>
<ul>
<li><p>/add_tags/clean_tags?tags=%5B%22%20bikini%20%22%2C%20%22blue%20%20%20%20eyes%22%2C%20%22%20character%20%3A%20samus%20aran%20%22%2C%20%22%3A%29%22%2C%20%22%20%20%20%22%2C%20%22%22%2C%20%2210%22%2C%20%2211%22%2C%20%229%22%2C%20%22system%3Awew%22%2C%20%22-flower%22%5D</p></li>
</ul>
</li>
<li>
<p>Response description: The tags cleaned according to hydrus rules. They will also be in hydrus human-friendly sorting order.</p>
</li>
<li>
<p>Example response:</p>
<ul>
<li>
<pre>{
"tags": [ "9", "10", "11", "::)", "bikini", "blue eyes", "character:samus aran", "flower", "wew" ]
}</pre>
</li>
</ul>
<p>Mostly, hydrus simply trims excess whitespace, but the other examples are rare issues you might run into. 'system' is an invalid namespace, tags cannot be prefixed with hyphens, and any tag starting with ':' is secretly dealt with internally as "[no namespace]:[colon-prefixed-subtag]". Again, you probably won't run into these, but if you see a mismatch somewhere and want to figure it out, or just want to sort some numbered tags, you might like to try this.</p>
</li>
</ul>
</div>
<div class="apiborder" id="add_tags_get_tag_services">
<h3><b>GET /add_tags/get_tag_services</b></h3>
<p><i>Ask the client about its tag services.</i></p>
<ul>
Expand Down Expand Up @@ -167,11 +231,74 @@ <h3><b>GET /add_tags/get_tag_services</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="add_tags_add_tags">
<h3><b>POST /add_tags/add_tags</b></h3>
<p><i>Make changes to the tags that files have.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
</ul>
</li>
<li><p>Arguments (in JSON):</p></li>
<ul>
<li>hash : (an SHA256 hash for a file in 64 characters of hexadecimal)</li>
<li>hashes : (a list of SHA256 hashes)</li>
<li>service_names_to_tags : (an Object of service names to lists of tags to be 'added' to the files)</li>
<li>service_names_to_actions_to_tags : (an Object of service names to content update actions to lists of tags)</li>
</ul>
<p>You can use either 'hash' or 'hashes', and you can use either the simple add-only 'service_names_to_tags' or the advanced 'service_names_to_actions_to_tags'.</p>
<p>The service names are as in the <i>/add_tags/get_tag_services</i> call.</p>
<p>The permitted 'actions' are:</p>
<ul>
<li>0 - Add to a local tag service.</li>
<li>1 - Delete from a local tag service.</li>
<li>2 - Pend to a tag repository.</li>
<li>3 - Rescind a pend from a tag repository.</li>
<li>4 - Petition from a tag repository. (This is special)</li>
<li>5 - Rescind a petition from a tag repository.</li>
</ul>
<p>When you petition a tag from a repository, a 'reason' for the petition is typically needed. If you send a normal list of tags here, a default reason of "Petitioned from API" will be given. If you want to set your own reason, you can instead give a list of [ tag, reason ] pairs.</p>
<p>Some example requests:</p>
<p>Adding some tags to a file:</p>
<pre>{
"hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
"service_names_to_tags" : {
"local tags" : [ "character:supergirl", "rating:safe" ]
}
}</pre>
<p>Adding more tags to two files:</p>
<pre>{
"hashes" : [ "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56", "f2b022214e711e9a11e2fcec71bfd524f10f0be40c250737a7861a5ddd3faebf" ],
"service_names_to_tags" : {
"local tags" : [ "process this" ],
"public tag repository" : [ "creator:dandon fuga" ]
}
}</pre>
<p>A complicated transaction with all possible actions:</p>
<pre>{
"hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
"service_names_to_actions_to_tags" : {
"local tags" : {
0 : [ "character:supergirl", "rating:safe" ],
1 : [ "character:superman" ]
},
"public tag repository" : {
2 : [ "character:supergirl", "rating:safe" ],
3 : [ "filename:image.jpg" ],
4 : [ [ "creator:danban faga", "typo" ], [ "character:super_girl", "underscore" ] ]
5 : [ "skirt" ]
}
}
}</pre>
<p>This last example is far more complicated than you will usually see. Pend rescinds and petition rescinds are not common. Petitions are also quite rare, and gathering a good petition reason for each tag is often a pain.</p>
<p>Response description: 200 and no content.</p>
<p>Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended and not worry about an error--it will be discarded. The same for other reasonable logical scenarios: deleting a tag that does not exist will silently make no change, pending a tag that is already 'current' will again be passed over. It is fine to just throw 'process this' tags at every file import you add and not have to worry about checking which files you already added it to.</p>
</ul>
</div>
<h3>Adding URLs</h3>
<div class="apiborder">
<div class="apiborder" id="add_urls_get_url_files">
<h3><b>GET /add_urls/get_url_files</b></h3>
<p><i>Ask the client about an URL's files.</i></p>
<ul>
Expand Down Expand Up @@ -223,7 +350,7 @@ <h3><b>GET /add_urls/get_url_files</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="add_urls_get_url_info">
<h3><b>GET /add_urls/get_url_info</b></h3>
<p><i>Ask the client for information about a URL.</i></p>
<ul>
Expand Down Expand Up @@ -273,7 +400,7 @@ <h3><b>GET /add_urls/get_url_info</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="add_urls_add_url">
<h3><b>POST /add_urls/add_url</b></h3>
<p><i>Tell the client to 'import' a URL. This triggers the exact same routine as drag-and-dropping a text URL onto the main client window.</i></p>
<ul>
Expand All @@ -288,12 +415,24 @@ <h3><b>POST /add_urls/add_url</b></h3>
<p>Arguments (in JSON):</p>
<ul>
<li>url : (the url you want to add)</li>
<li>destination_page_name : (optional page name to receive the url)</li>
<li>service_names_to_tags : (optional tags to give to any files imported from this url)</li>
</ul>
</li>
<p>If you specify a destination_page_name and an appropriate importer page already exists with that name, that page will be used. Otherwise, a new page with that name will be recreated (and used by subsequent calls with that name). Make sure it that page name is unique (e.g. '/b/ threads', not 'watcher') in your client, or it may not be found.</p>
<p>The service_names_to_tags uses the same system as for /add_tags/add_tags. You will need 'add tags' permission, or this will 403.</p>
<li>
<p>Example request body:</p>
<ul>
<li><p>{"url": "https://8ch.net/tv/res/1846574.html"}</p></li>
<li>
<pre>{
"url": "https://8ch.net/tv/res/1846574.html",
"destination_page_name": "kino zone",
"service_names_to_tags": {
"local tags" : [ "as seen on /tv/" ]
}
}</pre>
</li>
</ul>
</li>
<li><p>Response description: Some JSON with info on the URL added.</p></li>
Expand All @@ -310,8 +449,43 @@ <h3><b>POST /add_urls/add_url</b></h3>
</li>
</ul>
</div>
<div class="apiborder">
<div class="apiborder" id="add_urls_associate_url">
<h3><b>POST /add_urls/associate_url</b></h3>
<p><i>Manage which URLs the client considers to be associated with which files.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
<li>Content-Type : application/json</li>
</ul>
</li>
<li>
<p>Arguments (in JSON):</p>
<ul>
<li>url_to_add : (an url you want to associate with the file(s))</li>
<li>urls_to_add : (a list of urls you want to associate with the file(s))</li>
<li>url_to_delete : (an url you want to disassociate from the file(s))</li>
<li>urls_to_delete : (a list of urls you want to disassociate from the file(s))</li>
<li>hash : (an SHA256 hash for a file in 64 characters of hexadecimal)</li>
<li>hashes : (a list of SHA256 hashes)</li>
</ul>
</li>
<p>All of these are optional, but you obviously need to have at least one of 'url' arguments and one of the 'hash' arguments. Unless you really know what you are doing with URL Classes, I strongly recommend you stick to associating URLs with just one single 'hash' at a time. Multiple hashes pointing to the same URL is unusual and frequently unhelpful.</p>
<li>
<p>Example request body:</p>
<ul>
<li>
<pre>{
"url_to_add": "https://rule34.xxx/index.php?id=2588418&page=post&s=view",
"hash": "3b820114f658d768550e4e3d4f1dced3ff8db77443472b5ad93700647ad2d3ba"
}
}</pre>
</li>
</ul>
</li>
<li><p>Response description: 200 with no content. Like when adding tags, this is safely idempotent--do not worry about re-adding URLs associations that already exist or accidentally trying to delete ones that don't.</p></li>
</ul>
</div>
<h3>Searching Files</h3>
<div class="apiborder">
Expand Down
2 changes: 1 addition & 1 deletion help/running_from_source.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ <h3>what you will need</h3>
<p>That '. venv/bin/activate' line turns your venv on, and will be needed every time you run the client.pyw/server.py files. You can easily tuck it into a launch script.</p>
<p>After that, you can go nuts with pip. I think this will do for most systems:</p>
<ul>
<li>pip3 install beautifulsoup4 html5lib lxml nose numpy opencv-python six Pillow psutil PyOpenSSL PyYAML requests Send2Trash service_identity twisted</li>
<li>pip3 install beautifulsoup4 chardet html5lib lxml nose numpy opencv-python six Pillow psutil PyOpenSSL PyYAML requests Send2Trash service_identity twisted</li>
</ul>
<p>You may want to do all that in smaller batches.</p>
<p>And optionally, you can add these packages:</p>
Expand Down
Loading

0 comments on commit 4b6fc54

Please sign in to comment.