Skip to content

Commit

Permalink
Version 343
Browse files Browse the repository at this point in the history
  • Loading branch information
@hydrusnetwork
hydrusnetwork committed Mar 13, 2019
1 parent 1a7fe45 commit 67e39d8
Show file tree
Hide file tree
Showing 37 changed files with 2,031 additions and 1,124 deletions.
23 changes: 23 additions & 0 deletions help/changelog.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,29 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 343</h3></li>
<ul>
<li>client api:</li>
<li>fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working</li>
<li>fixed up some last /get_files/search_files stuff</li>
<li>added /get_files/file_metadata</li>
<li>added /get_files/file</li>
<li>added /get_files/thumbnail</li>
<li>added help and unit tests to reflect the above</li>
<li>updated client api version to 4</li>
<li>.</li>
<li>the rest:</li>
<li>the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired</li>
<li>animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media</li>
<li>increased aggression of media viewer image prefetch</li>
<li>added support for 'MM' Tiffs</li>
<li>fixed webm mime parsing for webms with no audio (these were falling back to mkv)</li>
<li>improved the error reports when a serialised png fails to import</li>
<li>the mass-open-urls popup is now pausable as well as cancellable</li>
<li>fixed several recently broken ui unit tests</li>
<li>misc old code cleanup</li>
<li>some misc test controller/constant refactoring</li>
</ul>
<li><h3>version 342</h3></li>
<ul>
<li>added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame</li>
Expand Down
195 changes: 175 additions & 20 deletions help/client_api.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
</head>
<body>
<div class="content">
<p class="warning">This is all currently under construction!</p>
<h3>client api</h3>
<p>The hydrus client now supports a very simple API so you can access it with external programs.</p>
<p>By default, the Client API is not turned on. Go to <i>services->manage services</i> and give it a port to get it started. I recommend you not allow non-local connections (i.e. only requests from the same computer will work) to start with.</p>
Expand Down Expand Up @@ -50,6 +49,9 @@ <h4>Adding URLs</h4>
<h4>Searching and Fetching Files</h4>
<ul>
<li><a href="#get_files_search_files">POST /get_files/search_files</a></li>
<li><a href="#get_files_file_metadata">POST /get_files/file_metadata</a></li>
<li><a href="#get_files_file">POST /get_files/file</a></li>
<li><a href="#get_files_thumbnail">POST /get_files/thumbnail</a></li>
</ul>
</ul>
<h3>Access Management</h3>
Expand Down Expand Up @@ -285,18 +287,19 @@ <h3><b>POST /add_tags/add_tags</b></h3>
"hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
"service_names_to_actions_to_tags" : {
"local tags" : {
0 : [ "character:supergirl", "rating:safe" ],
1 : [ "character:superman" ]
"0" : [ "character:supergirl", "rating:safe" ],
"1" : [ "character:superman" ]
},
"public tag repository" : {
2 : [ "character:supergirl", "rating:safe" ],
3 : [ "filename:image.jpg" ],
4 : [ [ "creator:danban faga", "typo" ], [ "character:super_girl", "underscore" ] ]
5 : [ "skirt" ]
"2" : [ "character:supergirl", "rating:safe" ],
"3" : [ "filename:image.jpg" ],
"4" : [ [ "creator:danban faga", "typo" ], [ "character:super_girl", "underscore" ] ]
"5" : [ "skirt" ]
}
}
}</pre>
<p>This last example is far more complicated than you will usually see. Pend rescinds and petition rescinds are not common. Petitions are also quite rare, and gathering a good petition reason for each tag is often a pain.</p>
<p>Note that the enumerated status keys in the service_names_to_actions_to_tags structure are strings, not ints (JSON does not support int keys for Objects).</p>
<p>Response description: 200 and no content.</p>
<p>Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended and not worry about an error--it will be discarded. The same for other reasonable logical scenarios: deleting a tag that does not exist will silently make no change, pending a tag that is already 'current' will again be passed over. It is fine to just throw 'process this' tags at every file import you add and not have to worry about checking which files you already added it to.</p>
</ul>
Expand Down Expand Up @@ -491,9 +494,9 @@ <h3><b>POST /add_urls/associate_url</b></h3>
</ul>
</div>
<h3>Searching Files</h3>

<p>File search in hydrus is not paginated like a booru--all searches return all results in one go. In order to keep this fast, search is split into two steps--fetching file identifiers with a search, and then fetching file metadata in batches. You may have noticed that the client itself performs searches like this--thinking a bit about a search and then bundling results in batches of 256 files before eventually throwing all the thumbnails on screen.</p>
<div class="apiborder" id="get_files_search_files">
<h3><b>POST /get_files/search_files</b></h3>
<h3><b>GET /get_files/search_files</b></h3>
<p><i>Search for the client's files.</i></p>
<ul>
<li>
Expand All @@ -506,8 +509,8 @@ <h3><b>POST /get_files/search_files</b></h3>
<p>Arguments (in percent-encoded JSON):</p>
<ul>
<li>tags : (a list of tags you wish to search for)</li>
<li>system_inbox : true or false (optional, defaulting to false)
<li>system_archive : true or false (optional, defaulting to false)
<li>system_inbox : true or false (optional, defaulting to false)</li>
<li>system_archive : true or false (optional, defaulting to false)</li>
</ul>
</li>
<li>
Expand All @@ -528,21 +531,173 @@ <h3><b>POST /get_files/search_files</b></h3>
</li>
</ul>
</li>
<p>File ids are internal and specific to an individual client. For a client, a file with hash H always has the same file id N, but two clients will have different ideas about which N goes with which H. They are a bit faster than hashes to retrieve and search with <i>en masse</i>, which is why they are exposed here.</p>
<p>The search will be performed on the 'local files' file domain and 'all known tags' tag domain.</p>
<p>Note that most clients will have an invisible system:limit of 10,000 files on all queries. I expect to add more system predicates to help searching for untagged files, but it is tricky to fetch all files under any circumstance. Large queries may take several seconds to respond.</p>
</ul>
</div>
<div class="apiborder">
<h3><b>GET file metadata</b></h3>
# id to info objects
# hashes only arg
<div class="apiborder" id="get_files_file_metadata">
<h3><b>GET /get_files/file_metadata</b></h3>
<p><i>Get metadata about files in the client.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
</ul>
</li>
<li>
<p>Arguments (in percent-encoded JSON):</p>
<ul>
<li>file_ids : (a list of numerical file ids)</li>
<li>hashes : (a list of hexadecimal SHA256 hashes)</li>
<li>only_return_identifiers : true or false (optional, defaulting to false)</li>
</ul>
</li>
<p>You need one of file_ids or hashes. If your access key is restricted by tag, you cannot search by hashes, and <b>the file_ids you search for must have been in the most recent search result</b>.</p>
<li>
<p>Example request for two files with ids 123 and 4567:</p>
<ul>
<li><p>/get_files/file_metadata?file_ids=%5B123%2C%204567%5D</p></li>
</ul>
</li>
<li>
<p>The same, but only wants hashes back:</p>
<ul>
<li><p>/get_files/file_metadata?file_ids=%5B123%2C%204567%5D&only_return_identifiers=true</p></li>
</ul>
</li>
<li>
<p>And one that fetches two hashes, 4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2 and 3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82:</p>
<ul>
<li><p>/get_files/file_metadata?hashes=%5B%224c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2%22%2C%20%223e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82%22%5D</p></li>
</ul>
</li>
<p>This request string can obviously get pretty ridiculously long. It also takes a bit of time to fetch metadata from the database. In its normal searches, the client usually fetches file metadata in batches of 256.</p>
<p>Response description: A list of JSON Objects that store a variety of file metadata.</p>
<li>
<p>Example response:</p>
<ul>
<li>
<pre>{
"metadata" : [
{
"file_id" : 123,
"hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
"size" : 63405,
"mime" : "image/jpg",
"width" : 640,
"height" : 480,
"duration" : null,
"num_frames" : null,
"num_words" : null,
"service_names_to_statuses_to_tags" : {}
},
{
"file_id" : 4567,
"hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
"size" : 199713,
"mime" : "video/webm",
"width" : 1920,
"height" : 1080,
"duration" : 4040,
"num_frames" : 102,
"num_words" : null
"service_names_to_statuses_to_tags" : {
"local tags" : {
"0" : [ "blonde hair", "blue eyes", "looking at viewer" ]
"1" : [ "bodysuit" ]
}
}
}
]
}</pre>
</li>
</ul>
<p>And one where only_return_identifiers is true:</p>
<ul>
<li>
<pre>{
"metadata" : [
{
"file_id" : 123,
"hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2"
},
{
"file_id" : 4567,
"hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82"
}
]
}</pre>
</li>
</ul>
</li>
<p>Size is in bytes. Duration is in milliseconds, and may be an int or a float.</p>
<p>The tags structure is similar to the /add_tags/add_tags scheme, excepting that the status numbers are:</p>
<ul>
<li>0 - current</li>
<li>1 - pending</li>
<li>2 - deleted</li>
<li>3 - petitioned</li>
</ul>
<p>Note that since JSON Object keys must be strings, these status numbers are strings, not ints.</p>
</ul>
</div>
<div class="apiborder">
<h3><b>GET file</b></h3>
<div class="apiborder" id="get_files_file">
<h3><b>GET /get_files/file</b></h3>
<p><i>Get a file.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
</ul>
</li>
<li>
<p>Arguments :</p>
<ul>
<li>file_id : (numerical file id for the file)</li>
<li>hash : (a hexadecimal SHA256 hash for the file)</li>
</ul>
</li>
<p>Only use one. As with metadata fetching, you may only use the hash argument if you have access to all files. If you are tag-restricted, you will have to use a file_id in the last search you ran.</p>
<li>
<p>Example requests:</p>
<ul>
<li><p>/get_files/file?file_id=452158</p></li>
<li><p>/get_files/file?hash=7f30c113810985b69014957c93bc25e8eb4cf3355dae36d8b9d011d8b0cf623a</p></li>
</ul>
</li>
<li><p>Response description: The file itself. You should get the correct mime type as the Content-Type header.</p></li>
</ul>
</div>

<div class="apiborder">
<h3><b>GET thumbnail</b></h3>
<div class="apiborder" id="get_files_thumbnail">
<h3><b>GET /get_files/thumbnail</b></h3>
<p><i>Get a file's thumbnail.</i></p>
<ul>
<li>
<p>Headers:</p>
<ul>
<li>Hydrus-Client-API-Access-Key : (Your hexadecimal access key)</li>
</ul>
</li>
<li>
<p>Arguments :</p>
<ul>
<li>file_id : (numerical file id for the file)</li>
<li>hash : (a hexadecimal SHA256 hash for the file)</li>
</ul>
</li>
<p>Only use one. As with metadata fetching, you may only use the hash argument if you have access to all files. If you are tag-restricted, you will have to use a file_id in the last search you ran.</p>
<li>
<p>Example requests:</p>
<ul>
<li><p>/get_files/thumbnail?file_id=452158</p></li>
<li><p>/get_files/thumbnail?hash=7f30c113810985b69014957c93bc25e8eb4cf3355dae36d8b9d011d8b0cf623a</p></li>
</ul>
</li>
<li><p>Response description: The thumbnail for the file. It will give application/octet-stream as the mime type. Some hydrus thumbs are jpegs, some are pngs.</p></li>
</ul>
</div>
</div>
</body>
Expand Down
2 changes: 1 addition & 1 deletion help/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ <h3>hydrus help</h3>
<li><a href="advanced_siblings.html">advanced usage - tag siblings</a></li>
<li><a href="advanced_parents.html">advanced usage - tag parents</a></li>
<li><a href="database_migration.html">database migration</a></li>
<li><a href="client_api.html">client api (under construction)</a></li>
<li><a href="client_api.html">client api</a></li>
<li><a href="ipfs.html">ipfs</a></li>
<li><a href="local_booru.html">the local booru</a></li>
<li><a href="server.html">setting up your own server</a></li>
Expand Down
15 changes: 13 additions & 2 deletions include/ClientAPI.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -231,14 +231,25 @@ def CheckCanSearchTags( self, tags ):

if len( filtered_tags ) > 0:

return len( filtered_tags ) == len( tags )
return



raise HydrusExceptions.InsufficientCredentialsException( 'You do not have permission to do this search. Your tag search permissions are: {}'.format( self._search_tag_filter.ToPermittedString() ) )



def CheckCanSeeAllFiles( self ):

with self._lock:

if not ( self._HasPermission( CLIENT_API_PERMISSION_SEARCH_FILES ) and self._search_tag_filter.AllowsEverything() ):

raise HydrusExceptions.InsufficientCredentialsException( 'You do not have permission to see all files, so you cannot do this.' )




def CheckPermission( self, permission ):

if not self.HasPermission( permission ):
Expand Down Expand Up @@ -270,7 +281,7 @@ def CheckPermissionToSeeFiles( self, hash_ids ):

error_text = error_text.format( HydrusData.ToHumanInt( num_files_asked_for ), HydrusData.ToHumanInt( num_files_allowed_to_see ) )

raise HydrusExceptions.BadRequestException( error_text )
raise HydrusExceptions.InsufficientCredentialsException( error_text )


self._search_results_timeout = HydrusData.GetNow() + SEARCH_RESULTS_CACHE_TIMEOUT
Expand Down
24 changes: 23 additions & 1 deletion include/ClientDB.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5680,6 +5680,22 @@ def _GetHashIdsThatHaveTagAsNum( self, file_service_key, tag_service_key, namesp



def _GetHashIdsToHashes( self, hash_ids = None, hashes = None ):

if hash_ids is not None:

self._PopulateHashIdsToHashesCache( hash_ids, exception_on_error = True )

hash_ids_to_hashes = { hash_id : self._hash_ids_to_hashes_cache[ hash_id ] for hash_id in hash_ids }

elif hashes is not None:

hash_ids_to_hashes = { self._GetHashId( hash ) : hash for hash in hashes }


return hash_ids_to_hashes


def _GetHashIdStatus( self, hash_id, prefix = '' ):

hash = self._GetHash( hash_id )
Expand Down Expand Up @@ -7796,7 +7812,7 @@ def _OverwriteJSONDumps( self, dump_types, objs ):



def _PopulateHashIdsToHashesCache( self, hash_ids ):
def _PopulateHashIdsToHashesCache( self, hash_ids, exception_on_error = False ):

if len( self._hash_ids_to_hashes_cache ) > 25000:

Expand All @@ -7819,6 +7835,11 @@ def _PopulateHashIdsToHashesCache( self, hash_ids ):

if hash_id not in uncached_hash_ids_to_hashes:

if exception_on_error:

raise HydrusExceptions.DataMissing( 'Did not find all entries for those hash ids!' )


HydrusData.DebugPrint( 'Database hash error: hash_id ' + str( hash_id ) + ' was missing!' )

if not pubbed_error:
Expand Down Expand Up @@ -9223,6 +9244,7 @@ def _Read( self, action, *args, **kwargs ):
elif action == 'filter_existing_tags': result = self._FilterExistingTags( *args, **kwargs )
elif action == 'filter_hashes': result = self._FilterHashes( *args, **kwargs )
elif action == 'force_refresh_tags_managers': result = self._GetForceRefreshTagsManagers( *args, **kwargs )
elif action == 'hash_ids_to_hashes': result = self._GetHashIdsToHashes( *args, **kwargs )
elif action == 'hash_status': result = self._GetHashStatus( *args, **kwargs )
elif action == 'imageboards': result = self._GetYAMLDump( YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs )
elif action == 'in_inbox': result = self._InInbox( *args, **kwargs )
Expand Down
Loading

0 comments on commit 67e39d8

Please sign in to comment.