Skip to content

Retrieve additional IC/TS statistics with getattribute & get_image_info() #1438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lgritz merged 1 commit into from
Jun 27, 2016
Merged

Retrieve additional IC/TS statistics with getattribute & get_image_info() #1438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 126 additions & 1 deletion src/doc/imagecache.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,83 @@ \subsubsection*{Image cache attributes}
\end{code}
\apiend

\apiitem{int max_open_files}
The maximum number of file handles that the image cache will
hold open simultaneously. (Default = 100)
\apiend

\apiitem{int total_files {\rm ~(read only)}}
The total number of unique file names referenced by calls to the \ImageCache.
\apiend

\apiitem{string[] all_filenames {\rm ~(read only)}}
An array that will be filled with the list of the names of all files
referenced by calls to the \ImageCache. (The array is of {\cf ustring}s or
{\cf char*}'s.)
\apiend

\apiitem{int64 stat:cache_memory_used {\rm ~(read only)}}
Total bytes used by tile cache.
\apiend

\apiitem{int stat:tiles_created {\rm ~(read only)} \\
int stat:tiles_current {\rm ~(read only)} \\
int stat:tiles_peak {\rm ~(read only)}}
Total times created, still allocated (at the time of the query), and the
peak number of tiles in memory at any time.
\apiend

\apiitem{int stat:open_files_created {\rm ~(read only)} \\
int stat:open_files_current {\rm ~(read only)} \\
int stat:open_files_peak {\rm ~(read only)}}
Total number of times a file was opened, number still opened (at the time of
the query), and the peak number of files opened at any time.
\apiend

\apiitem{int stat:find_tile_calls {\rm ~(read only)}}
Number of times a filename was looked up in the file cache.
\apiend

\apiitem{int64 stat:files_totalsize {\rm ~(read only)}}
Total size (uncompressed bytes of pixel data) of all files referenced.
\apiend

\apiitem{int64 stat:bytes_read {\rm ~(read only)}}
Total size (uncompressed bytes of pixel data) read.
\apiend

\apiitem{int stat:unique_files {\rm ~(read only)}}
Number of unique files opened.
\apiend

\apiitem{float stat:fileio_time {\rm ~(read only)}}
Total I/O-related time (seconds).
\apiend

\apiitem{float stat:fileopen_time {\rm ~(read only)}}
I/O time related to opening and reading headers (but not pixel I/O).
\apiend

\apiitem{float stat:file_locking_time {\rm ~(read only)}}
Total time (across all threads) that threads blocked waiting for access to
the file data structures.
\apiend

\apiitem{float stat:tile_locking_time {\rm ~(read only)}}
Total time (across all threads) that threads blocked waiting for access to
the tile cache data structures.
\apiend

\apiitem{float stat:find_file_time {\rm ~(read only)}}
Total time (across all threads) that threads spent looking up files by name.
\apiend

\apiitem{float stat:find_tile_time {\rm ~(read only)}}
Total time (across all threads) that threads spent looking up individual tiles.
\apiend



\bigskip

\subsection{Opaque data for performance lookups}
Expand Down Expand Up @@ -479,12 +556,16 @@ \subsection{Getting information about images}
\item[\spc] \spc
\vspace{-12pt}

\item[\rm \kw{exists}] Stores the value 1 (as an {\cf int} if the file
\item[\rm \kw{exists}] Stores the value 1 (as an {\cf int}) if the file
exists and is an image format that \product can read, or 0 if the file
does not exist, or could not be properly read as an image. Note that
unlike all other queries, this query will ``succeed'' (return {\cf true})
even if the file does not exist.

\item[\rm \kw{udim}] Stores the value 1 (as an {\cf int}) if the file
is a ``virtual UDIM'' or texture atlas file (as described in
Section~\ref{sec:texturesys:udim}) or 0 otherwise.

\item[\rm \kw{subimages}] The number of subimages in the file, as an integer.

\item[\rm \kw{resolution}] The resolution of the image file, which
Expand Down Expand Up @@ -544,6 +625,50 @@ \subsection{Getting information about images}
coordinate system where $x$ and $y$ range from $-1$ to $+1$. Generally,
only rendered images will have this.

\item[\rm \kw{averagecolor}] If available in the metadata (generally only
for files that have been processed by {\cf maketx}), this will return the
average color of the texture (into an array of floats).

\item[\rm \kw{averagealpha}] If available in the metadata (generally only
for files that have been processed by {\cf maketx}), this will return the
average alpha value of the texture (into a float).

\item[\rm \kw{constantcolor}] If the metadata (generally only for files that
have been processed by {\cf maketx}) indicates that the texture has the same
values for all pixels in the texture, this will retrieve the constant color
of the texture (into an array of floats). A non-constant image (or one that
does not have the special metadata tag identifying it as a constant texture)
will fail this query (return false).

\item[\rm \kw{constantalpha}] If the metadata indicates that the texture has
the same values for all pixels in the texture, this will retrieve the
constant alpha value of the texture (into a float). A non-constant image (or
one that does not have the special metadata tag identifying it as a constant
texture) will fail this query (return false).

\item[\rm \kw{stat:tilesread}] Number of tiles read from this file ({\cf int64}).

\item[\rm \kw{stat:bytesread}] Number of bytes of uncompressed pixel data read
from this file ({cf int64}).

\item[\rm \kw{stat:redundant_tiles}] Number of times a tile was read, where
the same tile had been rad before. ({\cf int64}).

\item[\rm \kw{stat:redundant_bytesread}] Number of bytes (of uncompressed pixel
data) in tiles that were read redundantly. ({\cf int64}).

\item[\rm \kw{stat:redundant_bytesread}] Number of tiles read from this file ({\cf int}).

\item[\rm \kw{stat:timesopened}] Number of times this file was opened ({\cf int}).

\item[\rm \kw{stat:iotime}] Time (in seconds) spent on all I/O for this file ({\cf float}).

\item[\rm \kw{stat:mipsused}] Stores 1 if any MIP levels beyond the highest
resolution were accesed, otherwise 0. ({\cf int})

\item[\rm \kw{stat:is_duplicate}] Stores 1 if this file was a duplicate of
another image, otherwise 0. ({\cf int})

\item[Anything else] -- For all other data names, the
the metadata of the image file will be searched for an item that
matches both the name and data type.
Expand Down
35 changes: 34 additions & 1 deletion src/doc/texturesys.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1180,7 +1180,7 @@ \subsection{Texture Metadata and Raw Texels}
unlike all other queries, this query will ``succeed'' (return {\cf true})
even if the file does not exist.

\item[\rm \kw{udim}] Stores the value 1 (as an {\cf int} if the file
\item[\rm \kw{udim}] Stores the value 1 (as an {\cf int}) if the file
is a ``virtual UDIM'' or texture atlas file (as described in
Section~\ref{sec:texturesys:udim}) or 0 otherwise.

Expand Down Expand Up @@ -1214,6 +1214,16 @@ \subsection{Texture Metadata and Raw Texels}
\item[\rm \kw{channels}] The number of color channels in the file
(an integer).

\item[\rm \kw{format}] The native data format of the pixels in the
file (an integer, giving the {\cf TypeDesc::BASETYPE} of the data).
Note that this is not necessarily the same as the data format stored
in the image cache.

\item[\rm \kw{cachedformat}] The native data format of the pixels as
stored in the image cache (an integer, giving the {\cf
TypeDesc::BASETYPE} of the data). Note that this is not necessarily
the same as the native data format of the file.

\item[\rm \kw{datawindow}]
Returns the pixel data window of the image, which is either an array of 4
integers (returning xmin, ymin, xmax, ymax) or an array of 6 integers
Expand Down Expand Up @@ -1258,6 +1268,29 @@ \subsection{Texture Metadata and Raw Texels}
one that does not have the special metadata tag identifying it as a constant
texture) will fail this query (return false).

\item[\rm \kw{stat:tilesread}] Number of tiles read from this file ({\cf int64}).

\item[\rm \kw{stat:bytesread}] Number of bytes of uncompressed pixel data read
from this file ({cf int64}).

\item[\rm \kw{stat:redundant_tiles}] Number of times a tile was read, where
the same tile had been rad before. ({\cf int64}).

\item[\rm \kw{stat:redundant_bytesread}] Number of bytes (of uncompressed pixel
data) in tiles that were read redundantly. ({\cf int64}).

\item[\rm \kw{stat:redundant_bytesread}] Number of tiles read from this file ({\cf int}).

\item[\rm \kw{stat:timesopened}] Number of times this file was opened ({\cf int}).

\item[\rm \kw{stat:iotime}] Time (in seconds) spent on all I/O for this file ({\cf float}).

\item[\rm \kw{stat:mipsused}] Stores 1 if any MIP levels beyond the highest
resolution were accesed, otherwise 0. ({\cf int})

\item[\rm \kw{stat:is_duplicate}] Stores 1 if this file was a duplicate of
another image, otherwise 0. ({\cf int})

\item[Anything else] -- For all other data names, the
the metadata of the image file will be searched for an item that
matches both the name and data type.
Expand Down
14 changes: 13 additions & 1 deletion src/include/OpenImageIO/imagecache.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,19 @@ class OIIO_API ImageCache {
virtual bool attribute (string_view name, double val) = 0;
virtual bool attribute (string_view name, string_view val) = 0;

/// Get the named attribute, store it in value.
/// Get the named attribute, store it in *val. All of the attributes
/// that may be set with the attribute() call may also be queried with
/// getattribute().
///
/// Additionally, there are some read-only attributes that can be
/// queried with getattribute():
/// int total_files : the total number of unique files referenced by
/// calls to the ImageCache.
/// string[] all_filenames : an array that will be filled with the
/// list of the names of all files referenced by calls to
/// the ImageCache. (The array is of ustrings or char*'s.)
/// stat:* : a variety of statistics (see full docs for details).
///
virtual bool getattribute (string_view name, TypeDesc type,
void *val) const = 0;
// Shortcuts for common types
Expand Down
3 changes: 3 additions & 0 deletions src/include/OpenImageIO/unordered_map_concurrent.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,9 @@ class unordered_map_concurrent {
/// Return true if the entire map is empty.
bool empty() { return m_size == 0; }

/// Return the total number of entries in the map.
size_t size () { return size_t(m_size); }

/// Expliticly lock the bin that will contain the key (regardless of
/// whether there is such an entry in the map), and return its bin
/// number.
Expand Down
54 changes: 42 additions & 12 deletions src/libtexture/imagecache.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@
#include "imagecache_pvt.h"

#include <boost/foreach.hpp>
#include <boost/algorithm/string.hpp>
#include <boost/scoped_array.hpp>


Expand All @@ -81,7 +80,7 @@ static ustring s_resolution ("resolution"), s_texturetype ("texturetype");
static ustring s_textureformat ("textureformat"), s_fileformat ("fileformat");
static ustring s_format ("format"), s_cachedformat ("cachedformat");
static ustring s_channels ("channels"), s_cachedpixeltype ("cachedpixeltype");
static ustring s_exists ("exists");
static ustring s_exists ("exists"), s_broken ("broken");
static ustring s_UDIM ("UDIM");
static ustring s_subimages ("subimages"), s_miplevels ("miplevels");
static ustring s_datawindow ("datawindow"), s_displaywindow ("displaywindow");
Expand Down Expand Up @@ -2106,6 +2105,7 @@ ImageCacheImpl::getattribute (string_view name, TypeDesc type,
ATTR_DECODE ("deduplicate", int, m_deduplicate);
ATTR_DECODE ("unassociatedalpha", int, m_unassociatedalpha);
ATTR_DECODE ("failure_retries", int, m_failure_retries);
ATTR_DECODE ("total_files", int, m_files.size());

// The cases that don't fit in the simple ATTR_DECODE scheme
if (name == "searchpath" && type == TypeDesc::STRING) {
Expand Down Expand Up @@ -2134,17 +2134,26 @@ ImageCacheImpl::getattribute (string_view name, TypeDesc type,
*(const char **)val = m_substitute_image.c_str();
return true;
}
if (name == "all_filenames" && type.basetype == TypeDesc::STRING &&
type.is_sized_array()) {
ustring *names = (ustring *) val;
int n = type.arraylen;
for (FilenameMap::iterator f = m_files.begin(); f != m_files.end() && n-- > 0; ++f) {
*names++ = f->second->filename();
}
return true;
}

// Stats we can just grab
ATTR_DECODE ("stat:cache_memory_used", long long, m_mem_used);
ATTR_DECODE ("stat:tiles_created", int, m_stat_tiles_created);
ATTR_DECODE ("stat:tiles_current", int, m_stat_tiles_current);
ATTR_DECODE ("stat:tiles_peak", int, m_stat_tiles_peak);
ATTR_DECODE ("stat:open_files_created", int, m_stat_open_files_created);
ATTR_DECODE ("stat:open_files_current", int, m_stat_open_files_current);
ATTR_DECODE ("stat:open_files_peak", int, m_stat_open_files_peak);
if (Strutil::starts_with(name, "stat:")) {
// Stats we can just grab
ATTR_DECODE ("stat:cache_memory_used", long long, m_mem_used);
ATTR_DECODE ("stat:tiles_created", int, m_stat_tiles_created);
ATTR_DECODE ("stat:tiles_current", int, m_stat_tiles_current);
ATTR_DECODE ("stat:tiles_peak", int, m_stat_tiles_peak);
ATTR_DECODE ("stat:open_files_created", int, m_stat_open_files_created);
ATTR_DECODE ("stat:open_files_current", int, m_stat_open_files_current);
ATTR_DECODE ("stat:open_files_peak", int, m_stat_open_files_peak);

if (boost::algorithm::starts_with(name, "stat:")) {
// All the other stats are those that need to be summed from all
// the threads.
ImageCacheStatistics stats;
Expand Down Expand Up @@ -2411,6 +2420,12 @@ ImageCacheImpl::get_image_info (ImageCacheFile *file,
ustring dataname,
TypeDesc datatype, void *data)
{
#define ATTR_DECODE(_name,_ctype,_src) \
if (dataname == _name && datatype == BaseTypeFromC<_ctype>::value) { \
*(_ctype *)(data) = (_ctype)(_src); \
return true; \
}

file = verify_file (file, thread_info, true);
if (dataname == s_exists && datatype == TypeDesc::TypeInt) {
// Just check for existence. Need to do this before the invalid
Expand All @@ -2424,11 +2439,25 @@ ImageCacheImpl::get_image_info (ImageCacheFile *file,
error ("Invalid image file");
return false;
}
ATTR_DECODE (s_broken, int, file->broken());
if (Strutil::starts_with (dataname, "stat:")) {
ATTR_DECODE ("stat:tilesread", long long, file->m_tilesread);
ATTR_DECODE ("stat:bytesread", long long, file->m_bytesread);
ATTR_DECODE ("stat:redundant_tiles", long long, file->m_redundant_tiles);
ATTR_DECODE ("stat:redundant_bytesread", long long, file->m_redundant_bytesread);
ATTR_DECODE ("stat:timesopened", int, file->m_timesopened);
ATTR_DECODE ("stat:iotime", float, file->m_iotime);
ATTR_DECODE ("stat:mipused", int, file->m_mipused);
ATTR_DECODE ("stat:is_duplicate", int, bool(file->duplicate()));
}

if (file->broken()) {
if (file->errors_should_issue())
error ("Invalid image file \"%s\"", file->filename());
return false;
}
// No other queries below are expected to work with broken

if (dataname == s_UDIM && datatype == TypeDesc::TypeInt) {
// Just check for existence. Need to do this before the invalid
// file error below, since in this one case, it's not an error
Expand Down Expand Up @@ -2554,7 +2583,6 @@ ImageCacheImpl::get_image_info (ImageCacheFile *file,
return false; // Fail if it's not a constant image
}


// general case -- handle anything else that's able to be found by
// spec.find_attribute().
const ImageIOParameter *p = spec.find_attribute (dataname.string());
Expand All @@ -2574,6 +2602,8 @@ ImageCacheImpl::get_image_info (ImageCacheFile *file,
}

return false;

#undef ATTR_DECODE
}


Expand Down
Loading