README-stats.otl

---------------------------------------------------------------------------------
                                    STATS

---------------------------------------------------------------------------------
The following are the list of problems associated with the
existing stat system:
1) atomicity issues
2) consistency and coupling issues
3) clearing
4) persistence
5) aggregation and derivation
6) relationship with manager.

Specifically, some of the stats reported seemed incorrect.
In the cases where individual stats were incorrect (e.g. 
number of bytes transferred), it is suspected that atomicity
issues played a role. For example, if the stat variable was
64 bits long, were all 64 bits being read and written atomically?
Is there the possibility that while the lower/higher 32 bits
were being accessed, the other 32 bits were being changed?

In stats which were computed based on other stats (e.g. hit
rate is ratio of number of cache hits over total number of
cache lookups), it is suspected that there were coupling problems.
For example, if the number of cache hits is read and then
before the number of cache lookups can be read, it is updated,
the hit rate may appear incorrect.

Some stats are interrelated with other stats (e.g. number of
client requests and number of cache lookups). Inconsistencies
between such stats may show up if for example one of the 
values is accessed and before the other value is accessed, it
is allowed to change. A single client request may then, for example,
show two cache lookups.

These issues can be aggravated by allowing administrators to
clear stat values. This lead us to introduce persistence: 
disallow clearing of stats. While this does deal with the problems
associated with clearing, it does not address the coupling,
inconsistency or atomicity problems. Those remain.

The remaining issues, those of aggregation and interaction with
the manager did not suffer from any fundamental design problems.
They were more implementation related issues.

A new stat system which deals correctly with the issues of
atomicity, consistency and coupling will automatically address
the clearing and persistence problem. Atomicity can be handled
very simply by either of two techniques:
1) make ALL accesses to stat variable atomic operations or
2) protect ALL accesses to stat variables by placing them
   within critical regions.

The inconsistency and coupling problem is two-fold in nature.
There is the problem of grouping - in that related stats (either
indirectly, as illustrated by the inconsistency example, or
directly as illustrated by the coupling example), may be grouped
together. Access to either one of them, then, if controlled, will
guarantee that the other is not concurrently modified. So, this
will prevent values from changing while related values are being
accessed. It will not, however, solve the problem of maintaining
"transactional" consistency. Look at the number of request/
number of cache lookup example again. If the stat for the number
of requests is updated. Then the stat for the number of cache
lookups is updated, with both updates being within critical regions,
if the values of both stats are accessed in the time between when
one is updated and when the other is updated, the inconsistency
problem remains. So, even though the stats are grouped together in
that they are accessed through the same mutex, for example, the
inconsistency problem remains. It can be solved only if we introduce
the notion of a transaction. All stats which are for the same
transaction must be updated together. The values of these stats
at any time, then, will be consistent. This is a more difficult
problem, because not only does it introduce grouping in space,
it introduces grouping in time as well. Unfortunately, an implementation
requires the notion of "local" and "global" stats.

Now, stats fall into three categories:
1) transactional (dyanmic) stats: e.g. number of current origin server
   connections,
2) incremental stats: number of bytes from origin server,
3) transactional stats: number of requests served.

Stats of type 1) need to just be accessed atomically. They do not
need to be cleared. Stats of type 2) need to also be accessed atomically,
and may need to be cleared. They are not related to other stats, so they
do not need to be grouped. Stats of type 3) need to be accessed atomically,
but they are grouped and so must be accessed simultaneously in space-time.

The three types of stats can be implemented with three different mechanisms,
or a single mechanism can be defined which handles all three types. The latter
approach is more elegant, but it raises questions about performance.

Since transactional stats have to be accessed together, it is logical to
control access to them through a single lock. A stat structure, then, can
be defined to have a lock which controls access. For individual stats which
require just atomic access, however, it may be better to allow direct
access, without getting a lock. This may reduce lock contention.

We ran two experiments to investigate the performance issue.
The existing stat system allows for individual atomic access to all stats. 
We compared the performance of this system with a system where instead of
atomic access, all access requires getting a single lock. The results from
inkbench are as follows.

[keep_alive =4, 100% Hit rate, 4 client machines, 40 simultaneous users ]
atomic: 526 ops/sec
mutex: 499 ops/sec

Changing the setup to proxy-only, which reduces the number of
simultaneous accesses, gave:

[keep_alive =4, proxy-only, 4 client machines, 40 simultaneous users ]
atomic: 284 ops/sec
mutex: 282 ops/sec

We also ran the test with stats turned off, which gave unexpected results.
It is unclear, however, how much we really turn the stats off (there were
even compile-time problems and many of the stat reading/writing routines
were still being called, etc.).

In addition to the tests mentioned above, we spoke to Prof. Dirk
Grunwald.  He said that the most expensive operations are the memory
barriers. On the Alphas, atomic operations are implemented with
load-locked, store- conditional and memory-barrier instructions. I
think each operation is implemented with 2 mbs. Locks and Releases
have one mb each. So, the number of mbs is much higher with atomic
updates. Both atomic updates and mutex-based updates will have to pay
similar cache loading costs (if we assume that the loads for the
test-and-set and the loads for reads will fail, the stores will
obviously hit). The difference, if any, will come from the contention
for locks while all of the group elements are being updated and that
has to be compared with the cost of as many mbs as the stats are being
atomically updated. Context-switching while holding the lock may also
result in a performance hit, but it may be possible to address this by
doing "lazy" updates of global structures.

The consensus, then is to have a single mechanism for access to stats:
mutexes. This combined with a grouping scheme solves the atomicity,
consistency, coupling, clearing and persistence problems.


---------------------------------------------------------------------------------
Brian's Rant:
-------------
In any case, I want to change the stats system once to address all the issues
of clearing, persistence, monotonicity, coupled computations, cluster
aggregation, etc.  Let's be sure we all understand the design issues and do
the right thing one time.  We've gone through lots of band-aids in the past
which got us nowhere, so let's get the design right.
---------------------------------------------------------------------------------
Issues:
-------
1) Clearing
2) Persistence
3) Monotonicity
4) Coupled computations
5) Cluster aggregation
6) Inconsistencies between related stats

- Want to be able to clear stats
- Do not want to see inconsistencies between stats

stats which increase and decrease (e.g. current open connections should
not be clearable)

persistent stats should not be clearable

related stats should be updated together (updated/cleared)

divide stats into 2 groups:
1) dynamic stats - not clearable, read,written singly, atomically
2) grouped stats - must be updated together

---------------------------------------------------------------------------------
Types Of Stats:
---------------
* event system stats
	** average idle time per thread 
	** variance of idle time per thread 
	** average event exec rate per thread 
	** variance of event exec rate per thread 
	** average execution time per event 
	** variance of execution time per event 
	** average failed mutexes per event 
	** variance of failed mutexes per event 
	** average time lag per event
	** variance of time lag per event
	
* socks processor stats
	** connections unsuccessful
	** connections successful
	** connections currently open

---------------------------------------------------------------------------------
* cache stats
	*** total size in bytes
	*** # cache operations in progress
	*** Read bandwidth from the disk
	*** Write bandwidth to the disk
	*** RAM cache hit-rate.
	*** Read operations per second.
	*** Write operations per second.
	*** Update operations per second.
	*** Delete operations per second.
	*** Operations per second.

---------------------------------------------------------------------------------
* ICP Stats
proxy.process.icp.config_mgmt_callouts
  - Manager callouts for ICP config changes

proxy.process.icp.reconfig_polls
  - ICP config change periodic polls

proxy.process.icp.reconfig_events
  - ICP config change polls which detected config changes

proxy.process.icp.invalid_poll_data
  - Negative event callouts with invalid poll data

proxy.process.icp.no_data_read
  - Read on ICP socket returned <= 0 bytes

proxy.process.icp.short_read
  - ICP msg len != bytes read

proxy.process.icp.invalid_sender
  - ICP msg sender unknown

proxy.process.icp.read_not_v2_icp
  - Sent ICP msg not version 2

(UI) proxy.process.icp.icp_remote_query_requests
  - Remote ICP query requests received

    proxy.process.icp.icp_remote_responses
  - ICP replies received for locally generated queries

(UI) proxy.process.icp.cache_lookup_success
  - ICP queries resulting in Cache lookup success 

(UI) proxy.process.icp.cache_lookup_fail
  - ICP queries resulting in Cache lookup failure 

(UI) proxy.process.icp.query_response_write
  - ICP query responses sent

proxy.process.icp.query_response_partial_write
  - ICP query responses which were short writes

proxy.process.icp.no_icp_request_for_response
  - Received ICP query responses associated with no ICP request

proxy.process.icp.icp_response_request_nolock
  - ICP request lock acquire miss

proxy.process.icp.icp_start_icpoff
  - Local ICP queries aborted due to ICP off

proxy.process.icp.send_query_partial_write
  - ICP query sends which were short writes

proxy.process.icp.icp_queries_no_expected_replies
  - Local ICP queries sent where no response was expected

(UI) proxy.process.icp.icp_query_hits
  - ICP hits received for locally generated queries

(UI) proxy.process.icp.icp_query_misses
  - ICP misses received for locally generated queries

proxy.process.icp.invalid_icp_query_response
  - ICP responses with invalid opcodes

(UI) proxy.process.icp.icp_query_requests
  - Locally generated ICP requests

(UI) proxy.process.icp.total_icp_response_time
  - Avg response time of each locally generated ICP query

(UI) proxy.process.icp.total_udp_send_queries
  - Total ICP query sends

(UI) proxy.process.icp.total_icp_request_time
  - Overall avg response time of locally generated ICP request
---------------------------------------------------------------------------------
* hostdb/dns stats
	** hostdb
		*** total entries
		*** total number of lookups
		*** total number of hits
		*** total number of misses
		*** hit rate??? (or do we just compute this)
	// to help tuning
		*** average TTL remaining for hosts looked up
		*** total number re-DNS because of re-DNS on reload
		*** total number re-DNS because TTL expired
	** dns
		*** total number of lookups
		*** total number of hits
		*** total number of misses
		*** average service time
	// to help tuning
		*** total number of retries
		*** total number which failed with too many retries
---------------------------------------------------------------------------------
* network stats
>	** incoming connections stats
  shouldn't these stats be by protocol???
>		*** average b/w for each connection
  at the net level, I don't always know when I should be expecting bytes.
  for example, a keepalive connection may expect more bytes and then not
  get them.  When do I start and stop the timer???  This needs some thought
>		*** average latency for each connection
  same as above.  perhaps
		*** latency to first byte on an accept
  however for "latency to first by of a response" I would need to know when
  the request was done, but that is a protocol level issue.
>		*** number of documents/connection (keep-alive)
  this is a protocol issue (http)
>		*** number aborted
  I could record how many received a do_io(VIO::ABORT) for what that
  is worth... maybe this is also a protocol issue
		*** high/low watermark number of simultaneous connections
		*** high/low watermark number of connection time
  is this last one useful??
>	** outgoing connections stats
>		*** average b/w for each connection
>		*** average latency for each connection
>		*** number of documents/connection (keep-alive)
>		*** number aborted
  see above
		*** high/low watermark number of simultaneous connections
		*** high/low watermark number of connection time
>	** incoming connections stats
  what does this mean??
>		*** average b/w for each connection
>		*** average latency for each connection
>		*** number aborted
	see above
>		*** high/low watermark number of simultaneous connections
>		*** high/low watermark number of connection time

---------------------------------------------------------------------------------
* Http Stats
 	** protocol stats
 		*** number of HTTP requests (need for ua->ts, ts->os)
 			**** GETs
 			**** HEADs
 			**** TRACEs
 			**** OPTIONs
 			**** POSTs
 			**** DELETEs
 			**** CONNECTs
 		*** number of invalid requests 
 		*** number of broken client connections
 		*** number of proxied requests
 			**** user-specified
 			**** method-specified
 			**** config specified (config file)
 		*** number of requests with Cookies
 		*** number of cache lookups
 			**** number of alternates
 		*** number of cache hits
                        cache hits, misses, etc. stats should be catagorized
                        the same way they are categorized in WUTS (squid logs).
 			**** fresh hits
 			**** stale hits
 				***** heuristically stale
 				***** expired/max-aged
		*** number of cache writes
		*** number of cache updates
		*** number of cache deletes
 		*** number of valid responses
 		*** number of invalid responses
 		*** number of retried requests
 		*** number of broken server connections

	        *** For each response code, count the number of responses (need for ua->ts, ts->os)
		*** Histogram of http version (need for ua->ts, ts->os)
 		*** number of responses with expires
 		*** number of responses with last-modified
 		*** number of responses with age
 		*** number of responses indicating server clock skew
 		*** number of responses preventing cache store
 			**** Set-Cookie
 			**** Cache-Control: no-store
 		*** number of responses proxied directly
	** Timeouts and connection errors:
		*** accept timeout (ua -> proxy only)
		*** background fill timeout (os -> proxy only)
		*** normal inactive timeout (both ua -> proxy and proxy -> os)
		*** normal activity timeout (both ua -> proxy and proxy -> os)
		*** keep-alive idle timeout (both ua -> proxy and proxy -> os)
	** Connections and transactions time:
		*** average requests count per connection (both ua and os).
		*** average connection lifetime (both ua and os).
		*** average connection utilization (connection time / transactions count).
		*** average transaction time (from transaction start until transaction end).
		*** average transaction time histogram per document sizes.
		  sizes are:
		    <= 100 bytes, <= 1K, <= 3K, <= 5K, <= 10L, <= 1M, <= infinity
		*** average transaction processing time (think time).
		*** average transaction processing time (think time) histogram per document sizes.
	** Transfer rate:
		*** bytes per second ua connection and os connection.
		*** bytes per second ua connection and os connection histogram per document size.
	** Cache Stats
		*** cache lookup time hit
		*** cache lookup time miss
		*** cache open read time
		*** cache open write time
---------------------------------------------------------------------------------
* loggin stats
// bytes moved
STAT(Sum,   log2_stat_bytes_buffered),
STAT(Sum,   log2_stat_bytes_written_to_disk),
STAT(Sum,   log2_stat_bytes_sent_to_network),
STAT(Sum,   log2_stat_bytes_received_from_network),
// I/O
STAT(Count, log2_stat_log_files_open),			UI
STAT(Count, log2_stat_log_files_space_used),		UI
// events
STAT(Count, log2_stat_event_log_error),			UI
STAT(Count, log2_stat_event_log_access),		UI
STAT(Count, log2_stat_event_log_access_fail),		
STAT(Count, log2_stat_event_log_access_skip),		UI

---------------------------------------------------------------------------------
------------
Here's my version of the 20-stats list for the UI. I gathered all of the
stats proposed from our meeting last Thursday, added requests from Tait
Kirkham and Brian Totty, then added a few based on pilot site questions and
customer's monitoring scripts. Not all of these additional requests are
necessarily possible to measure. Sorted the proposed statistics into groups
based on the kind of information they provide the proxy administrator. Then I
picked the ones that seemed most important to me for the top 20 list. From
the top-level statistics page there should be links to pages with more detail
in each operational area.

A few notes:
"Average" statistics should be calculated over an interval configurable as 1
minute, 5 minutes or 15 minutes. This includes the cache hit rate. Probably
the cache detail info page should have a value for the overall average.
Numbers and measurements in the user interface are presented for the
convenience of the customer, and should be organized to suit customer needs
and interests. Statistics from the operating system, the HTTP state machine,
and anywhere else may be combined to create info presented in the UI.
-----------------------------------------------------------

  Statistics in the Traffic Manager UI
  Top "20" Recommended List

Client Responsiveness
avg/max total transaction time -- request received to last byte sent.
number of client aborts
network bytes/sec/client

Proxy Activity
total/max active client connections
total/max active server connections
total/max active cache connections
total/max active parent proxy connections
average transaction rate
total transaction count per protocol {dns, http, rtsp}
total bytes served to clients
total bytes from origin servers
total bytes from parent proxies
bandwidth savings: %  -- bandwidth per interface, per route options?

Cache
cache utilization %
avg cache hit rate
cache revalidates

-----------------------------------------------------------------------------------------------------------------

  Statistics in the Traffic Manager
  The Whole List

Client Responsiveness
avg/max time to first byte  -- request received to first byte sent.
avg/max total transaction time -- request received to last byte sent.
avg transaction time by type of transaction:
     cache write
     non-cached
     client revalidate
     proxy revalidate
     proxy refresh
number of client aborts
network bytes/sec/client

Proxy Activity
total/max active client connections
total/max active server connections
total/max active cache connections
number of active connections by connection status: ESTABLISHED, CLOSE_WAIT,
TIME_WAIT, ...
total transaction count per protocol {dns, http, rtsp}
total/max active parent proxy connections
total bytes served to clients
total bytes from origin servers
total bytes from parent proxies
bandwidth savings: bytes / %
number of TCP retransmits per client
ps-like status report of Traffic Server operations

System Utilization
proxy network utilization %
cpu utilization %
disk utilization %
memory (RAM) utilization %

Cache
total cache free space
cache utilization %
avg cache hit rate
cache revalidates:
     client-specified
     expired (expire/max-age)
     expired (heuristic)
cache miss time
cache hit time

Logging
total  logging space used

Cluster
cluster network utilization %

ICP
total icp responses
icp hit rate
total icp requests
---------------------------------------------------------------------------------
Subject: Re: Simple UI for TS100
Date: Thu, 26 Mar 1998 11:48:15 -0800

>From a Support standpoint, I do have some reservations about hacking
information and function out of the Traffic Manager for TS Lite. Less
information in the Monitor makes it more difficult to observe Traffic Server
operation. Less functionality in Configure means more manual editting of
config files and greater chance of admin errors. These factors could greatly
increase our support problems and cost for "Lite" customers.

Perhaps the TS Lite UI should just remove all cluster-related information
and controls, and material related to any other features that simply will
not be offered on TS Lite, like the maximum network connection limit. Other
counters and controls that relate to functions and features that are
operational in TS Lite could be left alone. There're already present in TS
Classic, so it costs us nothing extra.

-----------------------------------

Within the framework of Adam's proposal, the "Monitor" manager reduces to a
single screen. Great! With only one screen, gotta be sure that the key
top-level statistics are visible. I suggest that cache free is not really a
very interesting value for most sites. Bandwidth savings is the value we
really want to emphasize, so it must not disappear. So the Dashboard should
have its current "More Info" stats plus
    cache size
    DNS Hit rate
    Bandwidth savings
    tps count (the tps meter is nice, but can't be read very precisely)
The More/Less detail option itself can disappear. The word "cluster" should
not occur anywhere in the TS Lite UI.

In Configure: Server the node on/off switch and the cluster restart button
are redundant in TS Lite. Probably ought to lose the cluster button. A
configurable Traffic Server name also seems useless when you can't have a
cluster.

For Configure: Logging, perhaps custom log formatting should not be offered
for TS Lite?

With reduced UI configuration capability, it may be appropriate to remove
the Configure: Snapshots screen.

Adam Beguelin wrote:

> For TS100 we need a simplified UI.  Here's a first cut on how we should
> simplify the UI.
>
> For most of the removed pages we will simply use the installed defaults.
>
> Comments?
>
>         Adam
>
> On the dashboard add these stats:
>  o Cahce Free Space
>  o Cache Size
>  o DNS Hit rate
>
> Remove these monitor pages:
>  o node
>  o Graphs
>  o Protocols
>  o Cache
>  o Other
>
> Remove these sections under configure:
>  o Server/Throttling of Network Connections
>  o Protocols/HTTP timeouts (Leave anon and IP)
>  o Protocols/SSL
>  o Cache/Storage
>  o Cache/Garbage Collection
>  o Cache/Freshness
>  o Cache/Variable Content
>  o HostDB page
>  o Logging/Log Collation
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------