stats.org

EP Stats

1 Getting Started

For introductory information on stats within membase, start with the membase wiki stats page.

2 Stats Definitions

2.1 Toplevel Stats

Stat	Description
ep_version	Version number of ep_engine.
ep_storage_age	Seconds since most recently
	stored object was initially queued.
ep_storage_age_highwat	ep_storage_age high water mark
ep_min_data_age	Minimum data age setting.
ep_queue_age_cap	Queue age cap setting.
ep_max_txn_size	Max number of updates per transaction.
ep_data_age	Seconds since most recently
	stored object was modified.
ep_data_age_highwat	ep_data_age high water mark
ep_too_young	Number of times an object was
	not stored due to being too young.
ep_too_old	Number of times an object was
	stored after being dirty too long.
ep_total_enqueued	Total number of items queued for
	persistence.
ep_total_new_items	Total number of persisted new items.
ep_total_del_items	Total number of persisted deletions.
ep_total_persisted	Total number of items persisted.
ep_item_flush_failed	Number of times an item failed to flush
	due to storage errors.
ep_item_commit_failed	Number of times a transaction failed to
	commit due to storage errors.
ep_item_begin_failed	Number of times a transaction failed to
	start due to storage errors.
ep_expired	Number of times an item was expired.
ep_item_flush_expired	Number of times an item is not flushed
	due to the expiry of the item
ep_queue_size	Number of items queued for storage.
ep_flusher_todo	Number of items remaining to be written.
ep_flusher_state	Current state of the flusher thread.
ep_commit_num	Total number of write commits.
ep_commit_time	Number of seconds of most recent commit.
ep_commit_time_total	Cumulative seconds spent committing.
ep_vbucket_del	Number of vbucket deletion events.
ep_vbucket_del_fail	Number of failed vbucket deletion events.
ep_vbucket_del_max_walltime	Max wall time (µs) spent by deleting
	a vbucket
ep_vbucket_del_total_walltime	Total wall time (µs) spent by deleting
	vbuckets
ep_vbucket_del_avg_walltime	Avg wall time (µs) spent by deleting
	a vbucket
ep_flush_preempts	Num of flush early exits for read reqs.
ep_flush_duration	Number of seconds of most recent flush.
ep_flush_duration_total	Cumulative seconds spent flushing.
ep_flush_duration_highwat	ep_flush_duration high water mark.
ep_flush_all	True if disk flush_all is scheduled
curr_items	Num items in active vbuckets.
curr_items_tot	Num current items including those not
	active (replica, dead and pending states)
ep_kv_size	Memory used to store keys and values.
ep_overhead	Extra memory used by rep queues, etc..
ep_max_data_size	Max amount of data allowed in memory.
ep_mem_low_wat	Low water mark for auto-evictions.
ep_mem_high_wat	High water mark for auto-evictions.
ep_total_cache_size	The total size of all items in the cache
ep_oom_errors	Number of times unrecoverable OOMs
	happened while processing operations
ep_tmp_oom_errors	Number of times temporary OOMs
	happened while processing operations
ep_bg_fetched	Number of items fetched from disk.
ep_tap_bg_fetched	Number of tap disk fetches
ep_tap_bg_fetch_requeued	Number of times a tap bg fetch task is
	requeued.
ep_num_pager_runs	Number of times we ran pager loops
	to seek additional memory.
ep_num_expiry_pager_runs	Number of times we ran expiry pager loops
	to purge expired items from memory/disk
ep_num_checkpoint_remover_runs	Number of times we ran checkpoint remover
	to remove closed unreferenced checkpoints.
ep_items_rm_from_checkpoints	Number of items removed from closed
	unreferenced checkpoints.
ep_num_value_ejects	Number of times item values got ejected
	from memory to disk
ep_num_eject_replicas	Number of times replica item values got
	ejected from memory to disk
ep_num_eject_failures	Number of items that could not be ejected
ep_num_not_my_vbuckets	Number of times Not My VBucket exception
	happened during runtime
ep_warmup_thread	Warmup thread status.
ep_warmed_up	Number of items warmed up.
ep_warmup_dups	Duplicates encountered during warmup.
ep_warmup_oom	OOMs encountered during warmup.
ep_warmup_time	Time (µs) spent by warming data.
ep_tap_keepalive	Tap keepalive time.
ep_dbname	DB path.
ep_dbinit	Number of seconds to initialize DB.
ep_dbshards	Number of shards for db store
ep_db_strategy	SQLite db strategy
ep_warmup	true if warmup is enabled.
ep_io_num_read	Number of io read operations
ep_io_num_write	Number of io write operations
ep_io_read_bytes	Number of bytes read (key + values)
ep_io_write_bytes	Number of bytes written (key + values)
ep_pending_ops	Number of ops awaiting pending vbuckets
ep_pending_ops_total	Total blocked pending ops since reset
ep_pending_ops_max	Max ops seen awaiting 1 pending vbucket
ep_pending_ops_max_duration	Max time (µs) used waiting on pending
	vbuckets
ep_bg_num_samples	The number of samples included in the avg
ep_bg_min_wait	The shortest time (µs) in the wait queue
ep_bg_max_wait	The longest time (µs) in the wait queue
ep_bg_wait_avg	The average wait time (µs) for an item
	before it is serviced by the dispatcher
ep_bg_min_load	The shortest load time (µs)
ep_bg_max_load	The longest load time (µs)
ep_bg_load_avg	The average time (µs) for an item to be
	loaded from the persistence layer
ep_num_non_resident	The number of non-resident items
ep_num_active_non_resident	Number of non-resident items in active
	vbuckets.
ep_store_max_concurrency	Maximum allowed concurrency at the storage
	layer.
ep_store_max_readers	Maximum number of concurrent read-only.
	storage threads.
ep_store_max_readwrite	Maximum number of concurrent read/write
	storage threads.
ep_db_cleaner_status	Status of database cleaner that cleans up
	invalid items with old vbucket versions
ep_bg_wait	The total elapse time for the wait queue
ep_bg_load	The total elapse time for items to be
	loaded from the persistence layer
ep_latency_get_cmd	The total elapse time for get command
ep_latency_store_cmd	The total elapse time for store command
ep_latency_arith_cmd	The total eplase time for arith command
ep_onlineupdate	True if engine is in online updated mode
ep_onlineupdate_revert_add	Number of reverted newly added items
ep_onlineupdate_revert_delete	Number of reverted deleted items
ep_onlineupdate_revert_update	Number of reverted updated items

2.2 vBucket total stats

Stat	Description
ep_vb_total	Total vBuckets (count)
curr_items_tot	Total number of items
curr_items	Number of active items in memory
vb_dead_num	Number of dead vBuckets
ep_diskqueue_items	Total items in disk queue
ep_diskqueue_memory	Total memory used in disk queue
ep_diskqueue_fill	Total enqueued items on disk queue
ep_diskqueue_drain	Total drained items on disk queue
ep_diskqueue_pendingWrites	Total bytes of pending writes

2.2.1 Active vBucket class stats

Stat	Description
vb_active_num	Number of active vBuckets
vb_active_curr_items	Number of in memory items
vb_active_num_non_resident	Number of non-resident items
vb_active_perc_mem_resident	% memory resident
vb_active_eject	Number of times item values got ejected
vb_active_ht_memory	Memory used to store keys and values
vb_active_itm_memory	Total item memory
vb_active_ops_create	Number of create operations
vb_active_ops_update	Number of update operations
vb_active_ops_delete	Number of delete operations
vb_active_ops_reject	Number of rejected operations
vb_active_queue_size	Active items in disk queue
vb_active_queue_memory	Memory used for disk queue
vb_active_queue_age	Sum of disk queue item age in milliseconds
vb_active_queue_pending	Total bytes of pending writes
vb_active_queue_fill	Total enqueued items
vb_active_queue_drain	Total drained items

2.2.2 Replica vBucket stats

Stat	Description
vb_replica_num	Number of replica vBuckets
vb_replica_curr_items	Number of in memory items
vb_replica_num_non_resident	Number of non-resident items
vb_replica_perc_mem_resident	% memory resident
vb_replica_eject	Number of times item values got ejected
vb_replica_ht_memory	Memory used to store keys and values
vb_replica_itm_memory	Total item memory
vb_replica_ops_create	Number of create operations
vb_replica_ops_update	Number of update operations
vb_replica_ops_delete	Number of delete operations
vb_replica_ops_reject	Number of rejected operations
vb_replica_queue_size	Replica items in disk queue
vb_replica_queue_memory	Memory used for disk queue
vb_replica_queue_age	Sum of disk queue item age in milliseconds
vb_replica_queue_pending	Total bytes of pending writes
vb_replica_queue_fill	Total enqueued items
vb_replica_queue_drain	Total drained items

2.2.3 Pending vBucket stats

Stat	Description
vb_pending_num	Number of pending vBuckets
vb_pending_curr_items	Number of in memory items
vb_pending_num_non_resident	Number of non-resident items
vb_pending_perc_mem_resident	% memory resident
vb_pending_eject	Number of times item values got ejected
vb_pending_ht_memory	Memory used to store keys and values
vb_pending_itm_memory	Total item memory
vb_pending_ops_create	Number of create operations
vb_pending_ops_update	Number of update operations
vb_pending_ops_delete	Number of delete operations
vb_pending_ops_reject	Number of rejected operations
vb_pending_queue_size	Pending items in disk queue
vb_pending_queue_memory	Memory used for disk queue
vb_pending_queue_age	Sum of disk queue item age in milliseconds
vb_pending_queue_pending	Total bytes of pending writes
vb_pending_queue_fill	Total enqueued items
vb_pending_queue_drain	Total drained items

2.3 Tap stats

ep_tap_total_queue	Sum of tap queue sizes on the current
	tap queues
ep_tap_total_fetched	Sum of all tap messages sent
ep_tap_bg_max_pending	The maximum number of bg jobs a tap
	connection may have
ep_tap_bg_fetched	Number of tap disk fetches
ep_tap_bg_fetch_requeued	Number of times a tap bg fetch task is
	requeued.
ep_tap_fg_fetched	Number of tap memory fetches
ep_tap_deletes	Number of tap deletion messages sent
ep_tap_throttled	Number of tap messages refused due to
	throttling.
ep_tap_keepalive	How long to keep tap connection state
	after client disconnect.
ep_tap_count	Number of tap connections.
ep_tap_bg_num_samples	The number of tap bg fetch samples
	included in the avg
ep_tap_bg_min_wait	The shortest time (µs) for a tap item
	before it is serviced by the dispatcher
ep_tap_bg_max_wait	The longest time (µs) for a tap item
	before it is serviced by the dispatcher
ep_tap_bg_wait_avg	The average wait time (µs) for a tap item
	before it is serviced by the dispatcher
ep_tap_bg_min_load	The shortest time (µs) for a tap item to
	be loaded from the persistence layer
ep_tap_bg_max_load	The longest time (µs) for a tap item to
	be loaded from the persistence layer
ep_tap_bg_load_avg	The average time (µs) for a tap item to
	be loaded from the persistence layer
ep_tap_noop_interval	The number of secs between a noop is added
	to an idle connection
ep_tap_backoff_period	The number of seconds the tap connection
	should back off after receiving ETMPFAIL
ep_tap_queue_fill	Total enqueued items
ep_tap_queue_drain	Total drained items
ep_tap_queue_backoff	Total back-off items
ep_tap_queue_backfill	Number of backfill remaining
ep_tap_queue_itemondisk	Number of items remaining on disk

2.3.1 Per Tap Client Stats

Each stat begins with ep_tapq: followed by a unique client_id and another colon. For example, if your client is named, slave1, the qlen stat would be ep_tapq:slave1:qlen.

type	The kind of tap connection (producer or	PC
	consumer)
created	Creation time for the tap connection	PC
supports_ack	true if the connection use acks	PC
connected	true if this client is connected	PC
disconnects	Number of disconnects from this client.	PC
qlen	Queue size for the given client_id.	P
qlen_high_pri	High priority tap queue items.	P
qlen_low_pri	Low priority tap queue items.	P
vb_filters	Size of connection vbucket filter set.	P
vb_filter	The content of the vbucket filter	P
rec_fetched	Tap messages sent to the client.	P
rec_skipped	Number of messages skipped due to	P
	tap reconnect with a different filter	P
idle	True if this connection is idle.	P
empty	True if this connection has no items.	P
complete	True if backfill is complete.	P
has_item	True when there is a bg fetched item	P
	ready.	P
has_queued_item	True when there is a key ready to be	P
	looked up (may become fg or bg item)	P
bg_wait_for_result	True if the max number of background	P
	operations is started	P
bg_queue_size	Number of bg fetches enqueued for this	P
	connection.	P
bg_queued	Number of background fetches enqueued.	P
bg_result_size	Number of ready background results.	P
bg_results	Number of background results ready.	P
bg_jobs_issued	Number of background jobs started.	P
bg_jobs_completed	Number of background jobs completed.	P
bg_backlog_size	Number of items pending bg fetch.	P
flags	Connection flags set by the client.	P
pending_disconnect	true if we’re hanging up on this client	P
paused	true if this client is blocked	P
pending_backfill	true if we’re still backfilling keys	P
	for this connection	P
pending_disk_backfill	true if we’re still backfilling keys	P
	from disk for this connection	P
backfill_completed	true if all items from backfill is	P
	successfully transmitted to the client	P
reconnects	Number of reconnects from this client.	P
backfill_age	The age of the start of the backfill.	P
ack_seqno	The current tap ACK sequence number.	P
recv_ack_seqno	Last receive tap ACK sequence number.	P
ack_log_size	Tap ACK backlog size.	P
ack_window_full	true if our tap ACK window is full.	P
expires	When this ACK backlog expires.	P
num_tap_nack	The number of negative tap acks received	P
num_tap_tmpfail_survivors	The number of items rescheduled due to	P
	a temporary nack.	P
queue_memory	Memory used for tap queue	P
queue_fill	Total queued items	P
queue_drain	Total drained items	P
queue_backoff	Total back-off items	P
queue_backfillremaining	Number of backfill remaining	P
queue_itemondisk	Number of items remaining on disk	P
total_backlog_size	Num of remaining items for replication	P
num_delete	Number of delete operations consumed	C
num_delete_failed	Number of failed delete operations	C
num_flush	Number of flush operations	C
num_flush_failed	Number of failed flush operations	C
num_mutation	Number of mutation operations	C
num_mutation_failed	Number of failed mutation operations	C
num_opaque	Number of opaque operation consumed	C
num_opaque_failed	Number of failed opaque operations	C
num_vbucket_set	Number of vbucket set operations	C
num_vbucket_set_failed	Number of failed vbucket set operations	C
num_unknown	Number of unknown operations	C

2.4 Tap Aggregated Stats

Aggregated tap stats allow named tap connections to be logically grouped and aggregated together by prefixes.

For example, if all of your tap connections started with rebalance_ or replication_, you could call stats tapagg _ to request stats grouped by everything before the first _ character, giving you a set for rebalance and a set for replication.

2.4.1 Results

[prefix]:count	Number of connections matching this prefix
[prefix]:qlen	Total length of queues with this prefix
[prefix]:backfill_remaining	Number of items needing to be backfilled
[prefix]:backoff	Total number of backoff events
[prefix]:drain	Total number of items drained
[prefix]:fill	Total number of items filled
[prefix]:itemondisk	Number of items remaining on disk
[prefix]:total_backlog_size	Num of remaining items for replication

2.5 Timing Stats

Timing stats provide histogram data from high resolution timers over various operations within the system.

2.5.1 General Form

As this data is multi-dimensional, some parsing may be required for machine processing. It’s somewhat human readable, but the stats script mentioned in the Getting Started section above will do fancier formatting for you.

Consider the following sample stats:

STAT disk_insert_8,16 9488
STAT disk_insert_16,32 290
STAT disk_insert_32,64 73
STAT disk_insert_64,128 86
STAT disk_insert_128,256 48
STAT disk_insert_256,512 2
STAT disk_insert_512,1024 12
STAT disk_insert_1024,2048 1

This tells you that disk_insert took 8-16µs 9,488 times, 16-32µs 290 times, and so on.

The same stats displayed through the stats CLI tool would look like this:

disk_insert (10008 total)
   8us - 16us    : ( 94.80%) 9488 ###########################################
   16us - 32us   : ( 97.70%)  290 #
   32us - 64us   : ( 98.43%)   73
   64us - 128us  : ( 99.29%)   86
   128us - 256us : ( 99.77%)   48
   256us - 512us : ( 99.79%)    2
   512us - 1ms   : ( 99.91%)   12
   1ms - 2ms     : ( 99.92%)    1

2.5.2 Available Stats

The following histograms are available from “timings” in the above form to describe when time was spent doing various things:

bg_wait	bg fetches waiting in the dispatcher queue
bg_load	bg fetches waiting for disk
bg_tap_wait	tap bg fetches waiting in the dispatcher queue
bg_tap_laod	tap bg fetches waiting for disk
pending_ops	client connections blocked for operations
	in pending vbuckets.
storage_age	Analogous to ep_storage_age in main stats.
data_age	Analogous to ep_data_age in main stats.
get_cmd	servicing get requests
store_cmd	servicing store requests
arith_cmd	servicing incr/decr requests
get_vb_cmd	servicing vbucket status requests
set_vb_cmd	servicing vbucket set state commands
del_vb_cmd	servicing vbucket deletion commands
tap_vb_set	servicing tap vbucket set state commands
tap_vb_reset	servicing tap vbucket reset commands
tap_mutation	servicing tap mutations
notify_io	waking blocked connections
disk_insert	waiting for disk to store a new item
disk_update	waiting for disk to modify an existing item
disk_del	waiting for disk to delete an item
disk_vb_del	waiting for disk to delete a vbucket
disk_vb_chunk_del	waiting for disk to delete a vbucket chunk
disk_commit	waiting for a commit after a batch of updates
disk_invalid_item_del	Waiting for disk to delete a chunk of invalid
	items with the old vbucket version

2.6 Hash Stats

Hash stats provide information on your per-vbucket hash tables.

Requesting these stats does affect performance, so don’t do it too regularly, but it’s useful for debugging certain types of performance issues. For example, if your hash table is tuned to have too few buckets for the data load within it, the max_depth will be too large and performance will suffer.

Each stat is prefixed with vb_ followed by a number, a colon, then the individual stat name.

For example, the stat representing the size of the hash table for vbucket 0 is vb_0:size.

state	The current state of this vbucket
size	Number of hash buckets
locks	Number of locks covering hash table operations
min_depth	Minimum number of items found in a bucket
max_depth	Maximum number of items found in a bucket
reported	Number of items this hash table reports having
counted	Number of items found while walking the table
resized	Number of times the hash table resized.
mem_size	Running sum of memory used by each item.
mem_size_counted	Counted sum of current memory used by each item.

2.7 Checkpoint Stats

Checkpoint stats provide detailed information on per-vbucket checkpoint datastructure.

Like Hash stats, requesting these stats has some impact on performance. Therefore, please do not poll them from the server frequently. Each stat is prefixed with vb_ followed by a number, a colon, and then each stat name.

open_checkpoint_id	ID of the current open checkpoint
num_tap_cursors	Number of referencing TAP cursors
num_checkpoint_items	Number of total items in a checkpoint datastructure
num_checkpoints	Number of checkpoints in a checkpoint datastructure

3 Details

3.1 Ages

The difference between ep_storage_age and ep_data_age is somewhat subtle, but when you consider that a given record may be updated multiple times before hitting persistence, it starts to be clearer.

ep_data_age is how old the data we actually wrote is.

ep_storage_age is how long the object has been waiting to be persisted.

3.2 Too Young

ep_too_young is incremented every time an object is encountered whose data age is more recent than is allowable for the persistence layer.

For example, if an object that was queued five minutes ago is picked off the todo queue and found to have been updated fifteen seconds ago, it will not be stored, ep_too_young will be incremented, and the key will go back on the input queue.

3.3 Too Old

ep_too_old is incremented every time an object is encountered whose queue age exceeds the ep_queue_age_cap setting.

ep_queue_age_cap generally exists as a safety net to prevent the ep_min_data_age setting from preventing persistence altogether.

3.4 Warming Up

Opening the data store is broken into three distinct phases:

3.4.1 Initializing

During the initialization phase, the server is not accepting connections or otherwise functional. This is often quick, but in a server crash can take some time to perform recovery of the underlying storage.

This time is made available via the ep_dbinit stat.

3.4.2 Warming Up

After initialization, warmup begins. At this point, the server is capable of taking new writes and responding to reads. However, only records that have been pulled out of the storage or have been updated from other clients will be available for request.

(note that records read from persistence will not overwrite new records captured from the network)

During this phase, ep_warmup_thread will report running and ep_warmed_up will be increasing as records are being read.

3.4.3 Complete

Once complete, ep_warmed_up will stop increasing and ep_warmup_thread will report complete.