Statistics

Statistics are available for users to monitor the behaviour of each filter and modify parameters when necessary. Stats are also a good way to get some insights on filters performance.

Filters Stats

Those stats are available when requesting directly the filter's monitoring socket (although, monitoring the manager will always be preferred). Those stats are:

• status [string]: one of [starting, configuring, running, stopping, unknown]
• connections [number]: represents the number of open connections from clients
• received [number]: represents the number of ENTRIES received (in case of bulk calls, will represent the number of entries in the call)
• entryErrors [number]: represents the number of entries that have failed filter parsing or processing, you can get the number of processed entries by doing received - entryErrors
• matches [number]: represents the number of matching entries, that is the entries with a certitude above or equal to the configured threshold

Manager Stats

Those stats are given by the manager, and are:

• failures [number]: represents the number of crashes/restart the filter has gone into. If you encounter those, please come to us!
• proc_stats [object]: this key represents system statistics, queried by the manager using psutil (refer to their documentation for detailed information, some keys might not be present on your system):
  • cmdline
  • connections
  • cpu_affinity
  • cpu_num
  • cpu_percent
  • cpu_times
  • create_time
  • cwd
  • environ
  • exe
  • gids
  • io_counters
  • ionice
  • memory_full_info
  • memory_info
  • memory_maps
  • memory_percent
  • name
  • nice
  • num_ctx_switches
  • num_fds
  • num_handles
  • num_threads
  • open_files
  • pid
  • ppid
  • status
  • terminal
  • threads
  • uids
  • username

Regular reporting

With Darwin v1.2, the manager gains statistics on requests, but also regular reporting to Redis pubsub/lists and regular files.

To activate this capability, the reporting should be configured in the main configuration file (the configuration file should be v2, see Darwin-configuration for details). It can be done before launching Darwin, but one can also modify the configuration while Darwin is running, and send an update_filters command to apply the changes (see Management-socket. If no filters need to be updated, simply apply the command without specifying the filters' list to update).

Reporting format

The format of statistics returned by the manager are as follows:

{
	"hostlookup": {
		"status": "running",
		"connections": 0,
		"received": 0,
		"entryErrors": 0,
		"matches": 0,
		"failures": 0,
		"proc_stats": {
			"cpu_percent": 0.0,
			"memory_percent": 0.22171612167936347
		}
	},
	"tanomaly": {
		"status": "running",
		"connections": 0,
		"received": 0,
		"entryErrors": 0,
		"matches": 0,
		"failures": 0,
		"proc_stats": {
			"cpu_percent": 0.0,
			"memory_percent": 0.6964609050296497
		}
	}
}

When the automatic reporting is activated, an additional key timestamp will be added, containing the timestamp as a valid RFC3339 UTC timestamp:

"timestamp": "1970-01-01T00:00:00.000Z"