Debugging (C)

C++ Guides: GAMS Primer | Creating Algorithms | Creating Platforms | Creating Threads | Creating Transports | Debugging

---

Introduction

GAMS provides many debugging features. The most powerful of these is a configurable logging system based on the MADARA Logger. Additionally, GAMS exposes a KnowledgeBase and special status variables and containers that track interactions between algorithms and platforms.

---

Table of Contents

• Introduction
• Transports
• Threads
• Checkpointing
• Logging
  • Logging Targets
  • Logging Levels
  • MADARA Logger
  • Custom Loggers

---

Transports

Transports now support debug information saved directly to the KnowledgeBase with the debug_to_kb (prefix) function. By default, with an empty function call, this will enable saving to .transport.*. With a user-specified prefix, you can save the information anywhere in a KnowledgeBase.

Debug Information

.transport.failed_receives=receives that did not return packets
.transport.failed_sends=sends that did not succeed
.transport.received_data=amount of data in bytes that have been received
.transport.received_data_max=the maximum bytes received in a packet
.transport.received_data_min=the minimum bytes received in a packet
.transport.received_packets=the number of received packets
.transport.sent_data=the amount of data in bytes that have been sent
.transport.sent_data_max=the maximum bytes sent in a packet
.transport.sent_data_min=the minimum bytes sent in a packet
.transport.sent_packets=number of sent packets

---

Threads

The Threader class now support debug information saved directly to the KnowledgeBase with the debug_to_kb (prefix) function. By default, with an empty function call, this will enable saving to .threader.*. With a user-specified prefix, you can save the information anywhere in a KnowledgeBase.

Debug Information

.threader.hertz=intended thread hertz rate
.threader.thread0.end_time=last timestamp of the thread executions
.threader.thread0.executions=number of executions
.threader.thread0.last_duration=last run duration of a thread
.threader.thread0.last_start_time=last start timestamp
.threader.thread0.max_duration=max run duration of a thread
.threader.thread0.min_duration=min run duration of a thread
.threader.thread0.start_time=the first timestamp of thread executions

---

Checkpointing

Checkpointing is the practice of saving a KnowledgeBase so that it may be used or referred to later. Example applications include forensic analysis, replay, and even rollbacks to previous state. Ultimately, the GAMS features are a direct correlation to MADARA's checkpointing features. However, the integration into the Agent BaseController is very specific. So, we'll briefly discuss this.

The BaseController checkpointing features require settings ControllerSettings that enable checkpointing be done in a certain way. There are two class members that you may find useful, and they interplay with the send_hertz and loop_hertz variables.

The first useful variable is checkpoint_prefix, which specifies where in the filesystem and any additional filename prefixes that you want checkpoints to be saved to. It is common to create a directory in the current directory where you are running a custom controller and include the directory, at the least, in the checkpoint_prefix variable.

The second useful variable is the checkpoint_strategy, which is a bitmask that contains one or more of the CheckpointStrategies enums. The enums possible are the following:

CHECKPOINT_NONE = 0,
CHECKPOINT_EVERY_LOOP = 1,
CHECKPOINT_EVERY_SEND = 2,
CHECKPOINT_SAVE_DIFFS = 4,
CHECKPOINT_SAVE_FULL_CONTEXTS = 8,
CHECKPOINT_SAVE_ONE_FILE = 16

checkpoint_strategy requires at least CHECKPOINT_EVERY_LOOP or CHECKPOINT_EVERY_SEND to be enabled for any checkpoints to be saved by the Agent Controller. The different strategies should be logically or'd together with the pipe (|). For instance, if you would like a diff-based checkpoint to be saved and layered onto the same file every loop, you could do something like this:

settings.checkpoint_strategy = CHECKPOINT_EVERY_LOOP | CHECKPOINT_SAVE_DIFFS | CHECKPOINT_SAVE_ONE_FILE = 16;

// note that there are convenience enums that will help accomplish the same thing
settings.checkpoint_strategy = CHECKPOINT_EVERY_LOOP | CHECKPOINT_SAVE_DIFFS_IN_ONE_FILE;

To print out the contents of a checkpoint, it is easiest to probably use the karl interpreter that comes with MADARA. For instance, if you have a checkpoint at my_checkpoint_0.kb, you can load and print it out with the following:

$MADARA_ROOT/bin/karl -0b my_checkpoint_0.kb -k

You can find out more useful checkpointing tidbits on the MADARA Checkpointing Wiki

---

Logging

Logging is the practice of printing messages to an output sink, usually to help with tracing execution of software. GAMS supports logging internal messages to any combination of terminal windows, system loggers, and files.

---

Logging Targets

Targets for GAMS logging are set in the same way that they are set for any MADARA Logger.

All MADARA loggers start with an output target of stderr. To add or remove targets, you can use any of the functions on the gams::loggers::global_logger instance.

// Clear all logging targets (this would remove terminal output as well)
gams::loggers::global_logger->clear ();

// Add a file output of "some_file.txt" to the logger
gams::loggers::global_logger->add_file ("some_file.txt");

// Add the terminal back to logging targets
gams::loggers::global_logger->add_term ();

---

Logging Levels

MADARA loggers allow you to set a logging level that generally goes from high priority (0) to low priority (6+). This is an inverted scale of priority that has helpful enums already setup in the gams::loggers namespace. The GAMS levels in gams::loggers are the following:

LOG_EMERGENCY // (0) use this log level if someone is an emergency
LOG_ALWAYS    // (0) the same log level as EMERGENCY
LOG_ERROR     // (1) print error information. This is the default log level.
LOG_WARNING   // (2) print warning information
LOG_MAJOR     // (3) major event information
LOG_MINOR     // (4) minor event information
LOG_TRACE     // (5) trace information through functions, rarely used
LOG_DETAILED  // (6) detailed information about execution. The most verbose logging level in GAMS

To set a log level, the global_logger provides the set_level function.

// Sets the logging level to LOG_MAJOR (any major events in GAMS)
gams::loggers::global_logger->set_level (gams::loggers::LOG_MAJOR);

---

MADARA Logger

In addition to GAMS logging, MADARA logging is also available. This logger is separate from the GAMS logger and has more fine-grained logging information available, including network transport related logging, knowledge retrieval and container usage. The MADARA logger, by default, is located at madara::logger::global_logger inside of the "madara/logger/Global_Logger.h".

// Sets the MADARA logging to detailed
madara::logger::global_logger->set_level (madara::logger::LOG_DETAILED);

// Sets the GAMS logging to minor (notice this is different from the MADARA logging level)
gams::loggers::global_logger->set_level (gams::loggers::LOG_MINOR);

// Save MADARA messages to a file called "madara.txt"
madara::logger::global_logger->add_file ("madara.txt");

// Save GAMS messages to a file called "gams.txt"
gams::loggers::global_logger->set_level (gams::loggers::LOG_MINOR);
gams::loggers::global_logger->add_file ("gams.txt");

// Note that because we did not clear either logger, the loggers will print to their files and stderr

---

Custom Loggers

Developers can create their own MADARA loggers with their own custom log levels. Creating such a custom logger in no way interferes with the GAMS or MADARA loggers, so developers are free to have multiple levels of simultaneous logging for MADARA, GAMS and custom loggers.

To create your own MADARA logger, you simply create an instance of the madara::logger::Logger class. For instance, if you wanted to create a logger for a custom algorithm and a separate logger for a custom platform, you could do so like this:

#include "madara/logger/Logger.h"

// passing false to constructor means to not add stderr to the logger
madara::logger::Logger algorithm_logger (false), platform_logger (false);

// add only file output to the loggers
algorithm_logger.add_file ("my_algorithm_debug.txt");
platform_logger.add_file ("my_platform_debug.txt");

// logging can be done very basically with the log method, which uses printf syntax and special characters
algorithm_logger.log (5, "Testing algorithm logger.\n");
platform_logger.log (4, "Testing platform logger.\n");

/** 
 * the log method is not the fastest type of logging available. MADARA
 * provides a series of macros that can reduce logging overhead by only
 * calling the log function when the log level is appropriate.
 * You can read more about them here: http://madara.sourceforge.net/docs/cpp/dd/da4/Logger_8h.html
 **/
madara_logger_log(algorithm_logger, 3, "2nd test of algorithm logger at higher priority.\n");
madara_logger_log(platform_logger, 3, "2nd test of platform logger at higher priority.\n");

C++ Guides: GAMS Primer | Creating Algorithms | Creating Platforms | Creating Threads | Creating Transports | Debugging