Logging

This document describes the logging and tracing system for the QiTech control server, including structured logging and different output formats.

Overview

The QiTech control server uses a modular tracing system built on top of the tracing crate, supporting multiple output formats. The system is designed to be flexible and configurable through Cargo features.

Features

The logging system supports two main features that can be enabled independently:

• tracing-fmt: Human-readable console output (default)
• tracing-journald: systemd journal integration for production Linux systems

Default Configuration

By default, the server uses tracing-fmt for development-friendly console output:

cargo run  # Uses tracing-fmt by default

Feature-Based Configuration

Format Logging (tracing-fmt)

The fmt logger provides structured, human-readable output to the console. This is the default feature and is ideal for development and debugging.

Features:

• Colored output (when supported)
• Thread names and IDs
• Line numbers and file locations
• Configurable time formats (debug vs release)
• Target filtering

Example output:

2025-06-01T10:30:45.123Z INFO server::main: Starting QiTech Control Server
2025-06-01T10:30:45.125Z DEBUG ethercat::init: Initializing EtherCAT master

Usage:

# Default behavior
cargo run

# Explicit feature selection
cargo run --features tracing-fmt --no-default-features

Journal Logging (tracing-journald)

The journald logger integrates with systemd's journal for production Linux deployments. This is the standard logging backend used on NixOS systems and other systemd-based distributions.

Features:

• Native systemd journal integration
• Structured metadata preservation
• System-level log aggregation
• Log rotation and retention policies
• Remote log collection support

Usage:

# Enable journald logging
cargo run --features tracing-journald --no-default-features

Viewing logs on NixOS:

# View all logs from the service
journalctl -u qitech-control-server -f

# Filter by log level
journalctl -u qitech-control-server -p info

# JSON output for structured data
journalctl -u qitech-control-server -o json-pretty

# Follow logs with timestamp
journalctl -u qitech-control-server -f --since "1 hour ago"

Environment Variables

Log Level Configuration

The log level is controlled through the RUST_LOG environment variable:

# Basic log levels
RUST_LOG=debug cargo run
RUST_LOG=info cargo run  # Default
RUST_LOG=warn cargo run
RUST_LOG=error cargo run

# Module-specific filtering
RUST_LOG=server=debug,ethercat=info cargo run

# Complex filtering
RUST_LOG="info,tower_http=debug,axum=debug" cargo run

Usage Examples

Development Setup

For local development with console output:

RUST_LOG=debug cargo run --features tracing-fmt

Production Setup (NixOS)

For production deployment on NixOS systems, we use journald logging as the primary backend:

RUST_LOG=info cargo run --features tracing-journald --no-default-features

This configuration is automatically used in our NixOS deployments through the system service configuration.

Adding Tracing to Code

Basic Logging

use tracing::{info, debug, warn, error, trace};

pub fn my_function() {
    trace!("Very detailed debug information");
    debug!("Debug information for developers");
    info!("General information about program execution");
    warn!("Something unexpected happened");
    error!("An error occurred: {}", error_message);
}

Structured Logging

Important: When logging to journald (systemd's logging service), key-value pairs in event logs are not properly captured. Instead, include structured data using string formatting within the message itself.

use tracing::{info, instrument};

// ❌ Don't use key-value pairs in event logs (won't work with journald)
info!(
    user_id = %user.id,
    email = %user.email,
    "Updating user profile"
);

// ✅ Use string formatting instead (works with journald)
info!(
    "Updating user profile user_id={} email={}",
    user.id,
    user.email
);

// ✅ Spans can still use structured fields
#[instrument(fields(user_id = %user.id, operation = "update"))]
pub fn update_user(user: &User) {
    info!(
        "Updating user profile user_id={} email={}",
        user.id,
        user.email
    );
}

Custom Spans

use tracing::{Span, instrument};

pub fn complex_operation() {
    let span = tracing::info_span!(
        "complex_operation",
        operation_id = 123,
        stage = "initialization"
    );
    let _enter = span.enter();

    info!("Starting complex operation");
    
    // Create child span
    let child_span = tracing::debug_span!("database_query");
    let _child_enter = child_span.enter();
    
    debug!("Executing database query");
}

Error Handling

use tracing::{error, warn};
use anyhow::Result;

pub fn operation_with_error_handling() -> Result<()> {
    match risky_operation() {
        Ok(result) => {
            info!("Operation completed successfully result={:?}", result);
            Ok(())
        }
        Err(e) => {
            error!("Operation failed error={}", e);
            Err(e)
        }
    }
}

Troubleshooting

No log output

• Check RUST_LOG environment variable
• Verify the correct features are enabled
• Ensure the logging initialization is called in main()

Journald logs not appearing

• Ensure systemd is running
• Check systemd service configuration
• Verify the tracing-journald feature is enabled

NixOS Configuration

The QiTech control server is automatically configured for optimal logging on NixOS systems:

Package Configuration

The server package is built with the tracing-journald feature enabled:

# In nixos/packages/server.nix
buildFeatures = [ "tracing-journald" ];
buildNoDefaultFeatures = true;

Service Configuration

The systemd service is configured to use journald logging:

# In nixos/modules/qitech.nix
systemd.services.qitech-control-server = {
  serviceConfig = {
    StandardOutput = "journal";
    StandardError = "journal";
    SyslogIdentifier = "qitech-control-server";
  };
  
  environment = {
    RUST_LOG = "info,tower_http=debug,axum=debug";
    QITECH_OS = "1";  # Legacy compatibility
  };
};

This configuration ensures that all logs are properly structured and integrated with the systemd journal.

For more specific configuration options, see the individual module documentation in the source code.