Architecture and Design

Architecture and Design

System Architecture

The Socat Network Operations Manager is organized as a layered architecture where each layer has a single responsibility and clear interfaces with adjacent layers. Data flows top-down from user input to system calls, and control flows bottom-up from process events (death, port binding) back to the management layer.

Layer Diagram

graph TB
    subgraph "Entry Points"
        CLI["__main__.py<br/>CLI Dispatch<br/>261 lines"]
        MENU["menu.py<br/>Interactive TUI<br/>948 lines"]
        STANDALONE["socat-manager.py<br/>Standalone Runner<br/>50 lines"]
    end

    subgraph "Mode Handlers (modes/)"
        LISTEN["listen.py<br/>251 lines"]
        BATCH["batch.py<br/>255 lines"]
        FORWARD["forward.py<br/>224 lines"]
        TUNNEL["tunnel.py<br/>277 lines"]
        REDIRECT["redirect.py<br/>209 lines"]
        STATUS["status.py<br/>162 lines"]
        STOP["stop.py<br/>207 lines"]
    end

    subgraph "Core Services"
        PROCESS["process.py<br/>Launch + Stop<br/>612 lines"]
        SESSION["session.py<br/>CRUD + Locking<br/>939 lines"]
        WATCHDOG["watchdog.py<br/>Monitor + Restart<br/>323 lines"]
        COMMANDS["commands.py<br/>Command Builders<br/>291 lines"]
        CERTS["certs.py<br/>TLS Certs<br/>122 lines"]
    end

    subgraph "Foundation"
        CONFIG["config.py<br/>Constants + Types<br/>378 lines"]
        VALIDATION["validation.py<br/>9 Validators<br/>551 lines"]
        LOGGING["logging_setup.py<br/>Structured Logging<br/>447 lines"]
    end

    STANDALONE --> CLI
    CLI --> MENU
    CLI --> LISTEN & BATCH & FORWARD & TUNNEL & REDIRECT & STATUS & STOP

    LISTEN & BATCH & FORWARD & REDIRECT --> COMMANDS
    TUNNEL --> COMMANDS & CERTS
    LISTEN & BATCH & FORWARD & TUNNEL & REDIRECT --> PROCESS
    LISTEN & BATCH & FORWARD & TUNNEL & REDIRECT --> WATCHDOG
    STATUS & STOP --> SESSION
    STOP --> PROCESS

    PROCESS --> SESSION
    PROCESS --> LOGGING
    SESSION --> CONFIG
    VALIDATION --> CONFIG
    COMMANDS --> CONFIG

Loading

Design Principles

1. Zero external runtime dependencies: The entire framework runs on the Python 3.12+ standard library. No PyPI packages are needed at runtime. This eliminates supply chain risk and simplifies deployment to any Linux system with Python.

2. Defense-in-depth security: Seven independent security layers (input validation, argument-list subprocess, process isolation, file permissions, protocol scoping, session integrity, process verification) provide overlapping protection. A failure in any single layer does not compromise the system.

3. Process group isolation: Every socat process runs in its own session and process group via os.setsid(). The PGID equals the PID (because the process is the session leader). This means killing the management script cannot affect managed sessions, and os.killpg() reliably targets the entire process tree.

4. Protocol-scoped operations: TCP and UDP sessions are independently tracked and managed even when sharing the same port number. The stop sequence reads the PROTOCOL field and only targets that specific protocol.

5. Session interoperability: Session files use KEY=VALUE text format readable by both the Python and bash variants. This enables cross-variant session management.

6. Watchdog monitor-first: The watchdog receives the PID of an already-running process and monitors it. It never launches the initial process. This eliminates the duplicate-launch bug class entirely. When a restart does replace the process, the watchdog writes the new process identity back to the session file, so the durable record always names the process that owns the port.

---

Process Lifecycle

This sequence diagram shows the complete flow from user command to running socat process with watchdog monitoring.

sequenceDiagram
    participant User
    participant CLI as __main__.py
    participant Mode as mode_listen.py
    participant Val as validation.py
    participant Cmd as commands.py
    participant Proc as process.py
    participant Session as session.py
    participant WD as watchdog.py
    participant Socat as socat (OS)

    User->>CLI: socat-manager listen --port 8080 --watchdog
    CLI->>CLI: build_parser().parse_args()
    CLI->>CLI: check_socat()
    CLI->>Mode: dispatch_mode(args)
    
    Note over Mode: Validation Phase
    Mode->>Val: validate_port("8080")
    Val-->>Mode: 8080 (int)
    Mode->>Val: validate_protocol(None) → "tcp4"
    Val-->>Mode: "tcp4"
    Mode->>Proc: check_port_available(8080, "tcp4")
    Proc-->>Mode: True
    
    Note over Mode: Command Construction
    Mode->>Cmd: build_socat_listen_cmd("tcp4", 8080, logfile)
    Cmd-->>Mode: ["socat", "-u", "TCP4-LISTEN:8080,...", "OPEN:..."]
    
    Note over Mode: Process Launch
    Mode->>Proc: launch_socat_session("tcp4-8080", "listen", "tcp4", 8080, cmd)
    Proc->>Session: generate_session_id()
    Session-->>Proc: "a1b2c3d4"
    Proc->>Socat: Popen(cmd, preexec_fn=os.setsid, close_fds=True)
    Socat-->>Proc: pid=12345
    Proc->>Proc: sleep(0.3) -- stability check
    Proc->>Proc: os.kill(12345, 0) -- alive?
    Proc->>Session: session_register(sid="a1b2c3d4", pid=12345, pgid=12345, ...)
    Proc-->>Mode: ("a1b2c3d4", 12345)
    
    Note over Mode: Watchdog Start
    Mode->>WD: start_watchdog("a1b2c3d4", "tcp4-8080", cmd, initial_pid=12345)
    WD->>WD: Thread(daemon=True).start()
    
    Note over WD: Phase 1: Monitor existing PID
    loop Every 1 second
        WD->>WD: os.kill(12345, 0) -- still alive?
    end
    
    Mode-->>User: Listener active on tcp4:8080 (SID a1b2c3d4)

Loading

---

Stop Sequence (9 Steps)

The stop sequence is protocol-scoped and uses escalating force. Each step is logged with the session's correlation ID.

sequenceDiagram
    participant User
    participant Stop as mode_stop.py
    participant Proc as process.py
    participant OS as Operating System
    participant Session as session.py

    User->>Stop: socat-manager stop a1b2c3d4
    Stop->>Session: Find session file
    Stop->>Proc: stop_session("a1b2c3d4")
    
    Proc->>Session: Read PID=12345, PGID=12345, PROTOCOL=tcp4, PORT=8080
    Note over Proc: Step 1 -- Read metadata
    
    Proc->>OS: touch sessions/a1b2c3d4.stop
    Note over Proc: Step 2 -- Signal watchdog (do not restart)
    
    Proc->>OS: os.killpg(12345, SIGTERM)
    Note over Proc: Step 3 -- SIGTERM process group
    
    Proc->>OS: os.kill(12345, SIGTERM)
    Proc->>OS: pkill -TERM -P 12345
    Note over Proc: Step 4 -- SIGTERM PID + children
    
    loop Step 5 -- Grace period (5s max, 0.5s polls)
        Proc->>OS: os.kill(12345, 0)
        OS-->>Proc: alive/dead?
    end
    
    alt Still alive after grace period
        Proc->>OS: os.killpg(12345, SIGKILL)
        Proc->>OS: os.kill(12345, SIGKILL)
        Proc->>OS: pkill -KILL -P 12345
        Note over Proc: Step 6 -- Force SIGKILL
    end
    
    Proc->>OS: process_is_running(12345) -- zombie-aware final check
    Proc->>OS: reap_child(12345) -- collect the terminated child
    Note over Proc: Step 6b -- Verify PID dead and collect
    
    Proc->>OS: ss -t -4 -l -n -p :8080 (TCP/IPv4 scope only)
    Proc->>OS: verify /proc/{pid}/comm == "socat"
    Proc->>OS: SIGKILL confirmed socat PIDs
    Note over Proc: Step 7 -- Protocol-scoped port cleanup
    
    Proc->>OS: check_port_available(8080, "tcp4") × 5
    Note over Proc: Step 8 -- Verify port freed
    
    Proc->>Session: rm sessions/a1b2c3d4.session .stop .launching
    Note over Proc: Step 9 -- Unregister session

Loading

---

Watchdog State Machine

The watchdog has two distinct phases: monitoring the initial (already-running) process, and the restart loop after death. This two-phase design eliminates the duplicate-launch bug.

stateDiagram-v2
    [*] --> MonitorInitial: start_watchdog(initial_pid)
    
    MonitorInitial --> CheckStopInitial: PID dies
    MonitorInitial --> MonitorInitial: PID alive (poll 1s)
    
    CheckStopInitial --> GracefulExit: .stop file exists
    CheckStopInitial --> RestartLoop: No .stop file
    
    state RestartLoop {
        [*] --> Sleep: backoff delay
        Sleep --> CheckStopRestart: after delay
        CheckStopRestart --> GracefulExit: .stop file exists
        CheckStopRestart --> Launch: no .stop file
        Launch --> UpdateRecord: PID returned
        Launch --> FatalExit: launch failed (PID=0)
        UpdateRecord --> MonitorReplacement: session_update_process(pid, pgid)
        MonitorReplacement --> CheckRestart: PID dies
        CheckRestart --> CheckStopRestart: restart_count < max
        CheckRestart --> MaxRestartsExit: restart_count >= max
    }
    
    GracefulExit --> [*]: session_unregister()
    FatalExit --> [*]: session_unregister()
    MaxRestartsExit --> [*]: session_unregister()
    
    note right of MonitorInitial
        Phase 1: Poll process_is_running()
        Zero CPU when healthy
        Check .stop file each poll
    end note
    
    note right of Sleep
        Exponential backoff:
        1s → 2s → 4s → 8s → 16s → 32s → 60s cap
        Configurable initial via --backoff
    end note
    
    note right of UpdateRecord
        Session file rewritten with the
        replacement PID and PGID.
        Liveness checks and the stop
        sequence read this record.
    end note

Loading

Death Detection and Child Collection

Every socat process launched by the framework is a direct child of the management process. When a child exits, the kernel retains its process table entry until the parent collects its exit status; until that happens the entry is a zombie.

A zombie still answers signal 0. Liveness therefore cannot be decided by os.kill(pid, 0) alone: an exited child would keep reporting as alive for as long as it went uncollected, and the watchdog -- which waits for exactly that signal to decide a process has died -- would block indefinitely on its first dead process and stop restarting at all.

The framework retains the Popen handle of every process it launches, in a lock-guarded registry keyed by PID. process_is_running() consults that handle first: polling it collects a terminated child and reports termination truthfully. For a process the current instance did not launch, it falls back to signal 0 qualified by a zombie check. When death is observed, reap_child() collects the exit status and drops the handle, which clears the process table entry and keeps the registry bounded to processes that are still running.

---

Process Identity Across Restarts

A restart replaces the process behind a session, so the session file is rewritten with the replacement PID and PGID before the watchdog begins monitoring it. The session file is the authoritative record of process identity: session_is_alive() and the stop sequence both read it, and neither has any other way to discover which process currently owns the port.

If the record were left naming the terminated predecessor, three things would follow. Status output would report the session dead while the replacement continued serving traffic. The stop sequence would signal a PID that no longer exists, so the live replacement would survive the stop and be reachable only through the port-based fallback in step 7. A stop --all would surface the survivor only as an orphaned socat process in the closing report.

The update runs under the advisory session lock and is published by writing a temporary file with 0o600 permissions and renaming it over the original. Rename within a directory is atomic, so a concurrent reader sees either the complete previous record or the complete new one. Only PID and PGID change; the session name, mode, protocol, ports, socat command, correlation ID, launcher PID, and the STARTED creation timestamp are preserved exactly. STARTED records when the session was created, not when the current process started, so it is not rewritten on a restart.

---

Session File Entity Relationship

erDiagram
    SESSION_FILE {
        string SESSION_FILE_VERSION "v2.3"
        string SESSION_ID "8-char hex: a1b2c3d4"
        string SESSION_NAME "tcp4-8080"
        int PID "socat process PID"
        int PGID "process group ID = PID under setsid"
        string MODE "listen|batch-listen|forward|tunnel|redirect"
        string PROTOCOL "tcp4|tcp6|udp4|udp6|tls"
        int LOCAL_PORT "1-65535"
        string REMOTE_HOST "hostname or IP (optional)"
        string REMOTE_PORT "port (optional)"
        string STARTED "ISO 8601 timestamp"
        string SOCAT_CMD "full socat command"
        string CORRELATION "8-char correlation ID"
        int LAUNCHER_PID "management script PID"
    }
    
    LOG_FILES ||--o{ SESSION_FILE : "associated via SID and port"
    LOG_FILES {
        string master_log "socat-manager-TIMESTAMP.log"
        string session_log "session-SID.log"
        string listener_log "listener-PROTO-PORT.log"
        string capture_log "capture-PROTO-PORT-TIMESTAMP.log"
    }
    
    STOP_SIGNAL ||--o| SESSION_FILE : "coordination"
    STOP_SIGNAL {
        string stop_file "SID.stop"
        string launching_file "SID.launching"
    }

Loading

---

Input Validation Flow

Every path from user input to system call passes through whitelist validation.

flowchart LR
    USER[User Input] --> V{Which Validator?}
    V -->|port number| VP["validate_port()<br/>Numeric, 1-65535"]
    V -->|hostname/IP| VH["validate_hostname()<br/>RFC 1123, IPv4, IPv6<br/>Rejects: ;|&$$\`(){}[]<>!#"]
    V -->|protocol| VPR["validate_protocol()<br/>Exact set match"]
    V -->|socat opts| VSO["validate_socat_opts()<br/>Whitelist: a-zA-Z0-9=,.:/_-"]
    V -->|session name| VSN["validate_session_name()<br/>Whitelist: a-zA-Z0-9._-<br/>Max 64 chars"]
    V -->|file path| VFP["validate_file_path()<br/>Exists + readable<br/>No metacharacters"]
    V -->|port range| VPR2["validate_port_range()<br/>START-END, max span 1000"]
    V -->|port list| VPL["validate_port_list()<br/>Comma-separated valid ports"]
    V -->|session ID| VSI["validate_session_id()<br/>Exact: ^[a-f0-9]{8}$$"]

    VP & VH & VPR & VSO & VSN & VFP & VPR2 & VPL & VSI --> VALID[Validated Input]
    VALID --> CMD["commands.py<br/>Build argument list"]
    CMD --> POPEN["subprocess.Popen(args_list)<br/>shell=False, close_fds=True"]

Loading

---

Module Dependency Graph

Arrows indicate import dependencies. The foundation layer (config, validation, logging) has no dependencies on higher layers. This prevents circular imports.

graph TD
    MAIN["__main__<br/>261 lines"] --> CLI["cli<br/>337 lines"]
    MAIN --> MENU["menu<br/>948 lines"]
    MAIN --> DISPATCH["modes/*"]

    DISPATCH --> PROCESS["process<br/>612 lines"]
    DISPATCH --> WATCHDOG["watchdog<br/>323 lines"]
    DISPATCH --> COMMANDS["commands<br/>291 lines"]
    DISPATCH --> CERTS["certs<br/>122 lines"]
    DISPATCH --> VALIDATION["validation<br/>551 lines"]

    PROCESS --> SESSION["session<br/>939 lines"]
    PROCESS --> COMMANDS
    PROCESS --> LOGGING["logging_setup<br/>447 lines"]
    WATCHDOG --> SESSION
    WATCHDOG --> LOGGING

    SESSION --> CONFIG["config<br/>378 lines"]
    SESSION --> LOGGING
    COMMANDS --> CONFIG
    VALIDATION --> CONFIG
    VALIDATION --> LOGGING
    LOGGING --> CONFIG
    CERTS --> LOGGING
    MENU --> VALIDATION

    style CONFIG fill:#e8f4e8
    style VALIDATION fill:#fff3e0
    style LOGGING fill:#e3f2fd

Loading

---

Session ID System

Why Session IDs?

PID-based tracking (used in v1.0 of the bash variant) broke because:

• PIDs get reused by the OS after process death
• Multiple sessions on the same port (dual-stack) need unique identifiers
• Cross-invocation tracking requires a stable identifier that survives PID recycling

Implementation

Session IDs are 8-character lowercase hex strings derived from uuid.uuid4():

sid = uuid.uuid4().hex[:SESSION_ID_LENGTH]  # SESSION_ID_LENGTH = 8

Collision checking: before use, verify that sessions/{sid}.session doesn't already exist. Retry up to 100 attempts (failure is a RuntimeError).

Session Naming

Session names are human-readable identifiers auto-generated from mode, protocol, port, and host:

Mode	Pattern	Example
listen	{proto}-{port}	tcp4-8080
batch	{proto}-{port}	tcp4-8080
forward	fwd-{lport}-{rhost}-{rport}	fwd-8080-10.0.0.5-80
tunnel	tunnel-{lport}-{rhost}-{rport}	tunnel-4443-10.0.0.5-22
redirect	redir-{proto}-{lport}-{rhost}-{rport}	redir-tcp4-8443-example.com-443

Custom names can be provided via --name on listen, forward, tunnel, and redirect modes.

---

Process Launch: Why Each Component Matters

The Python variant's launch is simpler than bash because Popen.pid gives direct access to the real socat PID -- no PID-file handoff needed.

Component	What It Does	Without It
preexec_fn=os.setsid	Creates new session + process group	os.killpg() would kill the management script too
Popen.pid	Direct socat PID access	You'd need a PID-file handoff (bash's problem)
close_fds=True	Prevents FD leakage to child	Child inherits management script's open files
stdout=DEVNULL	Discards socat stdout	Stdout pipe keeps parent blocked until child exits
stdin=DEVNULL	Prevents stdin inheritance	Child could read management script's terminal input
Return (sid, pid)	PID available to watchdog	Watchdog must launch its own process (BUG-01)
sleep(0.3) + kill(pid, 0)	Stability check	Silently register a session that died on startup

Why os.setsid() Eliminates disown

The bash variant uses setsid to create a new session. The Python variant does the same via preexec_fn=os.setsid. In both cases:

• The child process is NOT in the parent's session
• The parent's exit does NOT send SIGHUP to the child
• No "terminated" messages appear on the parent's terminal
• wait() in the parent never blocks on the child

This is stronger isolation than Python's os.setpgrp() alone, which only changes the process group without creating a new session.

---

Protocol Scoping in the Stop Path

The protocol model has four members: tcp4, tcp6, udp4, and udp6. Scope is therefore two-dimensional -- a transport and an address family -- and every port operation preserves both. Every function in the stop path reads the session's PROTOCOL field and derives its query scope from it:

• check_port_available(port, proto) lists only listeners of that transport and family
• check_port_freed(port, proto) verifies release only for that transport and family
• kill_by_port(port, proto) enumerates candidate PIDs only within that transport and family, and terminates only processes whose command name is socat

The scope is expressed through the socket-listing flags. Both ss and netstat accept the same vocabulary: -t or -u selects the transport, -4 or -6 selects the address family, and -l -n restricts the listing to listening sockets without name resolution. A tcp4 query runs ss -t -4 -l -n; a udp6 query runs ss -u -6 -l -n. The cleanup path adds -p to attach the owning process to each row. The lsof fallback carries the same family scope through its own -i4 and -i6 selectors.

Session protocol	Listing scope
tcp4	TCP, IPv4
tcp6	TCP, IPv6
udp4	UDP, IPv4
udp6	UDP, IPv6

Why the address family matters as much as the transport: a listener on tcp4 and a listener on tcp6 are independent sockets that can hold the same port number at the same time -- this is exactly what dual-stack operation creates. A query that names the transport but omits the family reports both of them. Two failures follow from that. A launch check would see the IPv6 listener and refuse an available IPv4 port. More seriously, the port-based cleanup step would enumerate the other family's socat process and terminate it, so stopping one session would take down an unrelated one.

Scoping on both dimensions means stopping a TCP session on port 8080 has zero effect on a UDP session on port 8080, and stopping an IPv4 session on port 8080 has zero effect on an IPv6 session on port 8080.

---

Module Ordering and Dependency Rules

The framework follows a strict dependency hierarchy to prevent circular imports:

Level 0 (foundation): config.py -- no internal imports
Level 1 (services):   logging_setup.py -- imports config
                       validation.py -- imports config, logging_setup
Level 2 (core):       session.py -- imports config, logging_setup
                       commands.py -- imports config
                       certs.py -- imports logging_setup
Level 3 (process):    process.py -- imports config, logging_setup, session, commands
                       watchdog.py -- imports config, logging_setup, session
Level 4 (modes):      modes/*.py -- imports validation, commands, process, watchdog, logging_setup
Level 5 (entry):      cli.py -- standalone (no internal imports)
                       menu.py -- imports validation, logging_setup (deferred: cli, __main__)
                       __main__.py -- imports cli, menu, modes (deferred imports to avoid cycles)

Menu and main use deferred imports (inside functions) for cli and dispatch_mode to break what would otherwise be circular dependencies.

Audit Store: Supplement, Not Replacement

The SQLite audit store in audit.py is a durable, append-only record of framework activity. It is deliberately not part of the live control path: the KEY=VALUE session files remain the single source of truth for live state, liveness checks, the stop sequence, and interoperability with the bash variant. The audit store only observes.

Two design rules keep this boundary clean:

• Failure isolation. Every public audit function catches all exceptions, logs at WARNING, and returns. A locked database, a full disk, or a permission error can never propagate into launch_socat_session, stop_session, or the watchdog. Auditing failing silently is always preferable to auditing breaking an operation.
• No live-state dependency. Nothing in the launch, stop, liveness, or restart logic reads back from the audit store to make decisions. Removing the store (or disabling it with --no-audit) changes nothing operationally.

Emission points

Source	Event(s)	History effect
process.launch_socat_session	launch	record_session_start (INSERT OR IGNORE)
process.stop_session	stop or stop_failed	record_session_end (stopped / stop_failed)
watchdog.watchdog_loop (death detected)	crash	-
watchdog.watchdog_loop (relaunch ok)	restart	record_restart (increment)
watchdog.watchdog_loop (max restarts)	-	record_session_end (watchdog_exhausted)

The correlation ID that ties log lines to a single invocation is stored on each event, so the audit trail and the logs can be joined after the fact.