Node Discovery and Transport

Node Discovery and Transport

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/axis/serve.go
• hack/lifecycle-check.go
• internal/discovery/discovery.go
• internal/discovery/discovery_test.go
• internal/discovery/udp.go
• internal/mesh/mesh_test.go
• internal/multipath/prober.go
• internal/runtimectx/context.go
• internal/runtimectx/context_test.go
• internal/transport/ssh.go
• internal/transport/ssh_config_test.go
• internal/transport/ssh_lifecycle_test.go
• internal/transport/ssh_security_test.go

Node discovery and transport constitute Layer 1 (Fact Plane) of the AXIS architecture. This layer is responsible for identifying available compute resources and establishing secure communication channels to collect telemetry. AXIS employs a hybrid discovery model combining static configuration, signed UDP beacons, and concurrent multipath probing to ensure high availability and low-latency command execution.

Discovery Mechanisms

AXIS discovers cluster members through three primary avenues: static configuration, UDP beacons, and (in scaffolded development) mesh gossip.

1. Static Configuration (nodes.yaml)

The primary source of truth is the nodes.yaml file. During the discovery phase, DiscoverResult prefills the discovery map with nodes defined in the configuration internal/discovery/discovery.go:123-135.

2. UDP Beaconing

When enabled via DiscoveryConfig, AXIS nodes broadcast and listen for encrypted UDP beacons on a configurable port (default 42424) internal/discovery/udp.go:204-213.

• Broadcaster: Each node periodically emits a Beacon containing its StableID, Hostname, IP, Role, and SSHPort internal/discovery/udp.go:232-241.
• Security: Beacons are signed using HMAC-SHA256 with a shared secret. The verifyBeacon function ensures only authorized nodes are added to the cluster internal/discovery/udp.go:69-78.
• Accumulation Window: CLI commands like axis facts open a temporary UDP listener for a duration defined by beaconWaitDuration (default ~3.25s) to capture active peers internal/discovery/discovery.go:148-160.

3. Discovery Fan-out Pattern

Discovery is executed concurrently to minimize latency. The discover function uses a semaphore pattern (maxParallel = 10) to limit simultaneous SSH probes, preventing goroutine storms in large clusters internal/discovery/discovery.go:49-51, 186-194.

Discovery Data Flow

graph TD
    subgraph "Discovery Logic (internal/discovery)"
        DR["DiscoverResult()"]
        UDP["startUDP()"]
        OC["orderedNodes()"]
    end

    subgraph "Transport & Collection"
        LC["LocalCollector"]
        RC["RemoteCollector"]
        SSH["SSHExecutor"]
    end

    CFG[("nodes.yaml")] --> DR
    DR --> UDP
    UDP -- "Signed Beacons" --> DR
    DR --> OC
    OC -- "Concurrent Probes" --> LC
    OC -- "Concurrent Probes" --> RC
    RC --> SSH
    LC --> NF[("NodeFacts")]
    RC --> NF

Loading

Sources: internal/discovery/discovery.go:72-74, internal/discovery/discovery.go:123-150, internal/discovery/discovery.go:186-203

Multipath Probing

When a node reports multiple network addresses (e.g., a physical LAN IP and a Tailscale IP), AXIS uses the multipath package to select the optimal route.

The Probe function evaluates all addresses concurrently using probeSSH internal/multipath/prober.go:24-44. To ensure the fastest physical link is preferred over overlay networks, AXIS applies a 50ms latency penalty to addresses classified as tailscale, vpn, or wireguard internal/multipath/prober.go:48-58. Docker bridge addresses (e.g., 172.17.0.0/16) are explicitly ignored as they are typically not routable for SSH internal/multipath/prober.go:104-123.

Sources: internal/multipath/prober.go:24-83, internal/multipath/prober.go:104-123

SSH Transport (SSHExecutor)

The SSHExecutor is the standard transport implementation for remote nodes. It manages the lifecycle of a secure connection, from host-key verification to command execution.

Lifecycle and Connection

1. Config Resolution: SSHExecutor calls ssh -G via runSSHConfigCommand to resolve local ~/.ssh/config settings, including IdentityFile and UserKnownHostsFile internal/transport/ssh.go:27-50, 128-134.
2. Host-Key Verification: AXIS strictly enforces host-key verification using knownhosts. It does not support an "insecure" mode; even if environment variables attempt to bypass it, the sshConfig constructor will fail if no valid keys or known_hosts entries are found internal/transport/ssh_security_test.go:10-30.
3. Connection: The Connect method attempts the dynamic fast-path (determined by multipath.Probe) first, falling back to the logical host resolution if the fast-path fails internal/transport/ssh.go:186-203.
4. Execution: Commands are executed via Run (for full output capture) or Stream (for real-time NDJSON output) internal/transport/ssh.go:229-232, 260-264.

Entity Relationship: Transport to Code

classDiagram
    class Executor {
        <<interface>>
        +Connect(ctx) error
        +Run(ctx, cmd) (string, error)
        +Close() error
     class SSHExecutor {
        +Host string
        +Port int
        +User string
        -client *ssh.Client
        +Connect(ctx) error
        +Run(ctx, cmd) (string, error)
        -resolveSSHConfig(ctx)
    }
    class Prober {
        +Probe(ctx, port, addresses) string
        -probeSSH(ctx, ip, port)
    }

    Executor <|.. SSHExecutor
    SSHExecutor ..> Prober : uses for ResolvedDialTarget
    SSHExecutor ..> "golang.org/x/crypto/ssh" : wraps

Loading

Sources: internal/transport/ssh.go:77-94, internal/multipath/prober.go:24-25

Discovery Freshness

AXIS tracks the quality of discovery via the DiscoveryFreshness struct. This metadata indicates:

• Source: Whether nodes came from static-config, udp-window, or a beacon-registry internal/discovery/discovery.go:161-171.
• Window Completeness: If a UDP window was interrupted (e.g., by context cancellation), a warning is attached to the snapshot to alert the operator that peer nodes might be missing internal/discovery/discovery.go:91-121.

Field	Description	Source
ExpectedWindowMS	Target duration for UDP accumulation.	internal/discovery/discovery.go:107
ObservedWindowMS	Actual time spent listening.	internal/discovery/discovery.go:108
BeaconNodeCount	Number of nodes found via UDP that were NOT in nodes.yaml.	internal/discovery/discovery.go:177-182

Sources: internal/discovery/discovery.go:91-121, internal/discovery/discovery.go:176-182

---