Feedback on working with tailscale and overlay networks - Hope it helps someone.

[CG-8663]

Troubleshooting Exo-Explorer Clustering: Resolving Overlay Network Conflicts

Overview

This guide documents the resolution of clustering issues with exo-explorer when multiple overlay networks (Tailscale and ZeroTier) are present on the same machines. The primary symptom was devices failing to discover each other despite proper network connectivity.

Environment

• Hardware: Two MacBook M1 systems (MacBook Air and Mac Studio)
• Networks: Tailscale mesh network + ZeroTier overlay network
• Software: exo-explorer (distributed AI inference framework)
• OS: macOS with Python 3.12

Problem Description

Initial Symptoms

• Both machines could run exo-explorer individually (web UI accessible)
• Manual discovery configuration attempts failed
• No peer discovery despite proper Tailscale connectivity
• Connection attempts to gRPC ports (50051) failed with "Connection refused"

Misleading Indicators

• Tailscale connectivity was perfect (ping and web UI access worked)
• Both machines showed active Tailscale status
• Network connectivity appeared normal in all standard tests

Root Cause Analysis

The Real Issue: Overlay Network Conflict

The fundamental problem was competing overlay networks rather than basic connectivity issues:

1. Discovery Protocol Interference: Both ZeroTier and Tailscale implement network discovery mechanisms that conflicted
2. IP Address Confusion: Devices discovered each other on local WiFi (192.168.x.x) instead of Tailscale IPs (100.x.x.x)
3. One-Way Discovery: MacBook Air could see Mac Studio, but clustering failed due to mixed network addressing

Key Diagnostic Findings

# MacBook Air debug output showed:
Server started, listening on 0.0.0.0:63651
Adding peer_id='...' at 192.168.68.59:53493  # Local IP, not Tailscale!
update_peers: added=[] removed=[] updated=[] unchanged=[]  # No actual connections

This revealed that discovery was happening on the wrong network interface.

Common Misconceptions

What Didn't Work (And Why)

1. Manual Discovery Configuration

   exo --discovery-module manual --discovery-config-path config.yaml

   • Issue: Manual discovery isn't implemented in exo-explorer yet
   • Status: Open bounty issue ([BOUNTY - $200] Manual networking with configuration files #279) on GitHub

2. Port-Specific Troubleshooting

   nc -v 100.105.50.99 50051  # Connection refused

   • Issue: Not a port problem, but overlay network routing conflict
   • Misleading: Suggested firewall or binding issues

3. macOS Port Reservation Theories

   • Issue: While port 50051 can be reserved on some macOS versions, this wasn't the cause
   • Evidence: exo was successfully binding to ports (visible in lsof output)

Solution

Immediate Fix: Disable Conflicting Overlay Network

# Temporarily disable ZeroTier on both machines
sudo zerotier-cli leave [network-id]
# Or stop the service entirely
sudo launchctl unload /Library/LaunchDaemons/com.zerotier.one.plist

# Restart exo with simple auto-discovery
DEBUG=9 exo

Result: Immediate successful clustering with full peer discovery.

Long-term Solutions

Option 1: Single Overlay Network

Choose either Tailscale or ZeroTier for your use case:

• Tailscale: Better for simple mesh networking, superior NAT traversal
• ZeroTier: Better for complex network topologies, more granular control

Option 2: Network Interface Binding

Force exo to use specific network interfaces:

# Bind to Tailscale interface only
DEBUG=9 exo --node-host 100.105.50.99 --node-port 50051

Option 3: Network Segmentation

Configure routing to prevent interference:

# Create specific routes for different overlay networks
sudo route add -net [zerotier-network] -interface [zt-interface]

Diagnostic Methodology

Essential Diagnostic Commands

1. Check Process Binding:

   lsof -p $(pgrep -f exo) | grep LISTEN

2. Network Connectivity Verification:

   tailscale status
   ping [tailscale-ip]
   nc -v [tailscale-ip] [port]

3. Debug Output Analysis:

   DEBUG=9 exo 2>&1 | grep -E "(discovery|peer|bind|listen|server|error)"

Key Indicators to Look For

• Successful binding: Server started, listening on 0.0.0.0:[port]
• Wrong network discovery: Peers showing local IPs instead of overlay IPs
• Empty peer updates: update_peers: added=[] removed=[] consistently
• Discovery vs. connection mismatch: Peers found but no successful connections

Prevention

Best Practices

1. Use single overlay network for distributed applications when possible
2. Explicitly bind to desired interfaces using --node-host
3. Monitor debug output during initial setup to catch network conflicts early
4. Test connectivity at multiple layers (ping, nc, application-specific)

Environment Configuration

# Recommended exo startup for Tailscale environments
DEBUG=9 exo --node-host [tailscale-ip] --node-port [fixed-port]

# Check for overlay network conflicts
netstat -rn | grep -E "(100\.|172\.|10\.)"  # Look for multiple overlay routes

Technical Notes

Why Overlay Networks Conflict

1. Broadcast Domain Overlap: Both networks attempt to handle local discovery broadcasts
2. Route Priority Conflicts: Operating system routing tables can prioritize incorrectly
3. Interface Binding Ambiguity: Applications may bind to first available interface rather than intended one

Exo-Explorer Specific Considerations

• Uses automatic network discovery by default
• Prefers local network interfaces over overlay networks in some configurations
• gRPC clustering requires bidirectional connectivity on consistent IP addresses
• Debug output is essential for diagnosing discovery vs. connection issues

Conclusion

This troubleshooting case demonstrates that modern networking issues often involve layer interaction problems rather than basic connectivity failures. When multiple overlay networks are present, applications may experience unexpected behavior even when each network functions correctly in isolation.

The resolution—disabling one overlay network—was simple once the root cause was identified, but the diagnostic process required understanding how distributed applications handle network discovery in complex networking environments.

For distributed AI frameworks like exo-explorer, network simplicity often trumps networking flexibility during initial setup and testing phases.

End Result:

[Evanev7]

Thanks for this writeup! A lot of this has changed in 1.0, hopefully for the better - but we're very much aware our defaults will not work for every overlay network and we'd love to get some feedback now that 1.0 is out. We explicitly don't network exo over tailscale as it interferes with our development setups, but we have many ideas about adding overlay support both through our own networking and through something like tailscale.

[CG-8663]

Thanks for this writeup! A lot of this has changed in 1.0, hopefully for the better - but we're very much aware our defaults will not work for every overlay network and we'd love to get some feedback now that 1.0 is out. We explicitly don't network exo over tailscale as it interferes with our development setups, but we have many ideas about adding overlay support both through our own networking and through something like tailscale.

Thanks and much appreciated, I have been decompiling the exo stuff and attempted to add android and iphones etc and got quite along the way. Love the approach and hope you guys continue to put out more great stuff. I am just pending my investment to go get my 10 x Mac M3 full spec....