-
Notifications
You must be signed in to change notification settings - Fork 1
Troubleshooting Guide
Symptom-based guide organized by category. Each entry: symptoms → likely cause → debugging steps → resolution.
Symptoms:
- Players see "No lobby found! (all servers unavailable)" when typing
/lobby - New players can't join the network at all
Likely cause: All lobby servers are offline or unreachable from the proxy.
Debugging steps:
- Check if backend servers are running:
vn debug server lobby-1 - Check circuit breaker states:
/vn status - Check if all servers are drained:
/vn drain status - Check the proxy console for connection errors
Resolution:
- Start the backend servers
- If circuit breakers are all OPEN, run
/vn reloadto reset them - If servers are accidentally drained, run
/vn undrain <server> - Enable degradation mode as a safety net:
[degradation] enabled = true mode = "random"
Symptoms:
- Sometimes
/lobbyworks, sometimes it doesn't - Error appears during peak times
Likely cause: Health check timeouts causing servers to be temporarily excluded.
Debugging steps:
- Check health check timeout: is
timeout_mstoo low for your servers? - Check if servers are hitting the
max_playerscap - Run
/vn debug server <name>during peak time
Resolution:
- Increase health check timeout:
[health_checks] timeout_ms = 1000 # Increase from 500
- Increase
max_playerscaps or set to-1(uncapped) - Reduce
cache_secondsfor more responsive health checks
Symptoms:
- Player uses
/lobby, connection starts but fails - Player stays on their current server
- No retry attempt is made
Likely cause: max_retries is set to 0, or retry message is empty.
Debugging steps:
- Check
max_retriesin config:[routing] max_retries = 2 # Should be > 0
- Check the console for connection failure logs
Resolution:
- Set
max_retriesto at least 2 - Verify the
messages.retryingmessage is configured
Symptoms:
- One lobby server has many more players than others
-
/vn statusshows unbalanced distribution
Likely causes and fixes:
| Cause | How to Check | Fix |
|---|---|---|
Using random with few players |
Check selection_mode
|
Switch to least_players or power_of_two
|
| Server is drained | /vn drain status |
/vn undrain <server> |
| Circuit breaker is OPEN | /vn debug server <name> |
Fix the server or /vn reload to reset breakers |
| Unequal weights in WRR | Check config default_lobbies
|
Adjust weight values or switch modes |
max_players too low |
Check LobbyEntry config | Increase max_players or set to -1
|
| Bursty traffic | Check connection timestamps | Switch to least_connections for EMA smoothing |
Symptoms:
- All players end up on one lobby regardless of load
- Other lobbies stay empty
Likely cause: Using consistent_hash mode with few players, or only one healthy server.
Debugging steps:
- Check selection mode:
/vn status - Check how many servers are healthy:
/vn debug server <name>for each lobby - Check circuit breaker states
Resolution:
- If using
consistent_hash: this is expected behavior — players with the same UUID always go to the same server. Switch topower_of_twoorleast_playersfor even distribution. - If only one server is healthy: fix the other servers or reduce
failure_thresholdif they're being incorrectly marked as unhealthy.
Symptoms:
- Player keeps getting routed to a server they don't want
- Even after a server was temporarily down, players return to it immediately when it recovers
Likely cause: Player affinity (sticky sessions) is sending players back to their previously connected lobby due to the 0.7 stickiness factor.
Resolution:
- You can tune or disable player affinity entirely in your
navigator.tomlunder the[routing.affinity]block:[routing.affinity] enabled = true # Set to false to completely disable sticky sessions stickiness = 0.4 # Decrease from 0.7 to reduce stickiness chance (40% chance)
- If a server is completely down, the affinity system will naturally skip it because it is missing from the healthy candidates list.
- If a server is running but in an unhealthy state, ensure health checks and circuit breakers are enabled so the proxy excludes it from the selection pool automatically.
Symptoms:
-
/vn debug servershows a server as unhealthy - Server is clearly running and players can connect directly
Likely cause: Health check timeout too low, or network latency between proxy and backend.
Debugging steps:
- Check the latency in debug output:
/vn debug server <name> - Compare the latency with your
timeout_mssetting
Resolution:
- Increase timeout:
[health_checks] timeout_ms = 1000 # or higher for high-latency setups
Symptoms:
- Servers repeatedly get excluded from routing
- Circuit breaker cycles between OPEN and HALF_OPEN
Likely cause: Intermittent health check failures (network blips, server GC pauses).
Debugging steps:
- Check the failure threshold — is it too low?
- Monitor server TPS and GC behavior
- Check network stability between proxy and backend
Resolution:
- Increase
failure_threshold:[circuit_breaker] failure_threshold = 5 # Increase from 3
- Increase
cooldown_secondsto give servers more time to recover - Increase
half_open_max_teststo require more successful tests before closing
Symptoms:
- Ordinary players type
/lobby(or configured aliases like/hub) and get "You do not have permission to use this command." - The command doesn't show up in chat autocomplete/tab-completion for players.
Likely cause: A custom commands.permission is set in the config, or the default "none" was overridden in a prior version.
Debugging steps:
- Check if the player has the
velocitynavigator.usepermission. - Check the
permissionsetting innavigator.toml.
Resolution:
- Assign players the
velocitynavigator.usepermission node in your permissions manager (e.g. LuckPerms). -
OR disable the permission requirement completely by setting it to
"none"in yournavigator.toml:[commands] permission = "none"
- Run
/vn reloadto apply configuration changes.
Symptoms:
- Players see the cooldown message: "Please wait X more second(s)." when trying to run the
/lobbycommand.
Likely cause: v4.1.0 enforces a default 3-second anti-spam cooldown on command executions.
Resolution:
- Grant bypass permission to players/groups who shouldn't have a cooldown:
velocitynavigator.bypass.cooldown(or legacyvelocitynavigator.bypasscooldown). -
OR reduce/disable the cooldown in your
navigator.toml:[commands] cooldown_seconds = 0
- Run
/vn reloadto apply configuration changes.
Symptoms:
- Edited
navigator.tomlbut behavior doesn't change
Likely cause: Forgot to reload the config.
Resolution:
/vn reload
Symptoms:
- Warnings like "Config field 'update_checker.enabled' is no longer supported"
Likely cause: v3 config fields that are no longer used in v4.
Resolution:
- These are harmless warnings — v4 ignores unknown fields
- To clean up your config, remove the deprecated fields listed in the Migration Guide
Symptoms:
- "No lobby found" error even though servers exist
- Servers show as offline in debug
Likely cause: Server names in default_lobbies don't exactly match the names in velocity.toml.
Debugging steps:
- Check
velocity.tomlfor exact server names - Compare with
navigator.tomldefault_lobbies - Names are case-sensitive
Resolution:
- Ensure exact match:
# velocity.toml [servers] lobby-1 = "..." # Note: lowercase, hyphen # navigator.toml default_lobbies = ["lobby-1"] # Must match exactly
Symptoms:
- Messages show literal
<player>instead of the player's name
Likely cause: Using v3 message format without the v4 placeholder.
Resolution:
- Make sure you're running v4.1.0+
- The
<player>placeholder is supported in all message fields:[messages] connecting = "<green>Connecting <player> to a lobby...</green>"
Symptoms:
- Console shows log error:
Failed to start Prometheus exporter on port 9225: Address already in use
Likely cause: Another proxy instance or service is already using port 9225 on the host machine.
Resolution:
- Change the Prometheus bind port in your
navigator.tomlunder[metrics.prometheus.port]to a free port (e.g.9226or9230). - Reload the configuration with
/vn reload.
Symptoms:
- Console shows log error containing:
java.net.BindException: Cannot assign requested address - Prometheus server fails to start on proxy launch
Likely cause: The bind_host in your navigator.toml is configured to a specific public IP (e.g. 45.82.121.194) which is not directly bound/assigned to the local network interfaces of the host or container (extremely common in containerized environments like Pterodactyl, Docker, or other game panel hosts).
Resolution:
- Change the Prometheus bind host in your
navigator.tomlto0.0.0.0(which binds to all local network interfaces):[metrics.prometheus] enabled = true bind_host = "0.0.0.0" port = 9225
- If your container system requires binding to a specific internal IP, consult your panel's networking settings or use
127.0.0.1(if only scraping locally). - Restart the proxy (or reload the configuration via
/vn reload).
Symptoms:
- Exporter starts successfully in the console (e.g.
Started Prometheus metrics exporter on 0.0.0.0:9225). - Visiting
http://<server-ip>:9225/metricsin your browser times out or fails. - External Prometheus fails to scrape the metrics.
Likely cause: In containerized game panels (like Pterodactyl), Docker containers run on an isolated network. External traffic is only forwarded if the port is explicitly allocated to your server in the panel's configuration.
Resolution:
-
Get a Port Allocation: In your panel's web interface, go to the Network / Allocations tab and allocate a new port (e.g.,
25582). -
Update config: In your
navigator.tomlunder[metrics.prometheus], set the port to this exact allocated port:[metrics.prometheus] enabled = true bind_host = "0.0.0.0" port = 25582 # Replace with your allocated port
-
Restart the proxy (or run
/vn reload) to bind to the new port.
Symptoms:
- Bedrock/Geyser players run
/lobbybut receive a standard text list or connect directly, rather than opening a Form GUI.
Likely cause: Geyser/Floodgate is not running on the proxy, or bedrock.use_gui_for_lobby is disabled.
Resolution:
- Ensure the Floodgate plugin is installed and active on the Velocity proxy.
- Set
bedrock.use_gui_for_lobby = trueinnavigator.toml. - Run
/vn reload.
Symptoms:
- Admin runs
/vn setup grafanabut nografana-dashboard.jsonis written, or an error is printed in console.
Likely cause: File system permission issues in the proxy's directory.
Resolution:
- Ensure the user running the Velocity proxy has write permissions to the
plugins/VelocityNavigatorfolder.
If you want to manually import the Grafana dashboard for VelocityNavigator without running the /vn setup grafana command (or if the command fails due to permission/container issues), copy the JSON configuration below and paste it in Grafana (Dashboard -> New -> Import -> Paste JSON):
Click to expand Example Grafana Dashboard JSON
{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"title": "Global Metrics",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 8,
"x": 0,
"y": 1
},
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(velocitynavigator_server_players{server=~\"^$server$\"})",
"legendFormat": "Total Players",
"range": true,
"refId": "A"
}
],
"title": "Total Lobby Players",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "green",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 8,
"x": 8,
"y": 1
},
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(velocitynavigator_server_online{server=~\"^$server$\"})",
"legendFormat": "Online Lobbies",
"range": true,
"refId": "A"
}
],
"title": "Online Lobbies Count",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 8,
"x": 16,
"y": 1
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(velocitynavigator_server_drained{server=~\"^$server$\"})",
"legendFormat": "Drained Lobbies",
"range": true,
"refId": "A"
}
],
"title": "Drained Lobbies",
"type": "stat"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 5
},
"id": 5,
"title": "Server Telemetry & Performance",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth"
},
"unit": "ms"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 6
},
"id": 6,
"options": {
"legend": {
"calcs": [
"mean",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "velocitynavigator_server_latency_ms{server=~\"^$server$\"} >= 0",
"legendFormat": "{{server}}",
"range": true,
"refId": "A"
}
],
"title": "Health Check Latency",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth"
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 6
},
"id": 7,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "velocitynavigator_server_players{server=~\"^$server$\"}",
"legendFormat": "{{server}}",
"range": true,
"refId": "A"
}
],
"title": "Player Count Distribution",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"fillOpacity": 70,
"lineWidth": 1
},
"mappings": [
{
"options": {
"from": 0,
"result": {
"color": "green",
"text": "CLOSED (Healthy)"
},
"to": 0
},
"type": "range"
},
{
"options": {
"from": 1,
"result": {
"color": "yellow",
"text": "HALF_OPEN (Testing)"
},
"to": 1
},
"type": "range"
},
{
"options": {
"from": 2,
"result": {
"color": "red",
"text": "OPEN (Tripped)"
},
"to": 2
},
"type": "range"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1
},
{
"color": "red",
"value": 2
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 24,
"x": 0,
"y": 14
},
"id": 8,
"options": {
"rowHeight": 0.9,
"showValue": "always",
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "velocitynavigator_server_circuit_breaker{server=~\"^$server$\"}",
"legendFormat": "{{server}}",
"range": true,
"refId": "A"
}
],
"title": "Circuit Breaker State history",
"type": "state-timeline"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineInterpolation": "linear"
}
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 24,
"x": 0,
"y": 20
},
"id": 9,
"options": {
"legend": {
"calcs": [
"sum"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "rate(velocitynavigator_routed_connections_total{server=~\"^$server$\"}[5m])",
"legendFormat": "{{server}}",
"range": true,
"refId": "A"
}
],
"title": "Routed Connections Rate (per second)",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 26
},
"id": 14,
"title": "Traffic & Event Diagnostics",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth"
},
"unit": "pps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 27
},
"id": 10,
"options": {
"legend": {
"calcs": [
"mean",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "rate(velocitynavigator_player_joins_total[5m])",
"legendFormat": "Joins",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "rate(velocitynavigator_player_leaves_total[5m])",
"legendFormat": "Leaves",
"range": true,
"refId": "B"
}
],
"title": "Join/Leave Rates (per second)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 27
},
"id": 11,
"options": {
"displayMode": "basic",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(velocitynavigator_redirects_total) by (reason)",
"legendFormat": "{{reason}}",
"range": true,
"refId": "A"
}
],
"title": "Redirect Reasons (Why Players Moved)",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 1
},
{
"color": "red",
"value": 5
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 35
},
"id": 12,
"options": {
"displayMode": "basic",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "velocitynavigator_circuit_breaker_trips_total{server=~\"^$server$\"}",
"legendFormat": "{{server}}",
"range": true,
"refId": "A"
}
],
"title": "Circuit Breaker Trips by Server",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 35
},
"id": 13,
"options": {
"displayMode": "basic",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showUnfilled": true
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "velocitynavigator_fallback_events_total",
"legendFormat": "{{type}}",
"range": true,
"refId": "A"
}
],
"title": "Fallback Events by Type",
"type": "bargauge"
}
],
"refresh": "5s",
"schemaVersion": 38,
"style": "dark",
"tags": [
"minecraft",
"velocitynavigator",
"loadbalancer"
],
"templating": {
"list": [
{
"current": {},
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"definition": "label_values(velocitynavigator_server_online, server)",
"hide": 0,
"includeAll": true,
"multi": true,
"name": "server",
"options": [],
"query": {
"query": "label_values(velocitynavigator_server_online, server)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "VelocityNavigator Lobby Diagnostics",
"uid": "vn_lobby_diagnostics",
"version": 1
}→ See also: Operations Runbook | FAQ | Configuration Guide
Home · Quick Start · Configuration · Operations · FAQ
GitHub · Support / Discord · Report a Bug
VelocityNavigator v4.3.0 · by DemonZ Development
![]()
Getting Started
Routing
- Routing Algorithms
- Algorithm Visualizations
- Initial Join Balancing
- Contextual Routing Guide
- Player Affinity
- Health & Circuit Breakers
- Retries & Fallbacks
Player Experience
Configuration
Network & Operations
- Advanced Proxy Systems
- Redis & Multi-Proxy
- Storage & Databases
- Server Management
- Backend Lifecycle States
- HTML Dashboard
- Operations Runbook
- Prometheus & Grafana Setup
- Troubleshooting Guide
- FAQ
VelocityNavigator 4.3.0