Repository navigation
17. Troubleshooting
Reference for components misbehaving after deployment. Each section names a specific failure mode with symptoms, root cause, and fix. If you don't find your issue here, check /var/log/user-data.log on the affected instance; it captures the full provisioning output.
All three C2 servers (Mythic, Sliver, Havoc) have no public IPs. All C2 traffic must flow through the redirector. From the redirector:
ping -c 3 mythic
ping -c 3 sliver
ping -c 3 havoc| C2 | Where to look |
|---|---|
| Mythic | Active Callbacks page in the web UI |
| Sliver |
sessions command in the Sliver console |
| Havoc | Active sessions in the Havoc client |
sudo tail -20 /var/log/apache2/redirector-ssl-access.log
sudo tail -20 /var/log/apache2/redirector-access.logURI prefixes identify which C2 is receiving traffic:
/cdn/media/stream/ = Mythic
/cloud/storage/objects/ = Sliver
/edge/cache/assets/ = Havoc
cd /opt/Mythic
sudo ./mythic-cli statusExpected: eight core containers + apollo + http, all running. The localhost-binding warnings are expected and harmless.
Get admin password:
sudo cat /opt/Mythic/.env | grep MYTHIC_ADMIN_PASSWORDWeb UI from the Windows workstation: https://mythic:7443. Login mythic_admin / password from above.
docker psExpected: three containers all up: guacamole/guacamole, postgres:15, guacamole/guacd.
sudo /home/admin/test_redirector.shChecks Apache status, VirtualHost config, connectivity to all three C2 backends, and header/decoy page behavior.
Check redirect.rules is loaded:
grep -c 'RewriteCond' /etc/apache2/redirect.rulesTest security layers manually (must use a browser User-Agent, curl is blocked by redirect.rules):
UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
HEADER_VALUE="<token from terraform output deployment_info>"
TARGET="yourdomain.tld" # or <REDIR_PUBLIC_IP> for IP-only/Tunneled Access
# Should return decoy page (no header)
curl -sk -A "$UA" https://$TARGET/
# Should return decoy page (wrong header)
curl -sk -A "$UA" -H "X-Request-ID: wrong-value" https://$TARGET/cdn/media/stream/test
# Should proxy to Mythic (connection refused if no listener, expected)
curl -sk -A "$UA" -H "X-Request-ID: $HEADER_VALUE" https://$TARGET/cdn/media/stream/testwhich sliver-serverIf missing, see Sliver below.
sudo systemctl status havocIf not running: sudo systemctl start havoc. If the binary is missing, see Havoc below.
Each Guacamole SSH connection should connect without a password prompt and land at the correct hostname:
Mythic (SSH) > admin@mythic:~$
Guacamole (SSH) > admin@guac:~$
Redirector (SSH) > admin@redirector:~$
Sliver (SSH) > admin@sliver:~$
Havoc (SSH) > admin@havoc:~$
Kali (SSH) > admin@kali:~$
SSH access model: All lab hosts are accessed via the Guacamole portal (SSH connections) or MobaXterm pre-configured sessions on the Windows operator workstation. The redirector accepts only ports 80 and 443 publicly. Guacamole has a break-glass direct SSH rule for localPub_ip (key-based, instructor only) if the portal itself is unavailable.
Warning
Do not destroy individual components with targeted terraform destroy -target or terraform taint on the redirector alone. Partial destroys can corrupt Terraform state and leave orphaned AWS resources (dangling ENIs, stale security group rules, broken VPC peering). If the redirector is in a broken state that cannot be recovered by restarting Apache or re-running setup scripts, the supported recovery path is a full terraform destroy && terraform apply.
Note
AWS and Azure cloud IP blocks are commented out by default in redirect.rules because this lab runs in AWS. If you deploy outside cloud environments you can re-enable them by editing /etc/apache2/redirect.rules and uncommenting the relevant blocks, then sudo systemctl reload apache2.
Symptoms: Agent executes but no callback in Mythic / Sliver / Havoc.
Checklist:
- Listener is running on the C2 server
- Callback Host and Port match the redirector's domain/IP and port 443
- Agent sends the correct
X-Request-IDheader with the auto-generated token - Agent URI uses the correct prefix (
/cdn/media/stream/,/cloud/storage/objects/, or/edge/cache/assets/) - Redirector Apache is running with all VirtualHosts enabled
- Redirector can reach the C2 server's private IP (
ping mythic) - Agent user-agent isn't blocked by
redirect.rules(no scanner/AV strings)
Debug on the redirector:
sudo apache2ctl -S
systemctl status apache2
# Logs (differentiate by URI prefix)
sudo tail -100 /var/log/apache2/redirector-ssl-access.log
sudo tail -100 /var/log/apache2/redirector-ssl-error.log
# Pre-installed end-to-end test
sudo /home/admin/test_redirector.shCause: AWS Marketplace EULA for the Kali Linux AMI hasn't been accepted on this account.
Fix: Visit https://aws.amazon.com/marketplace/pp/prodview-fznsw3f7mq7to, click Continue to Subscribe, then Accept Terms. Re-run terraform apply, no state cleanup needed. See README Step 0.5.
# List available keys
aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName'
# Update terraform.tfvars with correct nameAWS accounts have a default limit of five VPCs per region. redStack creates two (team server + redirector), so you need at least two free slots.
If at the limit, either delete unused VPCs or switch to the existing default VPC: in terraform.tfvars, set use_default_vpc = true.
The Kali Linux AMI is the only AWS Marketplace dependency in redStack. All other AMIs (Redirector, Mythic, Sliver, Havoc, Guacamole, Windows) are standard Amazon-published AMIs with no Marketplace gate. If you hit OptInRequired for any non-Kali host, check that terraform.tfvars is not pointing at a custom AMI ID.
← Previous: Cost Management | Next: redStack on GitHub →
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."
Brian Kernighan, The Elements of Programming Style (1978)