Terraform helm provider request failing with timeout

[prashantkalkar]

We have a k8s cluster setup with private load balancer in AWS. We are using sshutle to access the load balancer for k8s API access (for kubectl , helm and terraform).

When trying to install a helm chart (prometheus operator) using terraform, the terraform apply fails with following error.

create: failed to create: the server was unable to return a response in the time allotted, but may still be processing the request (post secrets)

Here are the verbose logs for the sshuttle,

Starting sshuttle proxy (version 1.1.1).
c : Starting firewall manager with command: ['/usr/bin/sudo', '-p', '[local sudo] Password: ', '/usr/bin/env', 'PYTHONPATH=/usr/local/Cellar/sshuttle/1.1.1/libexec/lib/python3.11/site-packages', '/usr/local/Cellar/sshuttle/1.1.1/libexec/bin/python3.11', '/usr/local/bin/sshuttle', '-v', '--method', 'auto', '--firewall']
[local sudo] Password:
fw: Starting firewall with Python version 3.11.0
fw: ready method name pf.
c : IPv6 enabled: Using default IPv6 listen address ::1
c : Method: pf
c : IPv4: on
c : IPv6: on
c : UDP : off (not available with pf method)
c : DNS : on
c : User: off (not available with pf method)
c : Subnets to forward through remote host (type, IP, cidr mask width, startPort, endPort):
c :   (<AddressFamily.AF_INET: 2>, '172.16.102.0', 24, 0, 0)
c : Subnets to exclude from forwarding:
c :   (<AddressFamily.AF_INET: 2>, '127.0.0.1', 32, 0, 0)
c :   (<AddressFamily.AF_INET6: 30>, '::1', 128, 0, 0)
c : DNS requests normally directed at these servers will be redirected to remote:
c :   (<AddressFamily.AF_INET: 2>, '8.8.8.8')
c :   (<AddressFamily.AF_INET: 2>, '103.8.45.5')
c :   (<AddressFamily.AF_INET: 2>, '103.8.46.5')
c : TCP redirector listening on ('::1', 12300, 0, 0).
c : TCP redirector listening on ('127.0.0.1', 12300).
c : DNS listening on ('::1', 12299, 0, 0).
c : DNS listening on ('127.0.0.1', 12299).
c : Starting client with Python version 3.11.0
c : Connecting to server...
 s: Running server on remote host with /usr/bin/python3 (version 3.7.10)
 s: latency control setting = True
 s: auto-nets:False
c : Connected to server.
fw: setting up.
fw: >> pfctl -s Interfaces -i lo -v
fw: >> pfctl -s all
fw: >> pfctl -a sshuttle6-12300 -f /dev/stdin
fw: >> pfctl -E
fw: >> pfctl -s Interfaces -i lo -v
fw: >> pfctl -s all
fw: >> pfctl -a sshuttle-12300 -f /dev/stdin
fw: >> pfctl -E
c : DNS request from ('192.168.1.73', 52106): 40 bytes
c : DNS request from ('192.168.1.73', 57108): 40 bytes
c : DNS request from ('192.168.1.73', 63409): 57 bytes
c : DNS request from ('192.168.1.73', 57048): 57 bytes
c : DNS request from ('192.168.1.73', 59114): 109 bytes
c : DNS request from ('192.168.1.73', 59844): 109 bytes
c : Accept TCP: 192.168.1.73:49329 -> 172.16.102.5:443.
c : DNS request from ('192.168.1.73', 64798): 57 bytes
c : DNS request from ('192.168.1.73', 59981): 57 bytes
c : DNS request from ('192.168.1.73', 49322): 68 bytes
c : DNS request from ('192.168.1.73', 54626): 68 bytes
c : DNS request from ('192.168.1.73', 54945): 48 bytes
c : DNS request from ('192.168.1.73', 55200): 48 bytes
c : DNS request from ('192.168.1.73', 53375): 28 bytes
c : DNS request from ('192.168.1.73', 57985): 28 bytes
c : DNS request from ('192.168.1.73', 51768): 47 bytes
c : DNS request from ('192.168.1.73', 64589): 47 bytes
c : DNS request from ('192.168.1.73', 63502): 57 bytes
c : DNS request from ('192.168.1.73', 58325): 57 bytes
c : DNS request from ('192.168.1.73', 50428): 57 bytes
c : DNS request from ('192.168.1.73', 56129): 57 bytes
c : DNS request from ('192.168.1.73', 57504): 48 bytes
c : DNS request from ('192.168.1.73', 62979): 48 bytes
c : DNS request from ('192.168.1.73', 58371): 28 bytes
c : DNS request from ('192.168.1.73', 63581): 28 bytes
c : DNS request from ('192.168.1.73', 61743): 47 bytes
c : DNS request from ('192.168.1.73', 56517): 47 bytes
c : DNS request from ('192.168.1.73', 56760): 48 bytes
c : DNS request from ('192.168.1.73', 57073): 48 bytes
c : DNS request from ('192.168.1.73', 53859): 28 bytes
c : DNS request from ('192.168.1.73', 63881): 28 bytes
c : DNS request from ('192.168.1.73', 62735): 47 bytes
c : DNS request from ('192.168.1.73', 55605): 47 bytes
c : DNS request from ('192.168.1.73', 53272): 98 bytes
c : DNS request from ('192.168.1.73', 52973): 98 bytes
c : Accept TCP: 192.168.1.73:49356 -> 172.16.102.93:443.
 s: SW#-1:172.16.102.5:443: deleting (3 remain)
 s: SW'unknown':Mux#7: deleting (2 remain)
c : SW#10:192.168.1.73:49329: deleting (3 remain)
c : SW#10:192.168.1.73:49329: error was: nowrite: [Errno 57] Socket is not connected
c : SW'unknown':Mux#7: deleting (2 remain)
c : DNS request from ('192.168.1.73', 56686): 28 bytes
c : DNS request from ('192.168.1.73', 55989): 58 bytes
c : DNS request from ('192.168.1.73', 55895): 23 bytes
c : DNS request from ('192.168.1.73', 54610): 28 bytes

I am using MacBook Pro with MacOS Ventura:
Here is the output for uname -a command:
Darwin 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64

I believe the issues is around these lines (in the logs above):

s: SW#-1:172.16.102.5:443: deleting (3 remain)
 s: SW'unknown':Mux#7: deleting (2 remain)
c : SW#10:192.168.1.73:49329: deleting (3 remain)
c : SW#10:192.168.1.73:49329: error was: nowrite: [Errno 57] Socket is not connected
c : SW'unknown':Mux#7: deleting (2 remain)

Please let me know if its my setup issue or some sshutle bug.

[anchurisatish]

@prashantkalkar Were you able to find the fix or work around? I am facing the exact same issue only on Mac (Monterey). And its working fine when i am connected to my company's VPN.