Can't get applications to respond #14862

aug70 · 2022-07-25T20:22:03Z

aug70
Jul 25, 2022

Expected behavior:

Applications are accessible.

Current behavior:

Applications are NOT accessible.

Bug details:

Teleport version: v10.0.2 git:v10.0.2-0-g47e0914fb go1.18.3
Recreation steps: Create a simple app that runs on teleport host like nginx or just simply enable debug app dumper.
Debug logs: Below

I'm trying to get teleport applications working, I tried every possible configuration, I have a virtual server forward on my router that connects to;

An Nginx Proxy Manager (this configuration works for everything except doesn't work for applications)

or

Directly to the Node (this configuration works for everything except doesn't work for applications)

Below is my teleport.yaml file. I integrated with Github connector and everything seems to be working fine. I can add hosts and be able to open a session with them, all works fine as desired.
Except when I try to add applications, like enable debug app or just try to add a simple nginx app. They show on the dashboard but none of them work.

Bottom line, applications are visible on the dashboard but NOT accessible. I appreciate if I can get any help on this.

# teleport version
Teleport v10.0.2 git:v10.0.2-0-g47e0914fb go1.18.3

version: v2
teleport:
  nodename: node1
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: teleport.my.domain
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: teleport.my.domain:443
  acme: {}
app_service:
  enabled: "yes"
  debug_app: false
  apps:
  - name: nginx
    uri: http://localhost
    insecure_skip_verify: true
    public_addr: ""

Errors appear with DEBUG

 teleport[8041]: 2022-07-25T19:58:36Z DEBU [PROXY:AGE] Failed to create client to teleport.my.domain:443. error:[
 teleport[8041]: ERROR REPORT:
 teleport[8041]: Original Error: *errors.errorString ssh: handshake failed: ssh: overflow reading version string
 teleport[8041]: Stack Trace:
 teleport[8041]:         /go/src/github.com/gravitational/teleport/api/observability/tracing/ssh/ssh.go:98 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:74 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:350 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:305 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:302 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:261 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:245 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
 teleport[8041]:         /opt/go/src/runtime/asm_arm64.s:1263 runtime.goexit
 teleport[8041]: User Message: ssh: handshake failed: ssh: overflow reading version string] cluster:teleport.my.domain reversetunnel/agent_dialer.go:81
 teleport[8041]: 2022-07-25T19:58:36Z DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:586 target:teleport.my.domain:443 cluster:teleport.my.domain reversetunnel/agent.go:273
 teleport[8041]: 2022-07-25T19:58:36Z DEBU [PROXY:AGE] Failed to connect agent. error:[
 teleport[8041]: ERROR REPORT:
 teleport[8041]: Original Error: *trace.BadParameterError failed to dial: all auth methods failed
 teleport[8041]: Stack Trace:
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:98 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:350 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:305 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:302 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:261 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
 teleport[8041]:         /go/src/github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:245 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
 teleport[8041]:         /opt/go/src/runtime/asm_arm64.s:1263 runtime.goexit
 teleport[8041]: User Message: failed to dial: all auth methods failed] cluster:teleport.my.domain reversetunnel/agentpool.go:263

Answered by aug70

Aug 4, 2022

I got this working finally. Here's my solution, it might need some adjustment so keep that in mind, but hope it helps someone save time.
1- I used Cloudflare Proxy but I set the SSL/TLS Qverview to use Full. Otherwise it always gives ERR_SSL_VERSION_OR_CIPHER_MISMATCH error.
2- I stopped using Nginx Proxy Manager. If you can get it to work with Nginx proxy manager, let me know.
3- I had to open ports 3023, 3024, 3080 on my router. 3023 for SSH, 3024 for tunnel, 3080 for web.
4- Configured ufw to accept traffic from those ports.
5- Below is my config and systemd service file.

---
version: v2
teleport:
  nodename: {{ ansible_facts['fqdn'] }}
  data_dir: /var/lib/teleport
  log:
    output: …

View full answer

webvictim · 2022-07-26T14:06:28Z

webvictim
Jul 26, 2022
Collaborator

This error actually looks like you have an agent attempting to connect back to the Teleport cluster to provide the app, rather than running all in one process as per the config file you posted.

If you really are using that exact config file and seeing this error, I'd be tempted to shut down Teleport, rm -rf /var/lib/teleport (this will delete all your users/roles etc so back those up first if you need them) and then start Teleport from scratch. It's possible that there's some cache left over from previous configurations which is putting you in a bad state.

0 replies

aug70 · 2022-08-04T04:01:23Z

aug70
Aug 4, 2022
Author

I got this working finally. Here's my solution, it might need some adjustment so keep that in mind, but hope it helps someone save time.
1- I used Cloudflare Proxy but I set the SSL/TLS Qverview to use Full. Otherwise it always gives ERR_SSL_VERSION_OR_CIPHER_MISMATCH error.
2- I stopped using Nginx Proxy Manager. If you can get it to work with Nginx proxy manager, let me know.
3- I had to open ports 3023, 3024, 3080 on my router. 3023 for SSH, 3024 for tunnel, 3080 for web.
4- Configured ufw to accept traffic from those ports.
5- Below is my config and systemd service file.

---
version: v2
teleport:
  nodename: {{ ansible_facts['fqdn'] }}
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  cluster_name: teleport.my.cloud
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: yes
  web_listen_addr: 0.0.0.0:3080
  listen_addr: 0.0.0.0:3023
  public_addr: teleport.my.cloud:2083 # 443 is taken that's why I'm using 2083, you can use 443
  tunnel_listen_addr: 0.0.0.0:3024
  https_cert_file: /etc/letsencrypt/webproxy_cert.pem
  https_key_file: /etc/letsencrypt/webproxy_key.pem
app_service:
  enabled: "yes"
  debug_app: true
  apps:
  - name: blah
    uri: http://blah
    insecure_skip_verify: true
    public_addr: blah.teleport.my.cloud

Not much to see here, but I played with the proxy env vars, I couldn't get it to work. See https://goteleport.com/docs/setup/reference/networking/#http-connect-proxies
maybe I missed something but for now I'm not using any environment variables.

[Unit]
Description=Teleport SSH Service
After=network.target

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport start --config=/etc/teleport.yaml --pid-file=/run/teleport.pid
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport.pid

[Install]
WantedBy=multi-user.target

Notes:

There are so many resources on Teleport out there but there so many junk too.
Always check teleport version in any reading material you find, my version is Teleport v10.0.2 git:v10.0.2-0-g47e0914fb go1.18.3
I'm not a security expert if you think I made a mistake please point it out.

Thank you!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't get applications to respond #14862

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Can't get applications to respond #14862

Uh oh!

aug70 Jul 25, 2022

Replies: 2 comments

Uh oh!

webvictim Jul 26, 2022 Collaborator

Uh oh!

Uh oh!

aug70 Aug 4, 2022 Author

aug70
Jul 25, 2022

webvictim
Jul 26, 2022
Collaborator

aug70
Aug 4, 2022
Author