Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Basic Auth - Critical worker timeout error #61

Closed
iFargle opened this issue Mar 25, 2023 · 18 comments
Closed

Basic Auth - Critical worker timeout error #61

iFargle opened this issue Mar 25, 2023 · 18 comments

Comments

@iFargle
Copy link
Owner

iFargle commented Mar 25, 2023 •
edited
Loading 

[2023-03-25 12:03:28 +0700] [1] [INFO] Starting gunicorn 20.1.0
[2023-03-25 12:03:28 +0700] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2023-03-25 12:03:28 +0700] [1] [INFO] Using worker: sync
[2023-03-25 12:03:28 +0700] [8] [INFO] Booting worker with pid: 8
[2023-03-25 12:03:30,351] INFO in server: Headscale-WebUI Version:  v0.5.6 / main
[2023-03-25 12:03:30,351] INFO in server: LOG LEVEL SET TO DEBUG
[2023-03-25 12:03:30,351] INFO in server: DEBUG STATE:  True
[2023-03-25 12:03:30,352] INFO in server: Loading basic auth libraries and configuring app...
[2023-03-25 12:06:38 +0700] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
[2023-03-25 12:06:38 +0700] [8] [INFO] Worker exiting (pid: 8)
[2023-03-25 12:06:38 +0700] [9] [INFO] Booting worker with pid: 9
[2023-03-25 12:06:39,526] INFO in server: Headscale-WebUI Version:  v0.5.6 / main
[2023-03-25 12:06:39,526] INFO in server: LOG LEVEL SET TO DEBUG
[2023-03-25 12:06:39,527] INFO in server: DEBUG STATE:  True
[2023-03-25 12:06:39,527] INFO in server: Loading basic auth libraries and configuring app...

Just tried today and keep getting bad gateway when trying to access the webui. Everytime I tried to access the URL, I always got worker exiting in the log.

Any idea what's wrong?

Originally posted by @inlophe in #60 (comment)
@iFargle iFargle mentioned this issue Mar 25, 2023
Closed
@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

First question:
Are you using Podman or Docker?
Can you paste your docker-compose.yml?

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

Also: What is the full path of the URL you are trying to access?
SCRIPT_NAME is the base path for your application, so if you're hosting on http://localhost/ and your SCRIPT_NAME is set to /admin (which I think I have the docker-compose.yml as /admin), you would need to access it from http://localhost/admin

@iFargle iFargle changed the title `Basic Auth - Critical worker timeout error Basic Auth - Critical worker timeout error Mar 25, 2023
@inlophe
Copy link

inlophe commented Mar 25, 2023 •
edited
Loading

I'm using docker.
This is my compose

version: '3.9'

services:
  headscale:
    image: headscale/headscale:latest
    pull_policy: always
    container_name: headscale
    restart: unless-stopped
    command: headscale serve
    networks:
      - proxy
    volumes:
      - /opt/container/headscale/config:/etc/headscale
      - /opt/container/headscale/data:/var/lib/headscale
    labels:
      - traefik.enable=true
      - traefik.http.routers.headscale.rule=PathPrefix(`/`) && Host(`[mydomain]`) # you might want to add: && Host(`your.domain.name`)"
      - traefik.http.services.headscale.loadbalancer.server.port=8080

  headscale-webui:
    image: ghcr.io/ifargle/headscale-webui:latest
    container_name: headscale-webui
    environment:
      - TZ=Asia/Jakarta
      - COLOR=red                              # Use the base colors (ie, no darken-3, etc) - 
      - HS_SERVER=https://[mydomain]    # Reachable endpoint for your Headscale server
      - DOMAIN_NAME=https://[mydomain]  # The base domain name for this container.
      - SCRIPT_NAME=/admin                     # This is your applications base path (wsgi requires the name "SCRIPT_NAME")
      - KEY="[xxxsecretxxx]"             # Generate with "openssl rand -base64 32" - used to encrypt your key on disk.
      - AUTH_TYPE=basic                         # AUTH_TYPE is either Basic or OIDC.  Empty for no authentication
      - LOG_LEVEL=DEBUG                         # Log level.  "DEBUG", "ERROR", "WARNING", or "INFO".  Default "INFO"
      # ENV for Basic Auth (Used only if AUTH_TYPE is "Basic").  Can be omitted if you aren't using Basic Auth
      - BASIC_AUTH_USER=[someuser]                   # Used for basic auth
      - BASIC_AUTH_PASS=[somepassword]                   # Used for basic auth
      # ENV for OIDC (Used only if AUTH_TYPE is "OIDC").  Can be omitted if you aren't using OIDC
      #- OIDC_AUTH_URL=https://auth.$DOMAIN/.well-known/openid-configuration # URL for your OIDC issuer's well-known endpoint
      #- OIDC_CLIENT_ID=headscale-webui         # Your OIDC Issuer's Client ID for Headscale-WebUI
      #- OIDC_CLIENT_SECRET=YourSecretHere      # Your OIDC Issuer's Secret Key for Headscale-WebUI
    volumes:
      - /opt/container/headscale/ui/data:/data                         # Headscale-WebUI's storage.  Make sure ./volume is readable by UID 1000 (chown 1000:1000 ./volume)
      - /opt/container/headscale/config:/etc/headscale:ro # Headscale's config storage location.  Used to read your Headscale config.
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.headscale-webui.entrypoints=https
      - traefik.http.routers.headscale-webui.rule=Host(`[mydomain]`) && (PathPrefix(`/admin/`) || PathPrefix(`/admin`))
      - traefik.http.services.headscale-webui.loadbalancer.server.port=5000
      
networks:
  proxy:
    external: true

I'm using subdomain the same subdomain for headscale and webui with the difference adding /admin after the subdomain (https://net.[mydomain]/admin

EDIT: When I access the URL, it prompted me for the basic auth, after entering the correct credentials it just keep loading for a minute or two until it gives me bad gateway

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

Are you trying to load /overview?

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

no scratch that... Hmmm... I can't see anything wrong with this config. Let me do some tests.

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

can you run with no auth and see if it still times out?

@inlophe
Copy link

inlophe commented Mar 25, 2023

Ok, I'll try it

@inlophe
Copy link

inlophe commented Mar 25, 2023

Before that, one quick question.
I need to mount headscale config folder to webui container right?
Then the permission inside the webui container will be like this:

~ $ ls -l /etc/headscale/
total 80
-rw-r--r--    1 app      app          11501 Mar 25 09:54 config.yaml
-rw-r--r--    1 root     root         57344 Mar 25 10:53 db.sqlite
-rw-------    1 root     root            72 Mar 25 10:29 noise_private.key
-rw-------    1 root     root            72 Mar 25 10:29 private.key

db and key owned by root. Is this correct?

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

yeah that looks fine.
headscale-webui only accesses the config.yaml, the other files don't matter. And your permissions look good -- just need read only

@inlophe
Copy link

inlophe commented Mar 25, 2023

Removing the auth doesn't work. Still getting WORKER timeout in log, and bad gateway on the browser

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

Give ghcr.io/ifargle/headscale-webui:testing a shot. I've made a few changes, though I'm unsure if I've made changes that would impact this error. Still worth a shot though.

It's under heavy testing so if you pull now and it works don't pull again until v0.6.0 is pushed to main, it may break lol

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

oh also worth asking: ARM or x86? Testing is only available as x86

@inlophe
Copy link

inlophe commented Mar 25, 2023 •
edited
Loading

Mine's x86, Debian 11

EDIT: still not working with testing. I'll try to spin up fresh server to test further

@inlophe
Copy link

inlophe commented Mar 25, 2023

Welp, turns out my nftables config is the one causing the issue. I'll try to look deeper into what's causing it.

@inlophe
Copy link

inlophe commented Mar 25, 2023 •
edited
Loading

Weird, my nftables input chain default to drop and that chain is the one causing it. Isn't docker container suppose to use it's own docker chain? (at least that's how it is for me until now)

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023

That's interesting.... Glad you figured it out. The application shouldn't be doing anything funky with Docker at all -- At least as far as I know.

Let me know if you have any more issues. Thanks!

@iFargle iFargle closed this as completed Mar 25, 2023
@inlophe
Copy link

inlophe commented Mar 25, 2023

I think I found the problem.

Mar 25 10:24:04 auxiliary kernel: [8744276.793412] Dropped: IN=br-87c38579dda4 OUT= PHYSIN=vetha087dad MAC=02:42:f9:92:64:15:02:42:ac:12:00:03:08:00 SRC=172.18.0.3 DST=[public IP] LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8897 DF PROTO=TCP SPT=48850 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0

Found this after trying to log the dropped packet of the firewall. I think this is webui trying to connect to headscale (judging by the destination port - 443) using domain/public ip? That's why the firewall drop it, because it is not using docker internal network (I think) and coming in to the bridge interface (IN=br-87c38579dda4) instead of the host network interface (the manual nftables rules only allow connection to certain listed port from the host network interface).

Is there particular reason why the webui does not probe the headscale using the internal docker network (using the usual service name and port)?

@iFargle
Copy link
Owner Author

iFargle commented Mar 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@inlophe @iFargle