Suspected bug with ip_prefixes

[mhameed]

Hi,

Thanks for your work on headscale.

I would be grateful if someone could reproduce the the following, which seems like a bug to me:

headscale v0.17.0
tailscale v1.34.0
Ubuntu 22.04

Following headscale config file:

    ---
    noise:
      private_key_path: /var/lib/headscale/noise.key
    
    # The url clients will connect to.
    # Typically this will be a domain like:
    #
    # https://myheadscale.example.com:443
    #
    server_url: https://test1.ephemeral.host
    
    # Address to listen to / bind to on the server
    #
    listen_addr: 0.0.0.0:443
    
    # Address to listen to /metrics, you may want
    # to keep this endpoint private to your internal
    # network
    #
    metrics_listen_addr: 0.0.0.0:9090
    
    # Address to listen for gRPC.
    # gRPC is used for controlling a headscale server
    # remotely with the CLI
    # Note: Remote access _only_ works if you have
    # valid certificates.
    grpc_listen_addr: 0.0.0.0:50443
    
    # Allow the gRPC admin interface to run in INSECURE
    # mode. This is not recommended as the traffic will
    # be unencrypted. Only enable if you know what you
    # are doing.
    grpc_allow_insecure: false
    
    # Private key used encrypt the traffic between headscale
    # and Tailscale clients.
    # The private key file which will be
    # autogenerated if it's missing
    private_key_path: /var/lib/headscale/private.key
    
    # List of IP prefixes to allocate tailaddresses from.
    # Each prefix consists of either an IPv4 or IPv6 address,
    # and the associated prefix length, delimited by a slash.
    ip_prefixes:
      #- fd7a:115c:a1e0::/48
      - 172.19.0.0/23
      #- 100.64.0.0/16
    
    # DERP is a relay system that Tailscale uses when a direct
    # connection cannot be established.
    # https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp
    #
    # headscale needs a list of DERP servers that can be presented
    # to the clients.
    derp:
      server:
        # If enabled, runs the embedded DERP server and merges it into the rest of the DERP config
        # The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place
        enabled: true
    
        # Region ID to use for the embedded DERP server.
        # The local DERP prevails if the region ID collides with other region ID coming from
        # the regular DERP config.
        region_id: 999
    
        # Region code and name are displayed in the Tailscale UI to identify a DERP region
        region_code: "headscale"
        region_name: "Headscale Embedded DERP"
    
        # Listens in UDP at the configured address for STUN connections to help on NAT traversal.
        # When the embedded DERP server is enabled stun_listen_addr MUST be defined.
        #
        # For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/
        stun_listen_addr: "0.0.0.0:3478"
    
      # List of externally available DERP maps encoded in JSON
      urls:
        - https://controlplane.tailscale.com/derpmap/default
    
      # Locally available DERP map files encoded in YAML
      #
      # This option is mostly interesting for people hosting
      # their own DERP servers:
      # https://tailscale.com/kb/1118/custom-derp-servers/
      #
      # paths:
      #   - /etc/headscale/derp-example.yaml
      paths: []
    
      # If enabled, a worker will be set up to periodically
      # refresh the given sources and update the derpmap
      # will be set up.
      auto_update_enabled: true
    
      # How often should we check for DERP updates?
      update_frequency: 24h
    
    # Disables the automatic check for headscale updates on startup
    disable_check_updates: true
    
    # Time before an inactive ephemeral node is deleted?
    ephemeral_node_inactivity_timeout: 30m
    
    # SQLite config
    db_type: sqlite3
    db_path: /var/lib/headscale/db.sqlite
    
    ### TLS configuration
    #
    # Client (Tailscale/Browser) authentication mode (mTLS)
    # Acceptable values:
    # - disabled: client authentication disabled
    # - relaxed: client certificate is required but not verified
    # - enforced: client certificate is required and verified
    tls_client_auth_mode: disabled
    
    ## Use already defined certificates:
    tls_cert_path: "/var/lib/headscale/certs/test1.ephemeral.host.crt"
    tls_key_path: "/var/lib/headscale/certs/test1.ephemeral.host.key"
    
    log:
      # Output formatting for logs: text or json
      format: text
      level: trace
    
    # Path to a file containg ACL policies.
    # ACLs can be defined as YAML or HUJSON.
    # https://tailscale.com/kb/1018/acls/
    acl_policy_path: ""
    
    ## DNS
    #
    # headscale supports Tailscale's DNS configuration and MagicDNS.
    # Please have a look to their KB to better understand the concepts:
    #
    # - https://tailscale.com/kb/1054/dns/
    # - https://tailscale.com/kb/1081/magicdns/
    # - https://tailscale.com/blog/2021-09-private-dns-with-magicdns/
    #
    dns_config:
      # List of DNS servers to expose to clients.
      nameservers:
        - 1.1.1.1
    
      # Split DNS (see https://tailscale.com/kb/1054/dns/),
      # list of search domains and the DNS to query for each one.
      #
      # restricted_nameservers:
      #   foo.bar.com:
      #     - 1.1.1.1
      #   darp.headscale.net:
      #     - 1.1.1.1
      #     - 8.8.8.8
    
      # Search domains to inject.
      domains: []
    
      # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/).
      # Only works if there is at least a nameserver defined.
      magic_dns: false
    
      # Defines the base domain to create the hostnames for MagicDNS.
      #  must be a FQDNs, without the trailing dot.
      # The FQDN of the hosts will be
      #  (e.g., _myhost.mynamespace.example.com_).
      base_domain: ephemeral.host
    
    # Unix socket used for the CLI to connect without authentication
    # Note: for local development, you probably want to change this to:
    # unix_socket: ./headscale.sock
    unix_socket: /var/run/headscale/headscale.sock
    unix_socket_permission: "0770"
    
    # Logtail configuration
    # Logtail is Tailscales logging and auditing infrastructure, it allows the control panel
    # to instruct tailscale nodes to log their activity to a remote server.
    logtail:
      # Enable logtail for this headscales clients.
      # As there is currently no support for overriding the log server in headscale, this is
      # disabled by default. Enabling this will make your clients send logs to Tailscale Inc.
      enabled: false
    
    randomize_client_port: true

registering nodes works fine, and headscale node list shows the expected ip addresses for the registered machines.
The problem is when you do tailscale status or tailscale ping:

    $ headscale node list
    2022-12-06T08:45:49Z DBG Setting timeout timeout=5000
    2022-12-06T08:45:49Z DBG HEADSCALE_CLI_ADDRESS environment is not set, connecting to unix socket. socket=/var/run/headscale/headscale.sock
    2022-12-06T08:45:49Z TRC ../../../home/runner/work/headscale/headscale/cmd/headscale/cli/utils.go:133 > Connecting via gRPC address=/var/run/headscale/headscale.sock
    ID | Hostname | Name  | NodeKey | Namespace | IP addresses | Ephemeral | Last seen           | Online | Expired
    1  | test1    | test1 | [j6yvQ] | test      | 172.19.0.1,  | false     | 2022-12-06 08:45:11 | online | no
    2  | test2    | test2 | [+dxgH] | test      | 172.19.0.2,  | false     | 2022-12-06 08:45:32 | online | no
    $ tailscale status
    172.19.0.1      test1                test         linux   -
                    test2                test         linux   -
    $ tailscale ping test2
    node found but lacks an IP
    $

Similarly from test2:

    $ tailscale status
    172.19.0.2      test2                test         linux   -
                    test1                test         linux   -
    $ tailscale ping test1
    node found but lacks an IP
    $

If the ip_prefixes is set to 100.64.0.0/16 it all works fine.

Thanks.

[kradalby]

I think we might have actually implemented this without testing or having a purpose for this. It actually looks like it is not supported in the client https://github.com/search?q=repo%3Atailscale%2Ftailscale%20IsTailscaleIP&type=code.

I'll make a PR to indicate this in the docs.

[juanfont]

Please use supported ranges.

[xrd]

hi @juanfont Does this mean we cannot use two headscale networks? Is there a way to create two headscale networks and have a user connect to both and avoid conflicting IP addresses with other users on either network? I don't see a way to safely do that without having separate prefixes?

[xrd]

@juanfont I suppose this is the right way for now: https://tailscale.com/blog/fast-user-switching/

This works perfectly and is great for what I need!