mobile client and desktop app returns 404:Not Found with webauthn.

[RT-Tap]

Subject of the issue

Vault warden from mobile started to return 404: Not Found | Rocket when I try to login via the mobile client and "An error Occured: Verification code is needed" in the desktop app. No issues via browser extension. Admittedly I don't use vaultwarden through the mobile app or desktop app all that often so I can't pinpoint the exact day/time this issue started but sometime in the past 2 weeks.

Deployment environment

Your environment (Generated via diagnostics page)

• Vaultwarden version: v1.22.2
• Web-vault version: v2.21.1
• Running within Docker: true
• Environment settings overridden: true
• Uses a reverse proxy: true
• IP Header check: true (X-Forwarded-For)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Time Check: true
• Domain Configuration Check: true
• HTTPS Check: true
• Database type: MySQL
• Database version: 10.6.4-MariaDB-1:10.6.4+maria~focal
• Clients used:
• Reverse proxy and version:
• Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden: DOMAIN, SENDS_ALLOWED, SIGNUPS_ALLOWED, SIGNUPS_VERIFY, ADMIN_TOKEN, IP_HEADER, YUBICO_CLIENT_ID, YUBICO_SECRET_KEY, SMTP_HOST, SMTP_FROM, SMTP_USERNAME, SMTP_PASSWORD

{
  "_duo_akey": null,
  "_enable_duo": false,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_ip_header_enabled": true,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_max_conns": 10,
  "database_url": "*****://****************:****************@***.**.*.*:****/*********",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://*********.************.***",
  "domain_origin": "*****://*********.************.***",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "enable_db_wal": true,
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Forwarded-For",
  "job_poll_interval_ms": 30000,
  "log_file": null,
  "log_level": "Info",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "org_attachment_limit": null,
  "org_creation_users": "",
  "password_iterations": 100000,
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": false,
  "signups_domains_whitelist": "************.***",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_explicit_tls": false,
  "smtp_from": "*****@************.***",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": "**-**.*******.***",
  "smtp_password": "***",
  "smtp_port": 587,
  "smtp_ssl": true,
  "smtp_timeout": 15,
  "smtp_username": "********************************",
  "templates_folder": "data/templates",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_syslog": false,
  "user_attachment_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": true,
  "websocket_port": 3012,
  "yubico_client_id": "67596",
  "yubico_secret_key": "***",
  "yubico_server": null
}

• vaultwarden version: 1.22.2

• Install method: Docker compose image using mysql (mariadb) database in another container in same compose file

• Clients used: chrome/brave browser extension no issues, 404: Not Found | Rocket when I try to login via the mobile client and "An error Occured: Verification code is needed" in the desktop app

• Reverse proxy and version: apache 2.4

• MySQL/MariaDB or PostgreSQL version: MariaDB 10.6

• Other relevant details:

Steps to reproduce

You can use these files (albeit exchanging a few things like your SSL certs and MSQL data base and what not)
docker-compose.yml

version: "3.8"

services:
  reverse-proxy:
    container_name: reverseProxy
    build:
      context: ../http/revProxy/
      dockerfile: Dockerfile
    ports: 
      - "192.168.1.162:80:80"
      - "192.168.1.162:443:443"
    volumes:
      - "../http/revProxy/httpd.conf:/usr/local/apache2/conf/httpd.conf" 
      - "../http/revProxy/httpd-vhosts.conf:/usr/local/apache2/conf/extra/httpd-vhosts.conf:ro"
      - "../http/revProxy/SSLCerts/fullchain.pem:/usr/local/apache2/conf/server.crt:ro"
      - "../http/revProxy/SSLCerts/privkey.pem:/usr/local/apache2/conf/server.key:ro"
    restart: always
    networks:
      default:
        ipv4_address: 172.16.0.8


  mysql:
    container_name: personalServicesDB
    image: mariadb:latest
    environment:
      - MYSQL_ROOT_PASSWORD=password
    expose: 
      - "3306"
    volumes: 
      - "../databases/personalServices/mysql:/var/lib/mysql"
    restart: always
    networks:
      default:
        ipv4_address: 172.16.0.2

  bitwarden:
    container_name: Bitwarden
    image: vaultwarden/server:latest
    env_file:
      - bitwardenconfig.env
    expose:
      - "3012"
      - "80"
    volumes:
      - "../bitwarden/data:/data"
    restart: always
    networks:
      default:
        ipv4_address: 172.16.0.3
    depends_on:
      - reverse-proxy
      - mysql


networks:
    default:
      ipam:
        driver: default
        config:
          - subnet: '172.16.0.0/16' 

bitwardenconfig.env

DATABASE_URL=mysql://bitwarden_worker:CqX8ZEotZ3EidwP6@172.16.0.2:3306/bitwarden
IP_HEADER=X-Forwarded-For
WEBSOCKET_ENABLED=true
WEBSOCKET_ADDRESS=0.0.0.0
WEBSOCKET_PORT=3012
SENDS_ALLOWED=true
LOG_LEVEL=debug
SIGNUPS_ALLOWED=false
SIGNUPS_VERIFY=true
SIGNUPS_DOMAINS_WHITELIST=example.com
ADMIN_TOKEN=MyAdminToken
DOMAIN=https://bitwarden.example.com
YUBICO_CLIENT_ID=****
YUBICO_SECRET_KEY=*******

httpd-vhosts.conf

<VirtualHost *:80>
        ServerName bitwarden.example.com
        Redirect / https://bitwarden.example.com
</VirtualHost>
<VirtualHost *:443>
        ServerName bitwarden.example.com
        SSLEngine on

        SSLCertificateFile /usr/local/apache2/conf/server.crt
        SSLCertificateKeyFile /usr/local/apache2/conf/server.key

        Options -Indexes
        ServerSignature off

        ProxyPreserveHost On
        ProxyRequests off
        RewriteEngine On
        RequestHeader set X-Forwarded-For %{REMOTE_ADDR}s
        RemoteIPHeader X-Forwarded-For

        RewriteCond %{REMOTE_ADDR} !^192\.168\.1
        RewriteRule /admin(.*) - [F,L]

        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /notifications/hub(.*) ws://172.16.0.3:3012/$1 [P,L]
        ProxyPass / http://172.16.0.3:80/
        ProxyPassReverse / http://172.16.0.3:80/
</VirtualHost>

httpd.conf

ServerRoot "/usr/local/apache2"

Listen 80

LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule cache_module modules/mod_cache.so
LoadModule cache_disk_module modules/mod_cache_disk.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule mime_module modules/mod_mime.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule remoteip_module modules/mod_remoteip.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule http2_module modules/mod_http2.so
LoadModule proxy_http2_module modules/mod_proxy_http2.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfModule !mpm_prefork_module>
        #LoadModule cgid_module modules/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
        #LoadModule cgi_module modules/mod_cgi.so
</IfModule>
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so

<IfModule unixd_module>
User daemon
Group daemon

</IfModule>


ServerAdmin Admin@mydomain.com

ServerName mydomain.com

<Directory />
        Options -Indexes +FollowSymLinks
        AllowOverride None
        Require all denied
</Directory>


<Directory "/usr/local/apache2/htdocs">

    Options -Indexes +FollowSymLinks

    AllowOverride None

    Require all denied
    ServerSignature off
</Directory>


<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog /proc/self/fd/2

LogLevel warn

<IfModule log_config_module>

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog /proc/self/fd/1 common

</IfModule>

<IfModule alias_module>

    ScriptAlias /cgi-bin/ "/usr/local/apache2/cgi-bin/"

</IfModule>

<IfModule cgid_module>

</IfModule>

<Directory "/usr/local/apache2/cgi-bin">
    AllowOverride None
    Options None
    Require all denied
</Directory>

<IfModule headers_module>
    RequestHeader unset Proxy early
</IfModule>

<IfModule mime_module>

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

</IfModule>
Include conf/extra/httpd-vhosts.conf
<IfModule proxy_html_module>
Include conf/extra/proxy-html.conf
</IfModule>
Include conf/extra/httpd-ssl.conf
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

Expected behaviour

To login

Actual behaviour

Mobile app will not even allow me to enter my webauthn yubikey key it goes from password entry to half a second on the security key screen then straight to 404 error.
Desktop app has a Loading... where the picture of a security key should be not allowing me to enter the key.

Troubleshooting data

I made sure that rocket TLS is not on as per : #760
Also set LOG_LEVEL=debug to try and debug although I couldn't really find a cause.

For mobile login originally I had

172.58.239.20 - - [13/Oct/2021:21:19:38 +0000] "GET /webauthn-mobile-connector.html?data=eyJjYWxsYmFja1VyaSI6ImJpdHdhcmRlbjovL3dlYmF1dGhuLWNhbGxiYWNrIiwiZGF0YSI6IntcImFsbG93Q3JlZGVudGlhbHNcIjpbe1wiaWRcIjpcIjZ6TzJ3U1o3YTNDa3NpNnJfbDJZQzdMM2dxZmlYcDdiNHgzdi0xZXJKa2lOWWhxSFdRMjNGYm9DbW1CM3JGQURQc0FGbkltZURxN2I0ck9nVkxQdXpBXCIsXCJ0eXBlXCI6XCJwdWJsaWMta2V5XCJ9XSxcImNoYWxsZW5nZVwiOlwicGkyNGItQUtqQnJsUWRleDhnNXVldlRCSjVIaW1RTDRPN29ZT3lGiZXh0ZW5zaW9uc1wiOntcImFwcGlkXCI6XCJodHRwczovL2JpdHdhcmRlbi5hcnRodXJ0YXBwZXIuZGV2L2FwcC1pZC5qc29uXCIsXCJnZXRDcmVkQmxvYlwiOmZhbHNlfSxcInJwSWRcIjpcImJpdHdhcmRlbi5hcnRodXJ0YXBwZXIuZGV2XCIsXCJ0aW1lb3V0jAwMDAsXCJ1c2VyVmVyaWZpY2F0aW9uXCI6XCJkaXNjb3VyYWdlZFwifSIsImJ0blRleHQiOiJBdXRoZW50aWNhdGUgV2ViQXV0aG4ifQ==&parent=bitwarden%3A%2F%2Fwebauthn-callback&v=2 HTTP/1.1
[2021-10-13 23:21:44.994][response][INFO] POST /identity/connect/token (login) => 400 Bad Request

^^ removed random sections of token in order to keep privacy

But then I pulled latest image and now I get

[2021-10-14 00:23:16.378][error][ERROR] 2FA token not provided
[2021-10-14 00:23:16.378][response][INFO] POST /identity/connect/token (login) => 400 Bad Request

For the desktop app it won't let me enter a yubikey as if it's waiting for as response from vaultwarden. If I go ahead anyway and press continue I obviously get:

[2021-10-13 23:47:40.708][request][INFO] POST /api/accounts/prelogin
[2021-10-13 23:47:40.710][response][INFO] POST /api/accounts/prelogin (prelogin) => 200 OK
[2021-10-13 23:47:40.774][request][INFO] POST /identity/connect/token
[2021-10-13 23:47:41.034][error][ERROR] 2FA token not provided
[2021-10-13 23:47:41.035][response][INFO] POST /identity/connect/token (login) => 400 Bad Request
[2021-10-14 00:20:37.895][request][INFO] POST /api/accounts/prelogin
[2021-10-14 00:20:37.897][response][INFO] POST /api/accounts/prelogin (prelogin) => 200 OK
[2021-10-14 00:20:38.659][request][INFO] POST /identity/connect/token
[2021-10-14 00:20:38.876][error][ERROR] 2FA token not provided

If I change the apps to use the bitwarden servers I get no issues.

It was up and working up until very recently and so I thought maybe vaultwarden updated something and changed their API or something but it hasnt been reported by anyone else so not sure what's going on.

EDIT:
During the webauthn login I can select use another fasctor and using yubikey rather than webauthn works on desktop app however I don't get that option on mobile app
EDIT2: Yubiokey works on mobile as well so it's only webauthn for mobile and desktop while browser extension works fine.

[BlackDex]

Thanks for the very detailed report.
It looks like you are using the latest tagged image.
I think this is fixed in the testing tagged image of Vaultwarden.

Could you try that version and see if it works.

[ApolloDS]

I can confirm that it works now with the testing image.
On a iPhone with the mobile client and WebAuthn I didn't got the 404 anymore.

Thank you

👍