Direct Logout with Error 422

[smanthe]

Hi,

I'm already using this Repo for a longer time and I'm pretty happy with it. So first of all thanks for the great work to the maintainers.

Now I have updated my GitLab instance to 15.0.2 and moved my domain which is working with a letsencrypt https certificate. Since then I most of the time I get logged out from GitLab and receive an error 422. Sometimes I'm also able to klick onto another page. But then I see in the developer tools already that some ressources like icons can't be loaded due to an error 401. If have googled and debugged this error now for a longer time but found nothing that really helped.

In front of my GitLab Instance I have setup an NGINX reverse proxy which worked also pretty good for a long time. It is in the same network as the GitLab-Docker-Containers and its current configuration looks like this right now:

upstream gitlab_upstream {
    server 172.21.3.1:80 fail_timeout=0;
}

server {
    listen 80;
    server_name gitlab.xyz.de;

    location / {
      return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name gitlab.xyz.de;

    server_tokens off;

    ssl_certificate /etc/letsencrypt/lego/xyz.de.crt;
    ssl_certificate_key /etc/letsencrypt/lego/xyz.de.key;

    ssl_buffer_size 8k;

    ssl_dhparam /etc/ssl/certs/dhparam-2048.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;

    ssl_ecdh_curve secp384r1;
    ssl_session_tickets off;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    # resolver 8.8.8.8;

    # proxy_buffering off;
    client_max_body_size 10G;

    location / {
        gzip off;
        include /etc/nginx/proxy_params
        proxy_set_header    X-Forwarded-Ssl     on;
        proxy_pass http://gitlab_upstream;
        # proxy_set_header Host $http_host;
        # proxy_set_header X-Real-IP $remote_addr;
        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # proxy_set_header X-Forwarded-Proto $scheme;
        # proxy_set_header X-Frame-Options   SAMEORIGIN;

        #security headers
        # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
        # add_header X-Content-Type-Options "nosniff" always;


        # add_header Referrer-Policy "strict-origin-when-cross-origin" always;


        # Hide server information
        # proxy_hide_header X-Powered-By;
    }
}

My docker-compose config looks like this:

version: '2'

networks:
    nginx_proxy-tier:
        external: true

services:
  redis:
    restart: 'no'
    image: redis:6.2.6
    command:
    - --loglevel warning
    volumes:
    - ./volumes/redis:/var/lib/redis:Z
    networks:
        nginx_proxy-tier:
            ipv4_address: 172.21.3.2

  postgresql:
    restart: 'no'
    image: sameersbn/postgresql:12-20200524
    volumes:
    - ./volumes/postgresql:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm,btree_gist
    networks:
        nginx_proxy-tier:
            ipv4_address: 172.21.3.3
  gitlab:
    restart: 'no'
    image: 'sameersbn/gitlab:15.0.2'
    #command: /bin/bash
    depends_on:
    - redis
    - postgresql
    ports:
    - "10080:80"
    - "10022:22"
    volumes:
    - ./volumes/gitlab:/home/git/data:Z

    networks:
        nginx_proxy-tier:
            ipv4_address: 172.21.3.1

    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Europe/Berlin
    - GITLAB_TIMEZONE=Berlin

    - GITLAB_HTTPS=true
    - SSL_SELF_SIGNED=true
    - GITLAB_HOST=gitlab.xyz.de
    - GITLAB_PORT=10080
    - GITLAB_SSH_PORT=10022
    - GITLAB_SSH_HOST=gitlab.xyz.de
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE=long-and-random-alphanumeric-string
    - GITLAB_SECRETS_SECRET_KEY_BASE=long-and-random-alphanumeric-string
    - GITLAB_SECRETS_OTP_KEY_BASE=long-and-random-alphanumeric-string

    - GITLAB_ROOT_PASSWORD=
    - GITLAB_ROOT_EMAIL=

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=
    - GITLAB_EMAIL_REPLY_TO=
    - GITLAB_INCOMING_EMAIL_ADDRESS=

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

    - SMTP_ENABLED=true
    - SMTP_DOMAIN=www.web.de
    - SMTP_HOST=smtp.web.de
    - SMTP_PORT=587
    - SMTP_USER=
    - SMTP_PASS=
    - SMTP_STARTTLS=true
    - SMTP_TLS=false
    - SMTP_AUTHENTICATION=login


    - IMAP_ENABLED=false
    - IMAP_HOST=imap.gmail.com
    - IMAP_PORT=993
    - IMAP_USER=mailer@example.com
    - IMAP_PASS=password
    - IMAP_SSL=true
    - IMAP_STARTTLS=false

    - OAUTH_ENABLED=false
    - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    - OAUTH_ALLOW_SSO=
    - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    - OAUTH_AUTO_LINK_LDAP_USER=false
    - OAUTH_AUTO_LINK_SAML_USER=false
    - OAUTH_EXTERNAL_PROVIDERS=

    - OAUTH_CAS3_LABEL=cas3
    - OAUTH_CAS3_SERVER=
    - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    - OAUTH_CAS3_LOGIN_URL=/cas/login
    - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    - OAUTH_GOOGLE_API_KEY=
    - OAUTH_GOOGLE_APP_SECRET=
    - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    - OAUTH_FACEBOOK_API_KEY=
    - OAUTH_FACEBOOK_APP_SECRET=

    - OAUTH_TWITTER_API_KEY=
    - OAUTH_TWITTER_APP_SECRET=

    - OAUTH_GITHUB_API_KEY=
    - OAUTH_GITHUB_APP_SECRET=
    - OAUTH_GITHUB_URL=
    - OAUTH_GITHUB_VERIFY_SSL=

    - OAUTH_GITLAB_API_KEY=
    - OAUTH_GITLAB_APP_SECRET=

    - OAUTH_BITBUCKET_API_KEY=
    - OAUTH_BITBUCKET_APP_SECRET=

    - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    - OAUTH_SAML_IDP_SSO_TARGET_URL=
    - OAUTH_SAML_ISSUER=
    - OAUTH_SAML_LABEL="Our SAML Provider"
    - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    - OAUTH_SAML_GROUPS_ATTRIBUTE=
    - OAUTH_SAML_EXTERNAL_GROUPS=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_USERNAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    - OAUTH_CROWD_SERVER_URL=
    - OAUTH_CROWD_APP_NAME=
    - OAUTH_CROWD_APP_PASSWORD=

    - OAUTH_AUTH0_CLIENT_ID=
    - OAUTH_AUTH0_CLIENT_SECRET=
    - OAUTH_AUTH0_DOMAIN=

    - OAUTH_AZURE_API_KEY=
    - OAUTH_AZURE_API_SECRET=
    - OAUTH_AZURE_TENANT_ID=

In the production.log I can find the following errors:

Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Can't verify CSRF token authenticity.
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 221)
Can't verify CSRF token authenticity.
Completed 200 OK in 29ms (Views: 0.2ms | ActiveRecord: 2.6ms | Elasticsearch: 0.0ms | Allocations: 12801)
Started GET "/uploads/-/system/project/avatar/34/docker-icon.png" for 95.89.243.88 at 2022-06-12 19:43:19 +0200
Processing by UploadsController#show as HTML
  Parameters: {"model"=>"project", "mounted_as"=>"avatar", "id"=>"34", "filename"=>"docker-icon.png"}
Completed 401 Unauthorized in 6ms (ActiveRecord: 1.0ms | Elasticsearch: 0.0ms | Allocations: 2362)

Thanks in advance for your help.

Kind regards,
Stephan

[smanthe]

I'm still having this problem. I tried a lot of things. Sometimes GitLab works. But most of the time I get logged out when clicking somewhere or I see the 422 error page.

[fidoedidoe]

I have a fairly similar setup, but no issues like you describe.

I'm running GitLab 15.0.1 (docker container), REDIS Offical docker image (version 7.0) & PostgreSQL Office docker image 13.7. My GitLab instance sits behind an apache 2.4.x reverse proxy where I terminate letsencrypt SSL cert (requests are sent using HTTP to gitlab and there are no certs applied inside the GitLab docker container). I know in the past I had issues with 500 errors (the mitigating strategy being to set the GitLab docker env var "RACK_ATTACK_ENABLED" to "false").

To troubleshoot I'd suggest "go back to basics". for example, bypass your nginx reverse proxy, to better understand whether the issue persists or not, and build up your understanding of the issue and where it might be originating from. Clearly, something is wrong, so try to strip back as many layers as possible.

[smanthe]

Thanks for your reply. This is probably the only thing I can do right now. I will setup a fresh installation of the docker-compose file from this repo and see if this is working. Luckily this is easy with docker.

All other googleing and testing did not help so far.

[fidoedidoe]

np. I hope you find a resolution quickly.

As you probably know, to bypass your nginx reserver proxy you should just be able to point your browser to the defined "proxypass" url, from your published config this would be: http://gitlab_upstream

One thing I do find interesting is your proxy pass is forwarding on http://gitlab_upstream (I'm assuming this url is contrived for this forum post). That said, if the target is http:// (not https), then I'm wondering whether the following env vars are needed (as I don't see any hint of you using self-signed certs within your gitlab docker-compose, perhaps I'm overlooking something though)

- GITLAB_HTTPS=true
- SSL_SELF_SIGNED=true

As I mentioned above, rightly / wrongly, traffic between my apache reverse proxy and GitLab service is just http (inbound SSL is terminated on the reverse proxy). As of now, I do not have either of the above set in my config.

[smanthe]

Hi @fidoedidoe,

sorry for my late answer. It works now. I have moved the containers to another computer with a newer host system. I think the old system was simply to old with Ubuntu 16.04. I didn't want to update it for some reasons ;) On a newer system with Ubuntu 20.04 my configuration worked more or less directly.

Thanks a lot again for your help!