GitLab Integration Problem

[Spice-King]

Important Details

How are you running Sentry?

On-Premise w/ Docker, version 21.6.1

Description

I'm trying to connect a new self hosted instance of GitLab to my existing self hosted Sentry instance. Now, my boss has been making a push to kill off non-secure HTTP traffic on our network, so we have everything set to require it, since it's easy enough to install the root cert on new servers and PCs automatically. Step-ca handles generating cert via ACME, so that takes care of that side, and for bonus points, it's got name restrictions to limit the domains it can sign for, so no one can steal it and sign a fake google.com one.

The GitLab instance is an internal only internal service sitting at gitlab.company_name.internal. It resolves correctly on the Sentry host machine and certificates work, checked with both curl and openssl on the Sentry host, once the root CA is installed. Sentry is public facing service (though we might change that at some point, since we really just need my tunnel end point exposed) and has both a valid public domain and TLS certificate (error.company_name.com and *.company_name.com).

Steps to Reproduce

1. Attempt to add GitLab instance.
2. Get SSL/TLS errors.
3. Try to add the root CA to all the container.
4. Get a different error.
5. Cry, because the logs are not much help.

The only bit of logs that looked tied to it at the attempts:

web_1 | 20:34:17 [INFO] sentry.identity: identity.token-exchange-error (error='invalid_state' pipeline_state='9334b7b6a88d4d6eab80d5800dea4d49')
web_1 | 20:34:17 [ERROR] sentry.integration.gitlab: pipeline error (organization_id=1 provider='gitlab' error='An error occurred while validating your request.')

Docker Compose overrides, for injecting the cert.
https://gist.github.com/Spice-King/0275c8629dc7b6c2e615d6ceda1a699a

What you expected to happen

Be able to get a working GitLab connection, and leave work for the day with a bit of a smile hidden under my mask.

Mild joking aside, I'll need to do up an issue (or a PR if I get the time to) for make adding a root CA less of a pain. Injection via environment variable is probably the simplest, with a bit of script added to the entrypoints to update the global certs. Any pointers to some better logs or clues for figuring out what I missed? Java keystores or local openssl installs would be something I did not hunt for off the top of my head.

[getsentry-release]

Routing to @getsentry/open-source for triage. ⏲️

[BYK]

Heya, sorry for the trouble. Are you able to share some request logs so we can see the pipeline state parameter that gets passed and see how they differ? I suspect a type/encoding issue here but need to verify.

Did this work without SSL btw?

Mild joking aside, I'll need to do up an issue (or a PR if I get the time to) for make adding a root CA less of a pain. Injection via environment variable is probably the simplest, with a bit of script added to the entrypoints to update the global certs. Any pointers to some better logs or clues for figuring out what I missed? Java keystores or local openssl installs would be something I did not hunt for off the top of my head.

We try to refrain from using env variables for everything but I think some auto mount and detection in entrypoint scripts would be great!

[Spice-King]

Heya, sorry for the trouble. Are you able to share some request logs so we can see the pipeline state parameter that gets passed and see how they differ? I suspect a type/encoding issue here but need to verify.

Back at work with a fresh mind, found out I overlooked resetting snuba-api's command in my override file. Docker forgets commands if you monkey with the entrypoint and I forgot about that in compose. Since &sentry-defaults sets both, it worked there and I never noticed since I was not looking at the dash where snuba failing would be more noticeable.

There appears to be zero access attempts from my Sentry server to the proxy that's doing routing on my dev-ops server, that or Traefik is discarding malformed, empty or partial requests with zero logging that it did (doubtful, but the logging is not cranked up to the limit). Disabling all the bits that forced HTTPS took far, far, far longer than expected since there was many places where it did stuff like that. (Pro tip, every URL for something GitLab config for something belong to it's routes might trigger redirects, including your SAML config, 😢) Spent the better part of the morning digging around and waiting 3 minutes for GitLab to come back for each config change.

That the hard HTTPS requirement was removed and my overrides file, I could add GitLab to Sentry via HTTP, and we get SSL cert errors over HTTPS. Or it did. Now it errors after removed and tried to re-add. Again, no access logs from GitLab/Traefik, so here are logs in the form of docker-compose logs output.

https://gist.github.com/Spice-King/ddced1747a9384ecf13286cfe25bb959

We try to refrain from using env variables for everything but I think some auto mount and detection in entrypoint scripts would be great!
Fair, env vars came to mind as a first option. Probably could just assume a given directory exists (unlike a file, one can assume that because Docker will make it for you) and add any certs inside said directory to the container's global CA roots store.

Now for lunch, a brief escape from this headache.

[Spice-King]

Post lunch it now accepts the HTTP URL, HTTPS as well (verifying turned off, still counter to boss' orders). I wonder if something is not in a ready state for a while after start up. Would make the feeling of going crazy more accountable. Reapplying overrides and waiting 5 before retrying adding GitLab.

[BYK]

From your logs:

nginx_1 | 172.23.0.1 - - [24/Jun/2021:16:48:10 +0000] "GET /extensions/gitlab/setup/?completed_installation_guide HTTP/1.0" 200 12089 "https://error.company_name.com/organizations/company-name/integrations/gitlab/setup/?" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "10.0.0.156"

I think something is stripping some query string arguments from this request, causing the state argument being removed causing the error.