Lost git over ssh access to GitLab after update

[mawis]

Describe the bug
After doing a general update (nixos-rebuild switch --upgrade) and running nix-collect-garbage -d followed by a system reboot, I lost access to my GitLab repositories. I cannot tell which of the above steps exactly caused the problem to start, but it worked before and didn't after.

At the beginning the problem I got was this:

$ git pull
The project you were looking for could not be found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I found the reason for this were entries in /var/gitlab/state/home/.ssh/authorized_keys' where the following path was used for ssh-agent`:

/nix/store/ccm7c2hmjqiah3y3k8xv56z9k58dh4i0-gitlab-shell-10.0.0/bin/gitlab-shell key-3

(Note the path directily into the nix-store.)

After removing the key and re-adding it (using the GitLab web interface) I saw that the following path is used now:

/run/current-system/sw/bin/gitlab-shell key-15

But still it didn't work, just the error message changed:

$ git pull
The project you were looking for could not be found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Following to that I changed the path on the other keys as well. Interestingly on the other keys I now get access to the repository. So I have one GitLab account with three ssh keys added. One of the ssh-keys produces the "The project you were looking for could not be found." error message while the other two keys started to work again after I updated the path to gitlab-shell.

The content of /var/gitlab/state/home/.ssh/authorized_keys is:

# Managed by gitlab-rails
command="/run/current-system/sw/bin/gitlab-shell key-3",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDL1NI9UmBTL8OoGfInxhvcRkTWXU+C9HJNy0rAfPpubOgtsvVWUMTHQY3epRBRT6Y68j49yEY0VJgzZ1R/7tZNnVh7uNX4IifD39DHNPQzkSUKgsLQc8UhkU4Vn7Tfqzz8oXrCCqWfbQhtliMFJvtKaZRfZN+TaOhOFo3A5xoG/gw6oC4626tzYIKWL9OyAihpNw55aWvAeiWNECY19FQQMwR78xwCuL0lmLwIYqt5WzK61/NGuZBfT7FPbFQMHCtGgeNFHsIuTXzzu6iO7HHkXITHUF3Jx3WsA9PF7Yzw51ZWDqDyAz+u1RQfQk1L8m1LBZYOQsCH20nqN6ceQ/V5
####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
command="/run/current-system/sw/bin/gitlab-shell key-5",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDxgOFmsFVEIX4oJgZYOtnBfXl6QkedUfdYZu6DPG8gdTwWjbnZeEwhbMf1ZP08aHPRrXGb25khHeZ2ZjYdFR8yEcx2UobI5aG9/S/uhpz/DrvFw7dqT3qT8roxd4yrzfQIpcEFbCN20i00vBmnIUoVx2NAFRtg4V5MrI3jCV2wy4msvYLvCJvntj49bxrXzI4NpqnJcSaOuogIOX4roZN0SVOZlNJKf1Ibk7kG84u/CcU6IN/BL0gqi4ZBfzkmtqKFo1DLO4c6lDfOJLzi780SrJd6J3HntrIoN9cqHReyymrYtieeM0jA+HwrBAaC0h58OAg3lRdzAAOJ4P67WZiZ
command="/run/current-system/sw/bin/gitlab-shell key-10",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBRnQyr2lliJ2y0NYIP9KAr5yKmt+VoQMBVSkqxyR3Ju
command="/run/current-system/sw/bin/gitlab-shell key-11",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOcMw5VjheX5cRMlrQ5rbmM4tSV+h+o7uiScDvdMAQX5
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
command="/run/current-system/sw/bin/gitlab-shell key-14",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDinClM7SYMUt/qil+dt34OFYYWf+ZyCk4XROKmf0KsyoTn7DCk7+xkhNiDc2O+n1GT9s5KBcsh/haJcKlQuqxnvjbQSpDgtjugORhR6GK47Q1RnBqEsnE2pP/KYyNk+iq8j3/QEphVHOvDy1ZD3sIRvSE01ly/kMdnYV2VnBNBcugjSObvQKICuOLubk9hClDFcNH50iI7Z/+1TgcOEAbbQ+Wk8WZgwL9ze3RO3G6j72ID5x4Dcv1+mt+hD0rGukoNsLyA+e9vpJ4jJLjpFiDHGTj2flYyttrS+cMkNR7IyzfzcVVVQVf1dB2yBopnTeVdptCx1wW8vSTY66FVtTI9
command="/run/current-system/sw/bin/gitlab-shell key-15",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDL1NI9UmBTL8OoGfInxhvcRkTWXU+C9HJNy0rAfPpubOgtsvVWUMTHQY3epRBRT6Y68j49yEY0VJgzZ1R/7tZNnVh7uNX4IifD39DHNPQzkSUKgsLQc8UhkU4Vn7Tfqzz8oXrCCqWfbQhtliMFJvtKaZRfZN+TaOhOFo3A5xoG/gw6oC4626tzYIKWL9OyAihpNw55aWvAeiWNECY19FQQMwR78xwCuL0lmLwIYqt5WzK61/NGuZBfT7FPbFQMHCtGgeNFHsIuTXzzu6iO7HHkXITHUF3Jx3WsA9PF7Yzw51ZWDqDyAz+u1RQfQk1L8m1LBZYOQsCH20nqN6ceQ/V5
command="/run/current-system/sw/bin/gitlab-shell key-16",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJsmQHXMn3jCaw1KH75Oi3TC/iYcPISTQdbgh5MH/WDUpiRUwDwSXKxVVQUZjFzXyMkPOUmUhj9yBNJr4Bp1t9HaadQod1LzWe1bxfVI7lCstZ06Ao9KA78lQ9qkwr3bPHcjnO+0p+Fe05Te5gJGVZhgGZGZcYk1ucoHZTT+wCMuqAnWT5nB0ueXz7fKnck6YrApdGBDJlNiJ98ZvfFhKyPZX3oI2gZ/tERyx5BZgMe8u66dJBnOzw+QK0T8VkwOa0hnkZahm6ENZ+bWtv94xO4xxx2/k8MsxnNFnYrS6efAO82EcowRyRxvgYn+5bnyox3JrCxzgZTTqwFTTRbXkx

Only key-15 isn't working.

To Reproduce
Sorry, I don't know. Probably the way GitLab on NixOS handled this changed and an older version of GitLab has to be installed first to generate configuration, that contains paths into the nix store. Then it seems to fail after you upgrade and get a system where the nix store location of gitlab-agent changes.

Expected behavior
I would expect, that I continue to be able to access my git repositories over ssh after an update of GitLab and NixOS.

Metadata

• system: "x86_64-linux"
• host os: Linux 4.19.87, NixOS, 19.09.1481.f3fa5a101eb (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

attribute:
- gitlab
- gitlab-shell
module:
- services.gitlab

[veprbl]

Must be due to backport of #74278
cc @flokli @petabyteboy @globin @fpletz @krav

[globin]

Try removing/commenting out the broken key and re-adding it from the interface, we had this broken state for a few keys too and that fixed it. It also is fixed on current nixpkgs branches

[flokli]

@veprbl why that PR specifically? f3eb063 and a33ddd7 patched gitlab-shell to write /run/current-system/sw/bin/gitlab-shell instead of just the hardcoded path to /var/gitlab/state/home/.ssh/authorized_keys (which was a bug previously) - but there's no code fixing up manually in that file (because it's hard to get right, and probably not a good idea too).

It's re-adding those keys via the web interface, or manually fixing in the file.

We might want to document this in the 19.09 release notes, though.

[mawis]

@globin I did that for all three keys. For two of them it helped, for the third one it didn't help. I just tried it once again and still this key is not working. Thanks anyway.

[mawis]

I found that the key that was not working was present in /var/gitlab/state/home/.ssh/authorized_keys twice. (I don't know why …) GitLab was always managing the last occurrence of the key in this file … while ssh uses the first match it found. Therefore a key number was passed to gitlab, that gitlab didn't know about anymore.
Removing the first occurrence of the duplicate key manually solved the problem for me.

[flokli]

I'm not sure if we can/should add code which tries to fixup halfway broken state - maybe it'd be a better idea to add some note to the changelog. @mawis, WDYT?

[mawis]

@flokli I agree that somehow broken authorized_keys (with the key being in there several times) should not be “fixed”. This could cause more harm than good (because you cannot know which is the correct entry). But I think the path to gitlab-shell should be fixed from hard coded to /run/current-system/sw/bin/gitlab-shell (as I read your message from Dec 6 as well).

[flokli]

@mawis shouldn't this already be the case in pkgs/applications/version-management/gitlab/remove-hardcoded-locations.patch:160? (both on master and 19.09)

[mawis]

@flokli Thanks for pointing this out. I must have missed, that this was fixed. Would have expected someone to have closed the ticket in that case already. I guess I should do so now :)