How troubleshoot SSHd on Docker ?

[ygbillet]

Hi,

I use GOGS as Docker appliance.
I just update to the last version.

I added my public key to my account.
Try to connect with ssh -vT git myhostname and get Permission denied (publickey,keyboard-interactive).

I check /home/git/.ssh/.authorized_keys and seems OK.
Is there a way to troubleshoot OpenSSH in GOGS running in Docker (where are the logs ?) ?

[unknwon]

Permission denied (publickey,keyboard-interactive)

Get into Docker and check SSH auth.log.

[unknwon]

cc @0rax

[ygbillet]

Did not find "auth.log" file in my container.

Am i missing something ?

[unknwon]

Check if /home/git/.ssh/.authorized_keys's permission is 600.

[0rax]

Can you try to connect to your serveur via ssh ?

ssh git@my-server.ext

It should output

Hi there, You've successfully authenticated, but Gogs does not provide shell access.
If this is unexpected, please log in with password and setup Gogs under another user.
Connection to my-server.ext closed.

[ygbillet]

permission is 600.

Cannot connect to ssh (Permission denied)

[ygbillet]

• Service SSH is up and running (get Permission denied and OK from ps)

• authorized_key's permission is 600 and public key is in file.

  1 root 0:00 /usr/bin/s6-svscan /app/gogs/docker/s6/
  13 root 0:00 s6-supervise gogs
  14 root 0:00 s6-supervise openssh
  15 root 0:00 /usr/sbin/sshd -D -f /app/gogs/docker/sshd_config
  16 git 0:00 /app/gogs/gogs web
  61 root 0:00 /bin/bash
  73 root 0:00 ps

Bun cannot find auth.log or any log from openssh.

[0rax]

To get access to sshd log, you will need to activate syslogd (not launched by default).

Just do a docker exec gogs-container-name syslogd, you will then be able to get sshd log inside /var/log/messages.

Currently working on a way to print sshd logs inside docker logs, that why syslogd was not launched by default. I should had it for now. my bad.

[unknwon]

@ygbillet can you dump your authorized_keys file here?

[ygbillet]

Actually with logging enabled it's better

Nov 4 17:57:52 d28c6f53a159 auth.info sshd[84]: Authentication refused: bad ownership or modes for directory /data/git

[0rax]

Quick fix would be chown -R git:git /data/git.

How did you launched the container itself, the command itself so I can try to reproduce this.

Will definitly add syslogd as a service for this purpose... While I try to find a way to output sshd logs to stdout !

[ygbillet]

Ok, so i change permissions for /data/git and it works. Not sure what caused the permissions errors.
I will try to get the error again

[ygbillet]

I can reproduce the error, here what i do

docker run --name=gogs-test-data --entrypoint /bin/true gogs/gogs
docker run -d --name=gogs-test --volumes-from gogs-test-data -p 10122:22 -p 10180:3000 gogs/gogs

[0rax]

That's weird, I cant reproduce on my own using a brand new instance of boot2docker.

[ygbillet]

Wait ! I may be skipped a step ! (i change my /.ssh/config for testing purpose ...)

I recreate an instance and try to get the same error again.
Sorry

[ygbillet]

Hmmm, get the same error again. Here what i do.

1. Create a proper config on my laptop (generate ssh key, add to config)
2. Create a data container for GOGS
3. Create container for GOGS

I will try with boot2docker and on another server.

[ygbillet]

By the way, i correct the permission problem with chmod g-w /data/git.

[0rax]

Can you do a ls -lah of the /data directory.

So I can see how it differs and unsure the correct rights to the folder during gogs service setup phase.

[ygbillet]

Hmmm.

I try on another server and get different result :/ (permissions are ok).

Need more time to investigate the differences.
I'll keep you posted.

[0rax]

Can you post the docker version of the servers you are testing this on with the output of `ls -lah /data` and tell me which one is triggering the error. Thanks a lot.

[ygbillet]

Server 1 (not working as expected)

Docker Client:

Version:      1.8.3
API version:  1.20
Go version:   go1.4.2
Git commit:   f4bf5c7
Built:        Mon Oct 12 05:33:35 UTC 2015
OS/Arch:      linux/amd64

Docker Server:

Version:      1.8.3
API version:  1.20
Go version:   go1.4.2
Git commit:   f4bf5c7
Built:        Mon Oct 12 05:33:35 UTC 2015
OS/Arch:      linux/amd64

output of ls -lah /data

drwxr-x---    5 git      git         4.0K Nov  4 18:36 .
drwxr-xr-x   40 root     root        4.0K Nov  4 18:36 ..
drwxrwx---    3 git      git         4.0K Nov  4 18:36 git
drwxrwx---    5 git      git         4.0K Nov  4 18:36 gogs
drwxrwx---    2 git      git         4.0K Nov  4 18:36 ssh

Server 2 (working as expected)

Docker Client:

Version:      1.8.1
API version:  1.20
Go version:   go1.4.2
Git commit:   d12ea79
Built:        Thu Aug 13 02:28:37 UTC 2015
OS/Arch:      linux/amd64

Docker Server:

Version:      1.8.1
API version:  1.20
Go version:   go1.4.2
Git commit:   d12ea79
Built:        Thu Aug 13 02:28:37 UTC 2015
OS/Arch:      linux/amd64

output of ls -lah /data

drwxr-xr-x    5 git      git         4.0K Nov  4 18:35 .
drwxr-xr-x   40 root     root        4.0K Nov  4 18:35 ..
drwxr-xr-x    3 git      git         4.0K Nov  4 18:35 git
drwxr-xr-x    5 git      git         4.0K Nov  4 18:35 gogs
drwxr-xr-x    2 git      git         4.0K Nov  4 18:35 ssh

[ygbillet]

I upgrade Server 2 to docker 1.9.0 and it's working as expected.

[0rax]

Ok, so that's a weird behaviour, currently writing a fix to ensurce proper rights on this directory.

[ygbillet]

I cannot update Server 1 before 2 weeks because it's running critical apps.
I make a reminder and keep you posted about the issue after an upgrade do Docker 1.9.0.

[0rax]

In the same time, I will try to upgrade one of mine to 1.8.3 and publish a fix for it ! Thanks for your information. Will keep you informed when the fix is ready.

[ygbillet]

Thanks for your help.
Great job for creating this docker image

[unknwon]

By merging #1898 to develop branch, this issue is claimed to be fixed, please help test again!

[ygbillet]

I tested the develop branch. Merge #1898 fix my issue. Thanks !

[unknwon]

Thanks your confirmation!