Gitea Docker in Windows shouldn't mount the ssh key directory to shared host directory, chmod doesn't work

[raidenlal]

• Gitea version (or commit ref): 1.14.0+dev-654-gbd28f1d64 (docker image gitea/gitea:latest)
• Git version: using gitea via web UI only
• Operating system: Docker for Windows, Windows 1809, Docker 2.2.0.5
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No (not applicable)
• Log gist:

2021-02-12T20:44:02.110591877Z Unable to load host key: /data/ssh/ssh_host_ecdsa_key
2021-02-12T20:44:02.111454288Z @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2021-02-12T20:44:02.111527089Z @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
2021-02-12T20:44:02.111535689Z @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2021-02-12T20:44:02.111538989Z Permissions 0755 for '/data/ssh/ssh_host_dsa_key' are too open.
2021-02-12T20:44:02.111542089Z It is required that your private key files are NOT accessible by others.
2021-02-12T20:44:02.111545089Z This private key will be ignored.
2021-02-12T20:44:02.111547989Z Unable to load host key "/data/ssh/ssh_host_dsa_key": bad permissions
2021-02-12T20:44:02.113117909Z Unable to load host key: /data/ssh/ssh_host_dsa_key

Description

I installed gitea on 2020-01-12 in docker for windows with a command docker run --name gitea -v C:/Docker/Gitea:/data:delegated --network=bridgenet -d gitea/gitea:latest
Since then I have noticed extraordinarily fast decline in SSD wear, which is monitored. On 2020-02-06 I have diagnosed culprit (gitea) and removed the container. Wear of the SSD stopped.
I now tried running gitea while storing its ssh keys somewhere in virtual machine of docker for windows - I don't care about those and obviously those will get generated anyway - docker run --name gitea -v C:/Docker/Gitea:/data:delegated -v /tmp/gitea/ssh:/data/ssh --network=bridgenet -d gitea/gitea:latest
It seems the extreme accessing of the host drive stopped.

Screenshots

Notice extreme lowering of wear level during running gitea:

Thousands of accesses to ssh files of gitea from docker:

[zeripath]

I suspect that the situation here is related to your docker install

The key should be being recreated like that constantly and it suggests that your docker is constantly being brought up and shut down.

Logs would have been helpful and probably would have shown that the docker was constantly been created and recreated.

[raidenlal]

I do not fully understand how my "docker is being constantly created and recreated". Also I should note that I have ~20+ containers residing in the same docker instance and none are misbehaving like this.

[raidenlal]

I've checked the logs and the issue is with permissions:

2021-02-12T20:44:02.110591877Z Unable to load host key: /data/ssh/ssh_host_ecdsa_key
2021-02-12T20:44:02.111454288Z @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2021-02-12T20:44:02.111527089Z @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
2021-02-12T20:44:02.111535689Z @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2021-02-12T20:44:02.111538989Z Permissions 0755 for '/data/ssh/ssh_host_dsa_key' are too open.
2021-02-12T20:44:02.111542089Z It is required that your private key files are NOT accessible by others.
2021-02-12T20:44:02.111545089Z This private key will be ignored.
2021-02-12T20:44:02.111547989Z Unable to load host key "/data/ssh/ssh_host_dsa_key": bad permissions
2021-02-12T20:44:02.113117909Z Unable to load host key: /data/ssh/ssh_host_dsa_key

Any ideas how to fix/workaround this? I checked and I always did have SSH disabled (as I didn't plan to use it) with
DISABLE_SSH = true
in settings.

Please check timestamps in the screenshot to see that this is happening during container running (not restarting). Recreation of the SSH keys happens hundreds times each second in some loop and it will eat up SSD.

[raidenlal]

Maybe the check should be altered to halt on exception regarding the wrong permission? Or continue and only log a warning?

Or could we get an option to set desired access rights? Maybe the 755 is exactly what I want.

Or maybe it just should not even touch the keys if I have disabled SSH in the settings, but this feels like hiding a problem.

[zeripath]

OK so the root cause of this problem is your docker drive mounting configuration.

OpenSSH enforces this access rule by immediately dying.

Docker/S6 will then, by default, just restart the dying service - and that is what your docker appears to be doing. Unfortunately our container is rather dumb and expects you to notice that it is repeatedly doing so - by paying attention to your logs.

We never expected that anyone would experience this - as OpenSSH's requirements for permissions for its files are fairly well known, and googling should have revealed the issue fairly rapidly.

So - if you really have no need for OpenSSH how do you disable it and prevent it starting?

Well, the Gitea setting DISABLE_SSH = true affects how Gitea interacts with openSSH (and if you were using the internal ssh server whether it would start that). OpenSSHd is a different service - and unfortunately as our docker is set-up there is no configuration provided to stop it from starting. Arguably because you may wish to use openSSH for other reasons on that container but most likely because having the service running but not being used has never caused a significant issue before and therefore no-one has provided a PR to stop it.

You will either have to edit the docker to stop OpenSSH from being started, i.e. remove /etc/s6/openssh from your dockers or adjust your docker drive configuration to fix this. PRs are welcome to provide configuration options to disable the OpenSSH server - however, when 1.14 is released there will be a rootless docker option available that does not use OpenSSH and in that case the DISABLE_SSH would disable the SSH service.

Whilst I am sorry that this docker instance appears to have damaged your SSD - you should have been watching your logs. Googling the logs would have revealed the problem immediately.

However, clearly there are some underlying issues here:

• We need some way of detecting that services are being constantly cycled.
• We need some way of disabling the openSSH service on the non-rootless docker.

Would you be able to tell me if it was docker that was cycling the service or was it S6 inside our container? If it was the external docker server then I'm not sure what we can do. If it was S6 inside our container then we at least have some chance to provide something to prevent this happening again.

[raidenlal]

Thanks for long and extensive answer. First of all I would like to clarify that the issue is not THE ssd, issue is that it could be potentially doing harm to someone else with similar setup. That SSD was an old one anyways and I have found workaround by having the keys inside the virtual machine by using "-v /tmp/gitea/ssh:/data/ssh", where they stay inside Linux and permissions are written correctly (I assume).

Root cause is my docker drive mounting configuration, that is true, but the issue here is that it can not be done any differently in Docker for Windows. Windows mangles the permissions requested by the guest linux OS. I don't have WSL2 available on that machine, which could fix that (?).

My docker is not restarting the container - container is running well. Inside the container some loop keeps checking and recreating the keys.

I think having container actually halt because of seemingly so significant security issue would be a better option. Docker would try to restart the container, but it would halt again. This cycling of container state is much easier to notice, even without having to change log level and look into logs.

I understand that the loop is trying to silently fix permissions issue for me, but it is failing to do so and actually it is leaving me with unsecured keys.

I am not sure how to help with diagnosing if it was the S6, as I am not a Linux guy. Gitea container was not being restarted, I was actually using it actively via the web UI without any issues.
Can you tell me what (and maybe how) to check? I can reconfigure the container to start failing again.

[zeripath]

Within your docker containers remove /etc/s6/openssh

[raidenlal]

rm -rf /etc/s6/openssh results in:
2021-02-16T15:01:26.327522583Z s6-supervise (child): fatal: unable to exec run: No such file or directory
2021-02-16T15:01:36.332349644Z s6-supervise openssh: warning: unable to spawn ./run - waiting 10 seconds
and repeating. Also the reading/recreating of keys stopped. Anything else I can try?

I reverted back to having ssh keys in tmp, because after next autoupdate new image will restore containers data.

[raidenlal]

Mention in webpack PR is completely random and this issue has not been actually touched.

This issue might be still creating enormous write loads on people's ssds, yet it was just dismissed.

[techknowlogick]

The webpack mention is unrelated spam, and this was closed due to it being upstream with opensshd. We have a rootless image now that can be used which doesn't use opensshd at all.

[wxiaoguang]

openssh requires that some files must have mode 0600, if your file system doesn't support file mode, then you might be stuck in the openssh key generation. I am not sure how docker volume works in windows, however it seems incorrect that you mount the ssh directory separately outside docker.

maybe the direction of this issue is to try to figure out how docker + gitea + openssh + windows volume work together, and check whether the file mode is correct.

update: it seems that docker + windows + volume can not do chmod: https://docs.docker.com/desktop/faqs/#can-i-change-permissions-on-shared-volumes-for-container-specific-deployment-requirements . the docker's answer is : "you need to either use non-host-mounted volumes or find a way to make the applications work with the default file permissions".

so, the conclusion is: you shouldn't mount the openssh private key directory to host directory in windows.

[raidenlal]

The webpack mention is unrelated spam, and this was closed due to it being upstream with opensshd. We have a rootless image now that can be used which doesn't use opensshd at all.

So now there is no need to separately mount ssh keys at all? I mean the
-v /tmp/gitea/ssh:/data/ssh
Part in my workaround.

[raidenlal]

so, the conclusion is: you shouldn't mount the openssh private key directory to host directory in windows.

I came to this conclusion in february. However as I dont know what I dont know I could not have known there are some ssh keys being stored in the given volume. Only thing that saved me was monitoring that was crying about falling wear level count.

[wxiaoguang]

I created a new issue for this problem:

• Prevent Gitea docker on Windows from running with OpenSSH private key directory mounted to a shared host directory (volume) #19395