Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start GoCD server in Azure Container Instance because of config.git file lock #5631

Open
rohitramu opened this issue Dec 27, 2018 · 5 comments

Comments

@rohitramu
Copy link

commented Dec 27, 2018

Issue Type

Bug Report

Summary

Full log file is here: go-server.log

I'm attempting to create an Azure Container Instance using the gocd/gocd-server:v18.12.0 Docker image. When I tested it out for the first time, it was working great. I then wanted to adjust the setup so I wouldn't lose all the data if the container went down - I tried to create the container instance with a mounted Azure file share, but then the GoCD server failed to start up.

This is the Azure CLI command I'm using to create the container (I've replaced the environment-specific values with variable names):

az container create `
    --resource-group $resourceGroupName `
    --name 'gocd-server' `
    --image 'gocd/gocd-server:v18.12.0' `
    --dns-name-label $dnsLabel `
    --ports '8154' `
    --azure-file-volume-account-name $storageAccountName `
    --azure-file-volume-account-key $storageAccountAccessKey `
    --azure-file-volume-share-name $fileShareName `
    --azure-file-volume-mount-path '/godata'

This same command works with other Docker images (e.g. I'm running JFrog Artifactory in a container with an attached file share, and that seems to work fine). This is the first error I see in the log, plus the first two lines of the stack trace:

2018-12-27 07:34:51,363 ERROR [main] ContextLoader:350 - Context initialization failed
java.lang.RuntimeException: java.nio.file.FileSystemException: /go-working-dir/db/config.git/.git/HEAD.lock.567b8b3585b44cc69d5aa41c9851af10 -> db/config.git/.git/HEAD.lock: Not supported
	at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:166)
	at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:52)

Further down, I see this error as well:

2018-12-27 07:34:51,542 ERROR [main] GoServer:76 - ERROR: Failed to start GoCD server.
java.lang.RuntimeException: java.nio.file.FileSystemException: /go-working-dir/db/config.git/.git/HEAD.lock.567b8b3585b44cc69d5aa41c9851af10 -> db/config.git/.git/HEAD.lock: Not supported
	at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:166)
	at com.thoughtworks.go.server.initializers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:52)
Environment
Basic environment details
  • GoCD Docker Image Version: v18.12.0
Additional Environment Details
Steps to Reproduce
  1. In the Azure Portal (https://portal.azure.com), create:
    1. An account (a free trial should work fine)
    2. A Resource Group
    3. A storage account
    4. A file share within that storage account
  2. Run the following PowerShell script from CloudShell (click the button on the top bar of the UI that looks like >_ and select "PowerShell" with a new/different file share in the existing storage account). Remember to first replace the variables in the script to point to the resources you just created:
az container create `
    --resource-group $resourceGroupName `
    --name 'gocd-server' `
    --image 'gocd/gocd-server:v18.12.0' `
    --dns-name-label $dnsLabel `
    --ports '8154' `
    --azure-file-volume-account-name $storageAccountName `
    --azure-file-volume-account-key $storageAccountAccessKey `
    --azure-file-volume-share-name $fileShareName `
    --azure-file-volume-mount-path '/godata'

You can get the $storageAccountAccessKey by going to "Access Keys" under the "Settings" section in your storage account.
$dnsLabel can be any string that is unique in the Resource Group's region.
3. Browse to your resource group, then select the newly created container instance. In a new browser tab, go to the URL specified by the FQDN field, specifying the GoCD server port (e.g. https://my-dns-label.my-region.azurecontainer.io:8154).
4. You can get the logs and see the rest of the /godata folder by navigating to your storage account and clicking on Storage Explorer (Preview) ->File Shares -> yourFileShareName.

@ketan

This comment has been minimized.

Copy link
Member

commented Dec 27, 2018

@prateekbaheti

This comment has been minimized.

Copy link
Contributor

commented Jan 4, 2019

Looks like the cause of the issue is that creation of links (soft and hard) is not supported on the mounted Fileshare. When go server tries to initialize a git repository for config.xml using jgit it tries to create a link and that fails.

We would need to set the mfsymlinks option while mounting the volume. Unfortunately there does not seem to be an option to do this yet.

@prateekbaheti

This comment has been minimized.

Copy link
Contributor

commented Jan 14, 2019

@rohitramu GoCD uses jgit for managing the config.xml git repository. In gocd 18.11 the library was upgraded from 4.9.0.201710071750-r to 5.1.3
This introduced a fix for atomic lock file creation on NFS in this commit. This seems to have introduced the link creation.

A quick workaround would be to use the GoCD server version 18.10 while we find the right fix for the issue in this scenario.

@prateekbaheti

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

I have verified that If the git configuration core.supportsatomicfilecreation is set to true for the go user on the server then jgit does not try to create files atomically using links. It should be true by default but that does not seem to be the case. I have raised a bug on jgit for the same.

A workaround would be to create the .gitignore file when starting the container by overriding the entrypoint command using the --command-line option.

az container create \
    --resource-group $resourceGroupName \
    --name 'gocd-server' \
    --image 'gocd/gocd-server:v19.1.0' \
    --dns-name-label $dnsLabel \
    --command-line $'/bin/bash -c \'echo "[core]\n\n    supportsatomicfilecreation = true" > /home/go/.gitconfig && /docker-entrypoint.sh\'' \
    --ports '8154' \
    --azure-file-volume-account-name $storageAccountName \
    --azure-file-volume-account-key $storageAccountAccessKey \
    --azure-file-volume-share-name $fileShareName \
    --azure-file-volume-mount-path '/godata'
@birchb1024

This comment has been minimized.

Copy link

commented Apr 4, 2019

We see the same exact issue in Azure AKS using the GoCD Helm chart

This issue will also affect NFS and most likely the BCP Addon.

The cure is as above. Here's the Helm chart patch:

diff --git a/templates/configmap.yaml b/templates/configmap.yaml
index 22d2baa..6095747 100644
--- a/templates/configmap.yaml
+++ b/templates/configmap.yaml
@@ -12,6 +12,7 @@ data:
   preconfigure_server.sh: |-
     #!/bin/bash
 
+    printf "[core]\n\n    supportsatomicfilecreation = true" > /home/go/.gitconfig
     SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount
     KUBE_TOKEN=$(<${SERVICE_ACCOUNT_PATH}/token)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.