/home/git/project.git/hooks/pre-receive: No such file or directory

[onbjerg]

Uh. I tried to push a new version to Deis. Mid-push my computer shut down, and when I returned I could not push again because a push was in progress.

I read in the issue tracker that you could a) delete the lock file or b) restart the builder. I didn't find any lockfile, so I restarted the builder.

Now it seems like nothing works.

I get this error when pushing

onbjerg at AIR in ~/Projects/wallz on develop
$ git push deis develop:master
/usr/local/bin/gitreceive: line 33: /home/git/wallz.git/hooks/pre-receive: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.```

[bacongobbler]

that looks like the repository in the builder could be corrupted somehow. If you enter into the builder again, what's the status of /home/git/wallz.git? As a quick workaround you can also just remove the repository entirely and re-push.

[bacongobbler]

basically it's coughing up on this line which originally came from the gitreceive project: https://github.com/deis/deis/blob/master/builder/rootfs/etc/confd/templates/gitreceive#L33

https://github.com/progrium/gitreceive

[davidcelis]

I'm having this issue with one of my repositories as well; it comes after performing an in-place upgrade of my Deis cluster from 1.4.x to 1.5.0:

% git push deis master
/usr/local/bin/gitreceive: line 33: /home/git/hubot-b9.git/hooks/pre-receive: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Destroying the repository and re-creating it didn't seem to work:

% deis destroy

 !    WARNING: Potentially Destructive Action
 !    This command will destroy the application: hubot-b9
 !    To proceed, type "hubot-b9" or re-run this command with --confirm=hubot-b9

> hubot-b9
Destroying hubot-b9... 
done in 8s
Git remote deis removed

% deis create hubot-b9
Creating application... done, created hubot-b9
Git remote deis added

% git push deis master
/usr/local/bin/gitreceive: line 33: /home/git/hubot-b9.git/hooks/pre-receive: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The other project I am running under Deis didn't have this problem. deis ps showed its processes all as destroyed, but I was able to push a new release and have it start up.

[carmstrong]

You can also try restarting builder - the repository cache does not persist through restart: deisctl restart builder.

[onbjerg]

@bacongobbler Everything is missing now.

core@deis-1 ~ $ docker exec -ti deis-builder /bin/bash -li
root@02f625b6ec74:/app# ls /tmp
root@02f625b6ec74:/app# ls /home/git
builder  check-repos  receiver  shim.dockerfile  wallz.git
root@02f625b6ec74:/app# cd /home/git/wallz.git
root@02f625b6ec74:/home/git/wallz.git# ls
build  cache

[onbjerg]

@carmstrong Restarted the builder. Nothing.

onbjerg at AIR in ~/Projects/wallz on master
$ git push deis master
/usr/local/bin/gitreceive: line 33: /home/git/wallz.git/hooks/pre-receive: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[onbjerg]

Tried re-creating. Same thing. Weird.

onbjerg at AIR in ~/Projects/wallz on master
$ deis destroy

 !    WARNING: Potentially Destructive Action
 !    This command will destroy the application: wallz
 !    To proceed, type "wallz" or re-run this command with --confirm=wallz

> wallz
Destroying wallz... 
done in 5s
Git remote deis removed

onbjerg at AIR in ~/Projects/wallz on master
$ deis create wallz
Creating application... done, created wallz
Git remote deis added

onbjerg at AIR in ~/Projects/wallz on master
$ git push deis master
/usr/local/bin/gitreceive: line 33: /home/git/wallz.git/hooks/pre-receive: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[bacongobbler]

that is certainly odd behaviour. Maybe something changed with the check-repos script?

[rvadim]

I have the same issue today morning. Can someone help with running command on builder?

[rvadim]

Recreate of project help.

[onbjerg]

I don't know what to do. The cluster is useless for this project at the minute. Can push other projects fine, though. This is the most important one, unfortunately.

[bacongobbler]

@onbjerg try the following:

fleetctl ssh deis-builder
nse deis-builder
rm -rf /home/git/wallz.git

after that, try pushing again. If that doesn't work then I'd try and see how we can reproduce this issue consistently (we haven't been seeing it over on our end)

[nathansamson]

@bacongobbler: your work around helped for me.

I noticed this issue today with (one) of my projects. I believe the issue MIGHT have been some general unstability earler today on my platform.

Deis builder was moved between hosts a few times during some period, and might not have completely started up properly...

[babarinde]

@bacongobbler ,
Hi, am having this same issue. as I can push to some apps and not to some others.
While trying rm -rf /home/git/wallz.git wallz.git was not found.
My deis-builder logs are these

Apr 17 09:28:22 core3.c.abbaandking.internal sh[22950]: 2015-04-17T09:28:22Z 16dbc64ae078 confd[418]: ERROR 401: The event in requested index is outdated and cleared (the requested history has been cleared [12425882/12423451]) [12426881]
Apr 17 09:28:22 core3.c.abbaandking.internal sh[22950]: 2015-04-17T09:28:22Z 16dbc64ae078 confd[418]: ERROR open /home/git/.ssh/.authorized_keys923354233: no such file or directory
Apr 17 09:28:22 core3.c.abbaandking.internal sh[22950]: 2015-04-17T09:28:22Z 16dbc64ae078 confd[418]: ERROR open /home/git/.ssh/.authorized_keys411746134: no such file or directory
Apr 17 09:28:23 core3.c.abbaandking.internal sh[22950]: 2015-04-17T09:28:23Z 16dbc64ae078 confd[418]: ERROR open /home/git/.ssh/.authorized_keys564369179: no such file or directory
Apr 17 09:28:23 core3.c.abbaandking.internal sh[22950]: 2015-04-17T09:28:23Z 16dbc64ae078 confd[418]: ERROR open /home/git/.ssh/.authorized_keys519431072: no such file or directory

[babarinde]

@bacongobbler
I restarted the deis builder and tried to do a push
the journal shows

Apr 17 09:38:39 core3.c.abbaandking.internal sh[18230]: time="2015-04-17T09:38:39Z" level="info" msg="-job push(10.240.173.211:5000/deis/slugrunner) = OK (0)"
Apr 17 09:38:39 core3.c.abbaandking.internal systemd[1]: Started deis-builder.
Apr 17 09:39:41 core3.c.abbaandking.internal sh[18230]: Accepted publickey for git from 172.17.42.1 port 58340 ssh2: RSA fb:3c:19:fb:4b:c4:1d:f8:a3:9c:76:00:df:db:70:27
Apr 17 09:39:41 core3.c.abbaandking.internal sh[18230]: Received disconnect from 172.17.42.1: 11: disconnected by user
Apr 17 09:40:10 core3.c.abbaandking.internal sh[18230]: Accepted publickey for git from 172.17.42.1 port 59073 ssh2: RSA fb:3c:11:pb:4b:c9:1d:f8:a3:9c:76:00:df:db:70:27
Apr 17 09:40:10 core3.c.abbaandking.internal sh[18230]: Received disconnect from 172.17.42.1: 11: disconnected by user

[bacongobbler]

@babarinde wallz.git was referring to the application name. If your application name is foo, then you'd need to remove /home/git/foo.git.

[onbjerg]

@bacongobbler Erh..

Cannot run exec command 8768c77e6dcc973d940d0d17f8a102ed8f2eb7e2587ad26f976269bb9bc5e821 in container d8e4c034203e28fd646c101c23df9bf82c0313a412416427fc7fba290b60b976: fork/exec /usr/bin/docker: cannot allocate memory
                         Error starting exec command in container 8768c77e6dcc973d940d0d17f8a102ed8f2eb7e2587ad26f976269bb9bc5e821: Cannot run exec command 8768c77e6dcc973d940d0d17f8a102ed8f2eb7e2587ad26f976269bb9bc5e821 in container d8e4c034203e28fd646c101c23df9bf82c0313a412416427fc7fba290b60b976: fork/exec /usr/bin/docker: cannot allocate memory

[carmstrong]

@onbjerg Can you see if the CoreOS host is OOM? And what's using the memory? What platform is this deployment on?

[onbjerg]

How can I check that in CoreOS? It is deployed on a 3-node cluster on DO.

[carmstrong]

How can I check that in CoreOS? It is deployed on a 3-node cluster on DO.

I'd start with top and investigate from there.

[onbjerg]

Definitely not OOM. Only one node has little memory left (deis-1, 233 megs) all of the others have >2GB of memory left.

[carmstrong]

Only one node has little memory left (deis-1, 233 megs)

Is that the node running builder? Confirm with deisctl list. Builder will spike memory usage during a build/deploy, so you could easily consume the rest of that memory.

[onbjerg]

@bacongobbler The way this happened was that my Macbook turned of mid-push. So I guess one way to reproduce it would be

1. Create a new project with something on it and run some sort of process (i.e. web processes)
2. Do a git push and wait until it starts building
3. CTRL+C (or terminate the terminal?) mid-push to simulate my Macbook hurting itself.
4. Try to push again.

[onbjerg]

@carmstrong Yes. It is running the builder. Can I move it somehow?

[carmstrong]

Can I move it somehow?

You could deisctl stop and deisctl uninstall the builder, and upon installing and starting again it may get scheduled to a new host. However, if all of your hosts have the same memory, you will likely run into this again.

What are the specs on these machines?

[babarinde]

@bacongobbler
Thanks for the comment on the wallz.git
Remove the existing app.git but i Just noticed something which may or may not be related
THe two apps that are experiencing this issue are apps that I had pushed using the buildpacks then later with Dockerfile.
This could just be a coincidence but the ones that I consistently pushed with Buildpack or Dockerfiles do not show this issue.

[onbjerg]

@carmstrong The 4GB instances on DO. Should do for 2 apps, though. Uninstalling and reinstalling didn't help. Dang it.

[carmstrong]

The builder has been completely rewritten to be a purely Go-based service for Deis v1.10. I'd encourage everyone to give it a go upon upgrade, and let us know if you continue to run into this issue.

[ineu]

@carmstrong It's still the case with 1.10. I've switched to stateless Deis during upgrade to 1.10 and redeployed all my apps to populate storage with docker images, so all repos were there in the builder. After some time one of repo directories was corrupted (i.e. no git files, only build and cache empty dirs). Builder was working all the time since upgrade till now, not restarted. I don't know how to reproduce this without builder restart, but here's the way to reproduce the problem with build restart (100% reproducible for me):

1. Remove /home/git/app.git
2. Push app to deis
3. Check out /home/git: consistent app.git should be there
4. Restart deis-builder

Result: /home/git/app.git is there, but it is corrupted: only cache and build directories exist.

It looks like something is removing existing directory and something creates 'corrupted' directory instead. It shouldn't be check-repos as app names are stored properly in etcd. It would be better if builder components, including scripts, could do some logging with loglevel configurable via etcd.

And also another question: why is /home/git is not stored in volume just like /var/lib/docker is? It makes it possible to push same revision again after restarting builder, which is not something Heroku allows.

[sstarcher]

We recently had the same issue with the v1.10 Builder. It looks to be functioning after doing a restart on the builder.

Sep 22 18:59:42 ip-10-0-30-227.ec2.internal sh[2007]: [info] Output:
Sep 22 18:59:42 ip-10-0-30-227.ec2.internal sh[2007]: [error] Failed git check-repos: fork/exec /home/git/check-repos: no such file or directory
Sep 22 18:59:42 ip-10-0-30-227.ec2.internal sh[2007]: [info] Output:
Sep 22 18:59:42 ip-10-0-30-227.ec2.internal sh[2007]: [error] Failed git check-repos: fork/exec /home/git/check-repos: no such file or directory
Sep 22 18:59:42 ip-10-0-30-227.ec2.internal sh[2007]: [info] Output:

[sstarcher]

I deployed a new Deis v1.10 cluster yesterday and just had the same builder failure repeat. I'll upgrade to Deis v1.10.1 and see if it repeats.

[bacongobbler]

@sstarcher I don't believe there are any relevant fixes in v1.10.1 to address this issue. We know and acknowledge that this is an issue, but there is a workaround for now and nobody has found a reliable reproduction case. If you can find a way to reliably reproduce the issue, we should be able to better nail it down, but so far we've spent a lot of time with no luck finding a common pattern on our end. :(

[ineu]

and nobody has found a reliable reproduction case

Let me quote myself: #3458 (comment)

(100% reproducible for me):

Remove /home/git/app.git
Push app to deis
Check out /home/git: consistent app.git should be there
Restart deis-builder
Result: /home/git/app.git is there, but it is corrupted: only cache and build directories exist.

[krancour]

@ineu do you happen to know if your steps for reproducing this work equally well with a stateful cluster?

[sstarcher]

@ineu reminds me we are doing something that would probably be considered unorthodox with one of our repos.

One of our Deis deployments comes from a git repo that runs some scripts and generates some artifacts.

1. Script ran artifacts generated
2. Artifacts are put into a directory that contains a Dockerfile
3. In that repo we git init and force push to Deis

So every time we are force pushing a new blank git repo to deis. What we are doing may not be the issue, but if it is it could be why only a few people are reporting this problem.

[krancour]

@ineu I am using a new v1.10.0 cluster (not stateless) and following the steps you outlined exactly. I was unable to reproduce.

I do have a few comments / questions about the steps you suggested.

1. Remove /home/git/app.git
2. Push app to deis
3. Check out /home/git: consistent app.git should be there
4. Restart deis-builder

Result: /home/git/app.git is there, but it is corrupted: only cache and build directories exist.

Although I did perform them, steps 1 - 3 strike me as superfluous because of what step 4 is. When builder restarts, its cache of git repos should go away as @carmstrong had previously indicated. In this experiment and every other I've performed, I've consistently gotten a clean slate after builder restart.

The result you documented in your instructions is that /home/git/app.git exists, but is "corrupted" (incomplete). What is odd here is that after restart, not only should that not be "corrupt--" it simply shouldn't exist at all. So I'm suspecting that somehow you're retaining some state between builder restarts.

Is it possible you could show us the exact deis-builder unit that's been scheduled via fleet?

deisctl ssh builder
fleetctl cat deis-builder

[ineu]

do you happen to know if your steps for reproducing this work equally well with a stateful cluster?

Unfortunately I don't have any stateful cluster to check.

[krancour]

@ineu, @jackfrancis just tried your steps on his stateless v.1.10.0 cluster in AWS as well with no luck reproducing. We believe this is an issue, for sure, but as hard as it's been to reproduce, we're still looking for a smoking gun.

[jackfrancis]

here's a log of that attempt to reproduce:

https://gist.github.com/jackfrancis/b412c3268267aa205a25

[krancour]

Ok! I finally had some luck reproducing this, although I had to follow different steps.

Instead of following @ineu's steps, I attempted to more accurately reproduce the OP's scenario by interrupting a push. Well... this wasn't reproducing the issue at first, then I realized that my repo was so small, I was already past the push and into the hook by the time I actually interrupted it...

So I added a large file to my repo to slow down the push so I could interrupt it during the upload instead of the hook. This worked!

The results I see are as described by others above. On a subsequent attempt to push, I'm told the repo doesn't exist or I may not have adequate permissions. When I deisctl dock builder and poke around, I can see that /home/git/app.git exists, but only contains build and cache directories.

Restarting builder does not resolve this issue as expected.

Oddly, what does seem to resolve the issue is if I make continues attempts to push the app. The failure seems to occur at most twice for me and then I'm able to successfully push again. If I interrupt this, I'm back to the failing again.

Can anyone else who has experienced this issue comment on whether continued attempts to push eventually succeed?

[sstarcher]

That sounds exactly like our problem. We are pushing around 300mb of files with our git push and it's going through a Jenkins job that sometimes people will stop before it finishes. Continued attempts to push have never worked for me. I believe every time I uninstall the builder entirely and recreate it.

[ineu]

@krancour I've also seen this problem after interrupted push first, but then retried with sucessfull push and it failed as well, which is weird. Maybe there's something wrong with gitreceive?

Also I have no idea how could it persist after restarting container, as it's not a mounted volume. To me it sounds like something is recreating it right after container starts.

[ineu]

I think this is the line that masks the issue: https://github.com/deis/deis/blob/master/builder/rootfs/etc/confd/templates/builder#L62. Parent directory must be there for builder to do something. I suspect that it doesn't exist, so builder creates it because of -p key. This explains why this directory is owned by root unlike normal *.git directories which are owned by the git user.

[carmstrong]

I suspect that it doesn't exist

This seems like a plausible explanation. Were you able to edit the builder to test this fix?

[rvadim]

I spent ~16 hours trying identify what process create catalog app.git/cache and app.git/build without success. Maybe this is bug of overlayfs or something near.
If add ls -la on /home/git/ catalog in builder/rootfs/bin/boot, it show only .ssh catalog on container start. After builder started, there are many /home/git/app.git catalogs with only empty build and cache catalog. On git hook, go code in git/git.go try to detect existence of /home/git/app.git catalog(it's exist), and skip git init and other functions, next, builder (sh file) fails with /home/git/app.git not a git repo.

I was create a working code which handle this situation by detecting some files in git repo, and if its not exists, remove catalog and create it again.

Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: [info] Found user rvadim for fingerprint 
Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: [warning] Path '/home/git/forecast.git' have not config file, remove it and recreate.
Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: [info] Creating new directory at /home/git/forecast.git
Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: [info] git-shell -c git-receive-pack 'forecast.git'
Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: Waiting for git-receive to run.
Oct 29 02:48:49 dev-node-3.novalocal sh[12086]: Waiting for deploy. 

Today I will start testing this on our dev cluster with 30+ projects, and try to create PR soon.

[carmstrong]

Thanks for the awesome work on this, @rvadim!

[carmstrong]

Maybe this is bug of overlayfs or something near.

It's a total shot in the dark, but the latest CoreOS release includes a fix for an OverlayFS issue. The error there manifests differently (could not allocate memory), but I'm wondering if we're swallowing that error in some cases when this is encountered.

[krancour]

Anyone interested in this topic who has been participating... are any of you willing to test this out with a builder built from master? It seems #4671 had inadvertently fixed this issue (altho I'm not yet clear on how). @rvadim and myself can confirm, but I'd like some confirmation from others as well.

[krancour]

Directions for the above: from the root of the project:

$ make -C builder build deploy

[colrack]

Building and deploying the deis-builder from master works for me.
Previously in my clusters simply restarting the builder was causing errors as described by @ineu and was 100% reproducible also for me (about 80MB git repo).

[krancour]

@colrack thanks for confirming what @rvadim and myself were seeing.