Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

error: failed to push some refs to 'ssh:/....' #596

Closed
jwalters-gpsw opened this issue Nov 4, 2016 · 17 comments
Closed

error: failed to push some refs to 'ssh:/....' #596

jwalters-gpsw opened this issue Nov 4, 2016 · 17 comments

Comments

@jwalters-gpsw
Copy link

jwalters-gpsw commented Nov 4, 2016 •
edited by bacongobbler
Loading

When pushing an app that has environment variables with config:set (doesn't happen if no env variables set) getting the following:

$ git push deis master

Counting objects: 34, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (32/32), done.
Writing objects: 100% (34/34), 7.27 KiB | 0 bytes/s, done.
Total 34 (delta 19), reused 0 (delta 0)
Starting build... but first, coffee!
2016/11/04 16:45:59 AccessDenied: Access Denied
remote: status code: 403, request id: 38096325BD79B427
remote: 2016/11/04 16:46:02 Error running git receive hook [Build pod exited with code 1, stopping build.]
To ssh://git@deis-builder.deis.xxx.com:2222/capci.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.deis.xxx.com:2222/capci.git'

This system has the patch from #64. And the upgrade process described at https://deis.com/docs/workflow/managing-workflow/upgrading-workflow/
and "upgraded" from v2.7.0 to v2.7.0.

I suspect the issue is related to these (numerous) events in the deis namespace:

Liveness probe failed: Get http://10.2.77.74:8092/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

MountVolume.SetUp failed for volume "kubernetes.io/secret/2cee00af-9f84-11e6-a55e-0236c9f7ddf1-deis-database-token-ae1iv" (spec.Name: "deis-database-token-ae1iv") pod "2cee00af-9f84-11e6-a55e-0236c9f7ddf1" (UID: "2cee00af-9f84-11e6-a55e-0236c9f7ddf1") with: secrets "deis-database-token-ae1iv" not found

MountVolume.SetUp failed for volume "kubernetes.io/secret/2dc3589d-9f84-11e6-a55e-0236c9f7ddf1-deis-logger-token-cfkc8" (spec.Name: "deis-logger-token-cfkc8") pod "2dc3589d-9f84-11e6-a55e-0236c9f7ddf1" (UID: "2dc3589d-9f84-11e6-a55e-0236c9f7ddf1") with: secrets "deis-logger-token-cfkc8" not found

MountVolume.SetUp failed for volume "kubernetes.io/secret/2f037527-9f84-11e6-a55e-0236c9f7ddf1-deis-nsqd-token-gq2lt" (spec.Name: "deis-nsqd-token-gq2lt") pod "2f037527-9f84-11e6-a55e-0236c9f7ddf1" (UID: "2f037527-9f84-11e6-a55e-0236c9f7ddf1") with: secrets "deis-nsqd-token-gq2lt" not found

MountVolume.SetUp failed for volume "kubernetes.io/secret/79ed25b5-a125-11e6-a55e-0236c9f7ddf1-deis-registry-token-8p0jo" (spec.Name: "deis-registry-token-8p0jo") pod "79ed25b5-a125-11e6-a55e-0236c9f7ddf1" (UID: "79ed25b5-a125-11e6-a55e-0236c9f7ddf1") with: secrets "deis-registry-token-8p0jo" not found

MountVolume.SetUp failed for volume "kubernetes.io/secret/2fecac0f-9f84-11e6-a55e-0236c9f7ddf1-deis-router-token-ca4of" (spec.Name: "deis-router-token-ca4of") pod "2fecac0f-9f84-11e6-a55e-0236c9f7ddf1" (UID: "2fecac0f-9f84-11e6-a55e-0236c9f7ddf1") with: secrets "deis-router-token-ca4of" not found

MountVolume.SetUp failed for volume "kubernetes.io/secret/3056bdad-9f84-11e6-a55e-0236c9f7ddf1-deis-workflow-manager-token-am7uc" (spec.Name: "deis-workflow-manager-token-am7uc") pod "3056bdad-9f84-11e6-a55e-0236c9f7ddf1" (UID: "3056bdad-9f84-11e6-a55e-0236c9f7ddf1") with: secrets "deis-workflow-manager-token-am7uc" not found
@bacongobbler
Copy link
Member

2016/11/04 16:45:59 AccessDenied: Access Denied
remote: status code: 403, request id: 38096325BD79B427

This issue probably indicates to a controller issue. Did you remember to run deis keys:add before running git push?

@bacongobbler
Copy link
Member

Is the rest of your cluster up and running? You can check using the guide here: https://deis.com/docs/workflow/troubleshooting/kubectl/

@jwalters-gpsw
Copy link
Author

Yes, the keys are there. I am able to push apps that don't have environment variables set with config:set.

@jwalters-gpsw
Copy link
Author

jwalters-gpsw commented Nov 4, 2016 •
edited by bacongobbler
Loading

Yes. All is running.

NAME                                     READY     STATUS    RESTARTS   AGE
deis-builder-560128805-2ygob             1/1       Running   0          1h
deis-controller-423606151-e2gfi          1/1       Running   0          1d
deis-database-1591134702-8et76           1/1       Running   1          4d
deis-logger-4136210725-ki4r9             1/1       Running   3          4d
deis-logger-fluentd-12ct1                1/1       Running   0          1d
deis-logger-fluentd-4r71y                1/1       Running   0          1d
deis-logger-fluentd-7cw3n                1/1       Running   0          1d
deis-logger-fluentd-d59us                1/1       Running   0          1d
deis-logger-fluentd-x7ddl                1/1       Running   0          1d
deis-logger-redis-573083235-5dkxm        1/1       Running   1          4d
deis-monitor-grafana-213867964-54c7h     1/1       Running   1          4d
deis-monitor-influxdb-2425701573-ljtl2   1/1       Running   1          4d
deis-monitor-telegraf-2npa4              1/1       Running   0          1d
deis-monitor-telegraf-cjaao              1/1       Running   0          1d
deis-monitor-telegraf-i2tbm              1/1       Running   0          1d
deis-monitor-telegraf-j86k3              1/1       Running   0          1d
deis-monitor-telegraf-nqdai              1/1       Running   0          1d
deis-nsqd-3097922368-iovwp               1/1       Running   1          4d
deis-registry-511029204-lkmyx            1/1       Running   1          1d
deis-registry-proxy-3uuq9                1/1       Running   1          4d
deis-registry-proxy-ae8ij                1/1       Running   0          4d
deis-registry-proxy-ccx8q                1/1       Running   1          4d
deis-registry-proxy-wt48x                1/1       Running   1          4d
deis-registry-proxy-zmrsf                1/1       Running   1          4d
deis-router-2099563716-tex20             1/1       Running   1          4d
deis-workflow-manager-2052028020-ybbee   1/1       Running   1          4d

@bacongobbler
Copy link
Member

Can you post the controller and builder logs as well?

@jwalters-gpsw
Copy link
Author

jwalters-gpsw commented Nov 4, 2016 •
edited by bacongobbler
Loading

I'll get the logs, but does this look right?

$ kubectl --namespace=deis get rs
NAME                               DESIRED   CURRENT   AGE
**deis-builder-1714497148            0         0         4d
deis-builder-560128805             1         1         1d**
**deis-controller-1866332879         0         0         4d
deis-controller-423606151          1         1         1d**
deis-database-1591134702           1         1         4d
deis-logger-4136210725             1         1         4d
deis-logger-redis-573083235        1         1         4d
deis-monitor-grafana-213867964     1         1         4d
deis-monitor-influxdb-2425701573   1         1         4d
deis-nsqd-3097922368               1         1         4d
deis-registry-511029204            1         1         4d
deis-router-2099563716             1         1         4d
deis-workflow-manager-2052028020   1         1         4d

@bacongobbler
Copy link
Member

I just edited your pastes so it's easier to read for others. For future reference, using Github markdown or pasting a link to a gist helps keep the ticket length from going too cray :)

It looks completely normal to me.

@jwalters-gpsw
Copy link
Author

jwalters-gpsw commented Nov 4, 2016 •
edited by bacongobbler
Loading

deis-builder log:

2016-11-04T16:42:18.453392599Z 
2016-11-04T16:42:18.506127414Z Key='LANG', Value='en_US.UTF-8'
2016-11-04T16:42:18.506157296Z 
2016-11-04T16:42:18.506163606Z receiving git repo name: slack-deis.git, operation: git-receive-pack, fingerprint: f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54, user: jwalters
2016-11-04T16:42:18.506170122Z creating repo directory /home/git/slack-deis.git
2016-11-04T16:42:18.511650409Z writing pre-receive hook under /home/git/slack-deis.git
2016-11-04T16:42:18.511671953Z git-shell -c git-receive-pack 'slack-deis.git'
2016-11-04T16:42:18.607194383Z Waiting for git-receive to run.
2016-11-04T16:42:18.607225665Z Waiting for deploy.
2016-11-04T16:44:15.631469907Z Deploy complete.
2016-11-04T16:45:54.916555413Z Accepted connection.
2016-11-04T16:45:56.146309950Z Starting ssh authentication
2016-11-04T16:45:56.304027254Z Channel type: session
2016-11-04T16:45:56.304051935Z 
2016-11-04T16:45:56.384790064Z Key='LANG', Value='en_US.UTF-8'
2016-11-04T16:45:56.385166405Z 
2016-11-04T16:45:56.385181956Z receiving git repo name: dashboard-capci.git, operation: git-receive-pack, fingerprint: f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54, user: jwalters
2016-11-04T16:45:56.385197961Z creating repo directory /home/git/dashboard-capci.git
2016-11-04T16:45:56.385203646Z writing pre-receive hook under /home/git/dashboard-capci.git
2016-11-04T16:45:56.385713700Z git-shell -c git-receive-pack 'dashboard-capci.git'
2016-11-04T16:45:56.638055556Z Waiting for git-receive to run.
2016-11-04T16:45:56.638090675Z Waiting for deploy.
2016-11-04T16:46:02.515767457Z Deploy complete.

deis-controller log:

2016-11-04T15:41:38.789250963Z 10.2.77.52 "POST /v2/apps/ HTTP/1.1" 201 170 "Deis Client v2.7.1"
2016-11-04T15:41:49.131402126Z INFO [dashboard-capci]: config dashboard-capci-341f1a3 updated
2016-11-04T15:41:49.139451654Z INFO [dashboard-capci]: jwalters added XXX, YYY
2016-11-04T15:41:49.139471793Z INFO [dashboard-capci]: No build associated with this release to publish
2016-11-04T15:41:49.140456295Z 10.2.77.52 "POST /v2/apps/dashboard-capci/config/ HTTP/1.1" 201 524 "Deis Client v2.7.1"
2016-11-04T15:41:49.258218125Z 10.2.77.52 "GET /v2/apps/dashboard-capci/config/ HTTP/1.1" 200 524 "Deis Client v2.7.1"
2016-11-04T15:41:58.164290971Z 10.2.21.59 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T15:41:59.352121911Z 10.2.21.59 "POST /v2/hooks/config/ HTTP/1.1" 200 524 "deis-builder"
2016-11-04T15:46:38.940128102Z 10.2.77.74 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T15:46:40.214195683Z 10.2.77.74 "POST /v2/hooks/config/ HTTP/1.1" 200 524 "deis-builder"
2016-11-04T16:42:18.330723356Z 10.2.77.74 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T16:42:19.998435988Z 10.2.77.74 "POST /v2/hooks/config/ HTTP/1.1" 200 225 "deis-builder"
2016-11-04T16:43:09.882229517Z INFO [slack-deis]: build slack-deis-9ff4bae created
2016-11-04T16:43:09.892134233Z INFO [slack-deis]: jwalters deployed 9d19dca
2016-11-04T16:43:09.906625098Z INFO [slack-deis]: buildpack type detected. Defaulting to $PORT 5000
2016-11-04T16:43:09.911525453Z INFO [slack-deis]: buildpack type detected. Defaulting to $PORT 5000
2016-11-04T16:43:10.988653577Z INFO [slack-deis]: adding 30s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
2016-11-04T16:43:10.988690260Z INFO [slack-deis]: This deployments overall timeout is 150s - batch timout is 150s and there are 1 batches to deploy with a total of 1 pods
2016-11-04T16:43:21.057388388Z INFO [slack-deis]: waited 10s and 1 pods are in service
2016-11-04T16:43:31.126709374Z INFO [slack-deis]: waited 20s and 1 pods are in service
2016-11-04T16:43:41.197210726Z INFO [slack-deis]: waited 30s and 1 pods are in service
2016-11-04T16:44:15.615858889Z 10.2.77.74 "POST /v2/hooks/build/ HTTP/1.1" 200 25 "deis-builder"
2016-11-04T16:45:56.162347106Z 10.2.77.74 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T16:45:57.319996058Z 10.2.77.74 "POST /v2/hooks/config/ HTTP/1.1" 200 524 "deis-builder"

@jwalters-gpsw
Copy link
Author

Thanks. I was just wondering about the "duplicate" replica sets, controller and builder.

@bacongobbler
Copy link
Member

bacongobbler commented Nov 4, 2016 •
edited
Loading

I was just wondering about the "duplicate" replica sets, controller and builder.

Assuming you "upgraded", deployments retain their older replica sets available for rollbacks. It is completely normal and is intentional by design.

http://kubernetes.io/docs/user-guide/deployments/

can you try pushing again, but before pushing run deis config:set DEIS_DEBUG=true then post the logs from the push again? The controller and builder logs look normal.

@jwalters-gpsw
Copy link
Author

jwalters-gpsw commented Nov 4, 2016 •
edited by bacongobbler
Loading

(Sorry about the formatting. I need to get that figured out.) Console:

$ git push deis master
Counting objects: 34, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (32/32), done.
Writing objects: 100% (34/34), 7.27 KiB | 0 bytes/s, done.
Total 34 (delta 19), reused 0 (delta 0)
Starting build... but first, coffee!
+ unset DEIS_DEBUG
+ app_dir=/app
+ build_root=/tmp/build
+ cache_root=/tmp/cache
+ cache_file=/tmp/cache.tgz
+ env_root=/tmp/env
+ buildpack_root=/tmp/buildpacks
+ mkdir -p /app
+ mkdir -p /tmp/cache
+ mkdir -p /tmp/env
+ mkdir -p /tmp/buildpacks
+ mkdir -p /tmp/build/.profile.d
+ [[ -z home/dashboard-capci:git-2e0f394b/tar ]]
+ get_object
2016/11/04 17:30:12 AccessDenied: Access Denied
remote: status code: 403, request id: 344FEAE3B181DC10
+ sleep_before_exit
+ sleep 3
remote: 2016/11/04 17:30:15 Error running git receive hook [Build pod exited with code 1, stopping build.]
To ssh://git@deis-builder.deis.k8.gopro-platform.com:2222/dashboard-capci.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.deis.k8.gopro-platform.com:2222/dashboard-capci.git'

deis-controller:

2016-11-04T16:43:21.057388388Z INFO [slack-deis]: waited 10s and 1 pods are in service
2016-11-04T16:43:31.126709374Z INFO [slack-deis]: waited 20s and 1 pods are in service
2016-11-04T16:43:41.197210726Z INFO [slack-deis]: waited 30s and 1 pods are in service
2016-11-04T16:44:15.615858889Z 10.2.77.74 "POST /v2/hooks/build/ HTTP/1.1" 200 25 "deis-builder"
2016-11-04T16:45:56.162347106Z 10.2.77.74 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T16:45:57.319996058Z 10.2.77.74 "POST /v2/hooks/config/ HTTP/1.1" 200 524 "deis-builder"
2016-11-04T17:29:45.082169468Z INFO [dashboard-capci]: config dashboard-capci-a0a7851 updated
2016-11-04T17:29:45.090812012Z INFO [dashboard-capci]: jwalters added DEIS_DEBUG
2016-11-04T17:29:45.090834581Z INFO [dashboard-capci]: No build associated with this release to publish
2016-11-04T17:29:45.091973363Z 10.2.77.52 "POST /v2/apps/dashboard-capci/config/ HTTP/1.1" 201 544 "Deis Client v2.7.1"
2016-11-04T17:29:45.212805763Z 10.2.77.52 "GET /v2/apps/dashboard-capci/config/ HTTP/1.1" 200 544 "Deis Client v2.7.1"
2016-11-04T17:30:09.146230539Z 10.2.77.74 "GET /v2/hooks/key/f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54 HTTP/1.1" 200 174 "deis-builder"
2016-11-04T17:30:10.618367778Z 10.2.77.74 "POST /v2/hooks/config/ HTTP/1.1" 200 544 "deis-builder"

deis-builder:

2016-11-04T16:45:56.385181956Z receiving git repo name: dashboard-capci.git, operation: git-receive-pack, fingerprint: f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54, user: jwalters
2016-11-04T16:45:56.385197961Z creating repo directory /home/git/dashboard-capci.git
2016-11-04T16:45:56.385203646Z writing pre-receive hook under /home/git/dashboard-capci.git
2016-11-04T16:45:56.385713700Z git-shell -c git-receive-pack 'dashboard-capci.git'
2016-11-04T16:45:56.638055556Z Waiting for git-receive to run.
2016-11-04T16:45:56.638090675Z Waiting for deploy.
2016-11-04T16:46:02.515767457Z Deploy complete.
2016-11-04T17:30:07.457841266Z Accepted connection.
2016-11-04T17:30:09.129625991Z Starting ssh authentication
2016-11-04T17:30:09.304002375Z Channel type: session
2016-11-04T17:30:09.304026703Z 
2016-11-04T17:30:09.358616412Z Key='LANG', Value='en_US.UTF-8'
2016-11-04T17:30:09.358640258Z 
2016-11-04T17:30:09.361219708Z receiving git repo name: dashboard-capci.git, operation: git-receive-pack, fingerprint: f3:e4:19:db:17:e7:39:be:08:de:ec:77:05:0d:2a:54, user: jwalters
2016-11-04T17:30:09.361245747Z creating repo directory /home/git/dashboard-capci.git
2016-11-04T17:30:09.361254384Z writing pre-receive hook under /home/git/dashboard-capci.git
2016-11-04T17:30:09.361260767Z git-shell -c git-receive-pack 'dashboard-capci.git'
2016-11-04T17:30:09.587103797Z Waiting for git-receive to run.
2016-11-04T17:30:09.587134568Z Waiting for deploy.
2016-11-04T17:30:15.739063962Z Deploy complete.

@kmala
Copy link
Contributor

kmala commented Nov 4, 2016 •
edited
Loading

Can you tell the name of the env variable you are setting...i sense it is overriding some of the object store variables.
This should fix your issues mostly deis/builder#447

@jwalters-gpsw
Copy link
Author

The variables set with config:set are:

JENKINS_USER
JENKINS_PASSWORD
SLACK_TOKEN
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
WIKI_TOKEN

@bacongobbler
Copy link
Member

bacongobbler commented Nov 4, 2016 •
edited
Loading

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Those are likely messing with the builder, since those are the environment variables we use to upload slugs to the object store. Try settings those up differently for the time being, such as ACCESS_KEY_ID and SECRET_ACCESS_KEY.

@jwalters-gpsw
Copy link
Author

That was it! Thanks!

@jwalters-gpsw
Copy link
Author

Closing issue. Maybe want to think about error message or ENV variable names.

@bacongobbler
Copy link
Member

We're planning on fixing this issue entirely in deis/builder#447 so that should cover this issue, but thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bacongobbler @kmala @jwalters-gpsw