fly intercept fails with "error: 403 Forbidden websocket: bad handshake"

[RichardBradley]

On a previous version of control-tower, I was able to ssh into build Docker tasks using fly intercept.
On the current version, it fails with the following error:

$ fly --verbose -t xxx intercept -j "xxx/xxx Admin App build" -- /bin/sh
2020/03/02 15:37:54 GET /api/v1/info HTTP/1.1
Host: ci.xxx.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


2020/03/02 15:37:55 HTTP/1.1 200 OK
Content-Length: 86
Content-Type: application/json
Date: Mon, 02 Mar 2020 15:37:55 GMT
Vary: Accept-Encoding
X-Concourse-Version: 5.8.0
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: deny
X-Xss-Protection: 1; mode=block

{"version":"5.8.0","worker_version":"2.2","external_url":"https://ci.xxx.com"}

2020/03/02 15:37:55 GET /api/v1/teams/main/containers?job_name=xxx+Admin+App+build&pipeline_name=xxx HTTP/1.1
Host: ci.xxx.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


2020/03/02 15:37:55 HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 02 Mar 2020 15:37:55 GMT
Vary: Accept-Encoding
X-Concourse-Version: 5.8.0
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: deny
X-Xss-Protection: 1; mode=block

[{"id":"a6afbe27-52a2-4149-571b-b73146dce533","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","type":"get","step_name":"email","pipeline_id":2,"job_id":34,"build_id
":124,"pipeline_name":"xxx","job_name":"xxx Admin App build","build_name":"14","working_directory":"/tmp/build/get"},{"id":"3c68329c-ba9f-42f4-73e7-db91a9052a5d","worker_name":"74bd2880
-200d-422f-a171-f91fd10136e7","state":"created","type":"get","step_name":"admin-app","pipeline_id":2,"job_id":34,"build_id":129,"pipeline_name":"xxx","job_name":"xxx Admin App build","b
uild_name":"15","working_directory":"/tmp/build/get"},{"id":"e9df16c5-9e2e-4ec7-79cc-fecb0c492b0e","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","type":"get","ste
p_name":"admin-app","pipeline_id":2,"job_id":34,"build_id":124,"pipeline_name":"xxx","job_name":"xxx Admin App build","build_name":"14","working_directory":"/tmp/build/get"},{"id":"eb8e
1aef-fa1d-448e-473b-d37999981158","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","type":"put","step_name":"email","pipeline_id":2,"job_id":34,"build_id":124,"pipel
ine_name":"xxx","job_name":"xxx Admin App build","build_name":"14","working_directory":"/tmp/build/put"},{"id":"ae9ebf50-48e6-4094-4dbb-bdabd2ead9a7","worker_name":"74bd2880-200d-422f-a
171-f91fd10136e7","state":"created","type":"task","step_name":"Copy git info into build dir","pipeline_id":2,"job_id":34,"build_id":129,"pipeline_name":"xxx","job_name":"xxx Admin App b
uild","build_name":"15","working_directory":"/tmp/build/3d63b01d"},{"id":"9bb081aa-c9a1-4d6f-7eb8-7b70b02ee94a","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","typ
e":"put","step_name":"admin-app-docker-image-int","pipeline_id":2,"job_id":34,"build_id":129,"pipeline_name":"xxx","job_name":"xxx Admin App build","build_name":"15","working_directory"
:"/tmp/build/put"},{"id":"18edd165-051a-4ffa-60f9-f54166a787ad","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","type":"task","step_name":"Copy git info into build
dir","pipeline_id":2,"job_id":34,"build_id":124,"pipeline_name":"xxx","job_name":"xxx Admin App build","build_name":"14","working_directory":"/tmp/build/3d63b01d"},{"id":"aea4eb58-ffe0-
4d5b-4a43-239e343d927f","worker_name":"74bd2880-200d-422f-a171-f91fd10136e7","state":"created","type":"put","step_name":"admin-app-docker-image-int","pipeline_id":2,"job_id":34,"build_id"
:124,"pipeline_name":"xxx","job_name":"xxx Admin App build","build_name":"14","working_directory":"/tmp/build/put"}]

1: build #14, step: Copy git info into build dir, type: task
2: build #14, step: admin-app, type: get
3: build #14, step: admin-app-docker-image-int, type: put
4: build #14, step: email, type: get
5: build #14, step: email, type: put
6: build #15, step: Copy git info into build dir, type: task
7: build #15, step: admin-app, type: get
8: build #15, step: admin-app-docker-image-int, type: put
choose a container: 8
error: 403 Forbidden websocket: bad handshake

I don't know if this changed due to the version change or is failing for another reason.
How can I get more information about what is wrong here and do you know how I might fix it?

Thanks,

Rich

[crsimmons]

I can still hijack containers on our Concourse which is running the latest Control Tower release.

What flags did you pass when you first deployed?

[RichardBradley]

What flags did you pass when you first deployed?

control-tower deploy --iaas AWS --domain ci.xxx.com --github-auth-client-secret xxx --github-aut
h-client-id xxx ci.xxx.com

[ostenbom]

Shot in the dark, had a similar error the other day: I noticed you're using github authentication, could it be your team permissions that's causing this?

When you fly set-team it sets the owner of that team, you might want more complex permissions (where you can use github auth but the "admin" user still works):

---
roles:
- name: owner
  local:
    users: ["admin"]
- name: member
  github:
    teams: ["org:team"]
- name: viewer
  github:
    orgs: ["org"]

[RichardBradley]

That could well be it, thanks.
Here is my auth config:

roles:
- name: owner
  local:
    users: ["admin"]
- name: pipeline-operator
  github:
    teams: ["myorg:myteam"]

Does that give sufficient permission for a GitHub authenticated user to perform this op?

If not a) what permission do I need and how can I grant it in addition to "pipeline-operator"? and b) I think it would be useful to update the system so the error message is clearer in this case.

[ostenbom]

That looks about right depending on what you want/ based on the docs. When admin looses privileges I think there are sporadic error messages all over the place. I noticed it in control-tower-self-update

[RichardBradley]

This is still happening for me.

Does anyone know how I can get more information about what is wrong here or how I might fix it? The error message doesn't even say which connection failed, even with --verbose .

[irbekrm]

Hi @RichardBradley ,

Sorry to hear you're having issues with Control Tower. We haven't seen this error before, but could try to replicate it.
What version of Control Tower are you using? (And do you perhaps remember what version it was that you said was working for you?)

It might be worth asking the question on Concourse Discord or their discussion list - this does seem like a very Concourse-specific issue, maybe someone there has seen it before.

[Anonymous-Coward]

I run into the same problem when using fly hijack. There's not much to it:

$ fly -t my-target hijack -j pipeline/job -c pipeline/step sh
error: 403 Forbidden websocket: bad handshake

Both fly and concourse are at version 7.10.0. I'm using git authentication too.