boundary targets authorize-session command failed with runtime error

[raylaijh]

Describe the bug
boundary targets authorize-session command failed with runtime error.

To Reproduce
Steps to reproduce the behavior:

1. Follow guide from https://learn.hashicorp.com/tutorials/boundary/vault-cred-brokering-quickstart?in=boundary/configuration#use-boundary-to-connect-to-the-northwind-demo-database
2. Run boundary targets authorize-session -id ttcp_hOMcjCWSVaand error occured as below.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x217b670]

goroutine 1 [running]:
github.com/hashicorp/boundary/internal/cmd/commands/targetscmd.printCustomActionOutputImpl(0xc0003b3720, 0x0, 0x0, 0x0)
        /go/internal/cmd/commands/targetscmd/funcs.go:822 +0x970
github.com/hashicorp/boundary/internal/cmd/commands/targetscmd.(*Command).Run(0xc0003b3720, 0xc000114080, 0x2, 0x2, 0xc000142858)
        /go/internal/cmd/commands/targetscmd/targets.gen.go:324 +0x806
github.com/mitchellh/cli.(*CLI).Run(0xc000369900, 0xc000369900, 0xc000142d08, 0xc00037c8c0)
        /root/go/pkg/mod/github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x41a
github.com/hashicorp/boundary/internal/cmd.RunCustom(0xc000114060, 0x4, 0x4, 0xc000427e60, 0xc000116058)
        /go/internal/cmd/main.go:186 +0x846
github.com/hashicorp/boundary/internal/cmd.Run(...)
        /go/internal/cmd/main.go:92
main.main()
        /go/cmd/boundary/main.go:13 +0xda

Expected behavior
Command to work and Vault is able broker credentials for Boundary user.

Additional context
Running Boundary 0.5.1 on docker container, and Vault as well. My setup is based in the zip archive below.

Archive.zip

[dcardozoo]

I have a Similar issue using the SSH Vault engine. I followed the example which uses the DB engine here: https://learn.hashicorp.com/tutorials/boundary/vault-cred-brokering-quickstart

For my test setup I ran Boundary and Vault in docker containers, setup the vault policies, created an ssh role with OTP and created a vault Token for it associated with both ssh and boundary-controller policies.

On the Boundary side Also I could add the Credential-store and library to my scope and target without issues.

But when trying to authorize-session:

$ boundary targets authorize-session -id=ttcp_pmqFSzOHyr
Error from controller when performing authorize-session on a session against target

Error information:
  Kind:                Internal
  Message:             targets.(Service).AuthorizeSession: vault.(Repository).Issue: vault.(client).get: vault: http://vault:8200: external system issue: error #3014: Error making API request.

  URL: GET http://vault:8200/v1/ssh/creds/backup1
  Code: 405. Errors:

  * 1 error occurred:
  * unsupported operation


  Status:              500
  context:             Error from controller when performing authorize-session on a session against target

Checking the Boundary logs I see almost the same output:

2021-08-27T14:30:37.622Z [ERROR] controller: internal error returned:
  error=
  | targets.(Service).AuthorizeSession: vault.(Repository).Issue: vault.(client).get: vault: http://vault:8200: external system issue: error #3014: Error making API request.
  |
  | URL: GET http://vault:8200/v1/ssh/creds/backup1
  | Code: 405. Errors:
  |
  | * 1 error occurred:
  | 	* unsupported operation
  |

also via Curl:

$ curl -X POST -H "Authorization: Bearer $(boundary config get-token -keyring-type keychain -token-name default)" -H "Content-Type: application/json" -d '{}' 'http://127.0.0.1:9200/v1/targets/ttcp_pmqFSzOHyr:authorize-session'

{"kind":"Internal", "message":"targets.(Service).AuthorizeSession: vault.(Repository).Issue: vault.(client).get: vault: http://vault:8200: external system issue: error #3014: Error making API request.\n\nURL: GET http://vault:8200/v1/ssh/creds/backup1\nCode: 405. Errors:\n\n* 1 error occurred:\n\t* unsupported operation\n\n"}%

[raylaijh]

Tested again with Vault running in systemd insted of as a docker container. Same issues. Problem lies with Boundary running in Docker

[dcardozoo]

For me is still not working. We have now tested with a Client library which uses a POST request and added in the body section the IP we want to create the OTP. When doing that we get the same error as @raylaijh has:

$ boundary targets authorize-session -id ttcp_k1bEH0A9Kq -addr=http://192.168.33.10:9200
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x217b670]

goroutine 1 [running]:
github.com/hashicorp/boundary/internal/cmd/commands/targetscmd.printCustomActionOutputImpl(0xc0003600a0, 0x0, 0x0, 0x0)
	/go/internal/cmd/commands/targetscmd/funcs.go:822 +0x970
github.com/hashicorp/boundary/internal/cmd/commands/targetscmd.(*Command).Run(0xc0003600a0, 0xc00003a270, 0x3, 0x3, 0xc00000c030)
	/go/internal/cmd/commands/targetscmd/targets.gen.go:324 +0x806
github.com/mitchellh/cli.(*CLI).Run(0xc0005a2140, 0xc0005a2140, 0xc00000c630, 0xc00008a560)
	/root/go/pkg/mod/github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x41a
github.com/hashicorp/boundary/internal/cmd.RunCustom(0xc00003a250, 0x5, 0x5, 0xc00041fe60, 0xc000096058)
	/go/internal/cmd/main.go:186 +0x846
github.com/hashicorp/boundary/internal/cmd.Run(...)
	/go/internal/cmd/main.go:92
main.main()
	/go/cmd/boundary/main.go:13 +0xda

Doing also another tests, we found: https://stackoverflow.com/questions/41858635/segmentation-violation-with-golang-channels - this makes sense when we do a PUT, and testing using a invalid vault path we had:

$ boundary targets authorize-session -id ttcp_k1bEH0A9Kq -addr=http://192.168.33.10:9200
Error from controller when performing authorize-session on a session against target

Error information:
  Kind:                Internal
  Message:             targets.(Service).AuthorizeSession: vault.(Repository).Issue: vault.(client).post: vault: http://192.168.33.13:8200: external system issue: error #3014: Error making API request.

  URL: PUT http://192.168.33.13:8200/v1/v1/ssh/creds/onetime
  Code: 403. Errors:

  * 1 error occurred:
  * permission denied


  Status:              500
  context:             Error from controller when performing authorize-session on a session against target

So it seems Boundary is doing a PUT request instead of POST. Using the -output-curl-string flag we get a valid curl with a POST request and it works!

$ boundary targets authorize-session -id ttcp_k1bEH0A9Kq -addr=http://192.168.33.10:9200 -output-curl-string
curl -X POST -H "Authorization: Bearer $(boundary config get-token -keyring-type keychain -token-name default)" -H "Content-Type: application/json" -d '{}' 'http://192.168.33.10:9200/v1/targets/ttcp_k1bEH0A9Kq:authorize-session'


$ curl -X POST -H "Authorization: Bearer $(boundary config get-token -keyring-type keychain -token-name default)" -H "Content-Type: application/json" -d '{}' 'http://192.168.33.10:9200/v1/targets/ttcp_k1bEH0A9Kq:authorize-session'
{"session_id":"s_zsFZzx4H8N", "target_id":"ttcp_k1bEH0A9Kq", "scope":{"id":"p_ZjZCnS71an", "type":"project", "name":"Generated project scope", "description":"Provides an initial project scope in Boundary", "parent_scope_id":"o_g2CpctCk0m"}, "created_time":"2021-08-30T13:28:12.657881Z", "user_id":"u_2Xc3PpgBKh", "host_set_id":"hsst_i8B6vWvLXm", "host_id":"hst_MAXRGoepbx", "type":"tcp", "authorization_token":"9RcST2WWXq4ttw62kfMqXGAwuAbJyBRgWKCzP9hA1EHEcKRaRqpbDw7pf5o6w3C9VxiB5wfRJFziddRb79zMQK7yH68KQaUfzZV3RQT8vNqhCsASBxuRT966ndcvLUU8xTjv3uWD2tRNwyJNJUKk9q2x1ytvBTQmBjSf644F4ANQYmv51ZBtww7dxCR1gcqsw6boZVNRXka5QL6uysbAxtDUtXAxzZ5HAaceGABHH3BK8cDwuSimWWqwJgcfmer16iBcwmuzZnv2av7LcJWPHFtLs84Xkyn7YHMeGEsUEYGGax4FaoVjj5Qkw9GaqRasQKHNVpHMU1emKdWi3wxkfwft5YDd8bhsjfSWi6VVFYPvthm8kzScPqy7yaj9HEBXKZd5ZWjtpPRnZXdBEmXEPQ6eYD9GZDYmRLW4Rko7RnV64nat77fTNMFaXCH8Sn8mTGVaTv3GsMQHU81DCDX7654FExsoXkJDGaxpDAfDcZjmHDzUo2ZjYGCN4o4WySjgzRdXdbvs5ToMZmKbGHhFfWKXXxXg6hZtPuDsx3iNMKWot4p3bXN5dfA8WEMVe8ueybCWibyN22DweHWBS5XhGdyUcsdY5zC8vcWhNBKApCu2e7FUyfxPXGDahMTDuzK2QMJiKASwKZ4ZNYT3RLBuRHBByak2wpZTKkmGKSow4ThRUXgw4EZLYhEPuEQYMvLDV9tTfBxxAa4iVZwxKwU2y7qmQgSTaJnVcNNXv3YSr79PkW5dogvBA8W5VuPeBpB6xXoxSYtjzqSegBoUhSEEM6qtzEPTQSDF5t7AmaUt15DgJbchT1nMp3kBCAxMRcPD5JJVCM19iCcVRrgchzZ", "endpoint":"tcp://157.90.33.194:22", "credentials":[{"credential_library":{"id":"clvlt_97JVcKhEVq", "credential_store_id":"csvlt_Zj7KhJiHLX", "type":"vault"}, "secret":{"raw":"eyJpcCI6IjEuMi4zLjQiLCJrZXkiOiIwMDAzZjdiNy1iZWIyLTdhYmMtMjEyMC1kZDYyYWUyODZjZGIiLCJrZXlfdHlwZSI6Im90cCIsInBvcnQiOjIyLCJ1c2VybmFtZSI6InJvb3QifQ==", "decoded":{"ip":"1.2.3.4", "key":"0003f7b7-beb2-7abc-2120-dd62ae286cdb", "key_type":"otp", "port":22, "username":"root"}}}]}%

So the issue is definitively Boundary using the wrong method to talk to the vault API.

We have tested with version 0.4.0 without errors. it seems since version 0.5.0 there is the issue which boundary uses the wrong HTTP method PUT instead POST.

[raylaijh]

thanks @dcardozoo for the findings. I've verified as well with 0.4.0 and it worked without the errors.

[dcardozoo]

@raylaijh Do you know how to make boundary to login automatically via vault otc SSH? so far we have created a credential library as follows:

Credential Library information:
  Created Time:          Mon, 30 Aug 2021 15:56:05 UTC
  Credential Store ID:   csvlt_vAD617Cehx
  ID:                    clvlt_RCnP83L42k
  Name:                  vault2
  Type:                  vault
  Updated Time:          Tue, 31 Aug 2021 08:14:00 UTC
  Version:               2

  Scope:
    ID:                  p_w23LrB185I
    Name:                Generated project scope
    Parent Scope ID:     o_MVv8keEFKD
    Type:                project

  Authorized Actions:
    no-op
    read
    update
    delete

  Attributes:
    HTTP Method:         POST
    HTTP Request Body:   {"ip":"192.168.33.13","username":"root"}
    Path:                ssh/creds/onetime

I expected that the ip and username will be used by boundary to login into the server. (without the body, the post method does not work at all with HTTP 500 response and again tries a PUT request)

Trying to login:

$ boundary connect ssh -target-id ttcp_1oo0dIVNZw -addr=http://192.168.33.10:9200
Credentials:
  Credential Library ID:   clvlt_RCnP83L42k
  Credential Library Name: vault2
  Credential Store ID:     csvlt_vAD617Cehx
  Credential Store Type:   vault
  Secret:
      {
          "ip": "192.168.33.13",
          "key": "b44303d7-c5f5-ff60-cfff-586422b3c0e1",
          "key_type": "otp",
          "port": 22,
          "username": "root"
      }
Password:
Password:
Password:
vagrant@127.0.0.1: Permission denied (publickey,keyboard-interactive).

the remote server says user mismatch so in order to work I had to add the -username flag so boundary knows it has to use root as username.

$ boundary connect ssh -target-id ttcp_1oo0dIVNZw -username=root -addr=http://192.168.33.10:9200
Credentials:
  Credential Library ID:   clvlt_RCnP83L42k
  Credential Library Name: vault2
  Credential Store ID:     csvlt_vAD617Cehx
  Credential Store Type:   vault
  Secret:
      {
          "ip": "192.168.33.13",
          "key": "f599a5f5-e358-4e7c-e8b2-498a351a150e",
          "key_type": "otp",
          "port": 22,
          "username": "root"
      }
Password:
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

Last login: Tue Aug 31 08:19:01 2021 from 192.168.33.11
root@vagrant:~#

The Scenario I was expecting from boundary is that we do not need even to put anything on the body request, so Boundary will be clever enough to send the Server IP and username to Vault, so it can create an OTP and once the OTP is created boundary will try to ssh automatically to the server without any prompts.

[louisruch]

Hello @raylaijh and @dcardozoo I have been able to repro the nil panic error and will have a fix for that shortly. The issue was due to a internal rename of credential lirbaries -> credential sources.

Re the PUT vs POST discussion. Currently Boundary will either perform a GET or POST depending on how the store was configured.

[louisruch]

@raylaijh and @dcardozoo,

I have a PR up that should fix this issue, but in the mean time you can include the - format json flag, which will bypass the code that is currently hitting this panic. json is likely a better way to consume the output of authorize-session

Example from my system:

 boundary targets authorize-session -id ttcp_YliG7PP5ed -format json
{"status_code":200,"item":{"session_id":"s_bYDY1Xo7wK","target_id":"ttcp_YliG7PP5ed","scope":{"id":"p_1234567890","type":"project","name":"Generated project scope","description":"Provides an initial project scope in Boundary","parent_scope_id":"o_1234567890"},"created_time":"2021-09-01T02:09:21.175544Z","user_id":"u_1234567890","host_set_id":"hsst_1234567890","host_id":"hst_1234567890","type":"tcp","authorization_token":"4qJdt9XHu5XwhegB2RcEFpfVJxPnV73CZrfkReA5F7HJ2NptkPX9K73yYjYbULTys5gsw8Vg3HMt9HceJaofQjreEvBvgMQ8oDwDPjmc1rKeLW3XA6kHwcwRnaCHDqPr1cdqfzZ7iBomPbyEF9yokdxPvYpAiMJ5yrhDt4HvhJjFXBe5nLeDDfZJmGFUSbqb2SRs4KW6S3nkdtVtndPeAhUfhW1MLVWQp8hucx5htzQxRdPWnL7j8VQbFK6kRTM9VWxuAi2kvdSFoEDnAHAA2JsvAvCGok3fmZ1pjgGzm4zpzJmzT6EvkreWSDxnpMa3SMihpuBVBzaRseCrfUkypP1Xd1jQyDzyjH8b9kpTRt8P8dxRC68qyCbVV7CdemCezAkRkZ1oXLGfsPSocGUiyTDzhPqSQzfTMDJgW7oabyG7pS5xst5mEpLNwCk3d9TL54gbjwK8L35BGrU9vEDmsDgJYVzc5vetdQKzSrwJRW6AbTjsXgDGBT2UXhRqbcim1cdPv3nTi2RfsSyAneQT1HTkCCcnqmQX2m2LLodfJJU5bUF3XYbytJdpdjmhKQbGbcy5Z46bAagyPNuMpevpK4FNW1qgFtAaxdc1dSEG6dZK2jXjzTvdAcvvLD3FcVwKzWpgP85UAmwqrwaNRBkxayDpLAGZftQcyuzzZws7ezhCKXC5rX4fL6HS7TsUezPS7bgJXixHhvN5CxUPPXA2Q2Vhc5i6LRTbpr8GXw8SJt9AngGkfvkpAmGz2EpMTnoVAeRmBvnfbDG1MjxpQroHjD5Sx66qxSwxr31DYty3HBmooEMiLJvyEmE3JLtaqNdVtLTradJBqW1","endpoint":"tcp://localhost:16001","credentials":[{"credential_library":{"id":"clvlt_scclxN7LMc","name":"northwind dba","credential_store_id":"csvlt_cvOG2JlUcp","type":"vault"},"secret":{"raw":"eyJwYXNzd29yZCI6ImpGbDJ2NXpFcXd3dy1FdHlzYW02IiwidXNlcm5hbWUiOiJ2LXRva2VuLWRiYS0ydXBrM1l3MXlrTmViYTd5Q1BxOC0xNjMwNDYyMTYxIn0=","decoded":{"password":"jFl2v5zEqwww-Etysam6","username":"v-token-dba-2upk3Yw1ykNeba7yCPq8-1630462161"}}}]}}

Also @dcardozoo in your previous example where you were hitting a 403 I see the URL being used is http://192.168.33.13:8200/v1/v1/ssh/creds/onetime note v1 appears twice which is likely the issue.

[dcardozoo]

I have a PR up that should fix this issue, but in the mean time you can include the - format json flag, which will bypass the code that is currently hitting this panic. json is likely a better way to consume the output of authorize-session

Thank you!

Also @dcardozoo in your previous example where you were hitting a 403 I see the URL being used is http://192.168.33.13:8200/v1/v1/ssh/creds/onetime note v1 appears twice which is likely the issue.

Thanks! yes I was aware of that, I just used the wrong path to check what would happen :)

[akhila98]

Trying to login to boundary from vault ,could anyone help me out here

boundary targets authorize-session -id ttcp_1234567890 -addr=http://172.26.5.104:9200

Error from controller when performing authorize-session on a session against target

Error information:
Kind: Internal
Message: targets.(Service).AuthorizeSession: vault.(Repository).Issue: vault.(client).get: vault: #3014: Error
making API request.

Code: 403. Errors:

• 1 error occurred:
• permission denied

Status: 500
context: Error from controller when performing authorize-session on a session against target