Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sshforward: implement ssh socket forwarding #608

Merged
merged 4 commits into from Sep 11, 2018

Conversation

Projects
None yet
3 participants
@tonistiigi
Copy link
Member

tonistiigi commented Sep 6, 2018

Add ssh agent socket forwarding support through a SSH mount type in LLB. Optional ID can be assigned on a mount to support multiple sockets.

buildctl build --ssh default
buildctl build --ssh default=$SSH_AUTH_SOCK // same as above
buildctl build --ssh default=key1,key2
buildctl build --ssh myapp.ssh=key1

@AkihiroSuda @tiborvass

Follow-up: expose in Dockerfile, expose to git sources

@tonistiigi tonistiigi force-pushed the tonistiigi:ssh-forwarding branch from 2618dcd to 56ef9fc Sep 6, 2018


service SSH {
rpc CheckAgent(CheckAgentRequest) returns (CheckAgentResponse);
rpc ForwardAgent(stream BytesMessage) returns (stream BytesMessage);

This comment has been minimized.

Copy link
@AkihiroSuda

AkihiroSuda Sep 7, 2018

Member

nit: indent

@tonistiigi tonistiigi force-pushed the tonistiigi:ssh-forwarding branch 2 times, most recently from 5a557bd to 1b7f365 Sep 7, 2018

// DefaultID is the default ssh ID
const DefaultID = "default"

const KeySSHID = "buildkitd.ssh.id"

This comment has been minimized.

Copy link
@AkihiroSuda

AkihiroSuda Sep 7, 2018

Member

buildkitd -> buildkit?

return nil, err
}
if conf.ID == "" {
conf.ID = "default"

This comment has been minimized.

Copy link
@AkihiroSuda

tonistiigi added some commits Sep 4, 2018

sshforward: implement ssh socket forwarding
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
client: add ssh socket test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

@tonistiigi tonistiigi force-pushed the tonistiigi:ssh-forwarding branch from 1b7f365 to 68502db Sep 7, 2018

}

s := &server{l: l}
go s.run(agent)

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

the error is ignored

This comment has been minimized.

Copy link
@tonistiigi

tonistiigi Sep 10, 2018

Author Member

An error on a single connection shouldn't fail the build. If it causes process to exit then that will become the error.

})
}

var SSHOptional = sshOptionFunc(func(si *SSHInfo) {

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

why make this writable?

This comment has been minimized.

Copy link
@tonistiigi

tonistiigi Sep 10, 2018

Author Member

?

func NewSSHAgentProvider(confs []AgentConfig) (session.Attachable, error) {
m := map[string]source{}
for _, conf := range confs {
if len(conf.Paths) == 0 || len(conf.Paths) == 1 && conf.Paths[0] == "" {

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

why have the OR condition if you error out for the same condition further below? nevermind I understand now.

}
fi, err := os.Stat(p)
if err != nil {
return source{}, errors.Wrapf(err, "failed to stat %s", p)

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

unnecessary wrapping, PathError already has similar message such as stat /my/path: No such file or directory

return errors.Errorf("removing keys not allowed by buildkit")
}

func (a *readOnlyAgent) Lock(_ []byte) error {

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

Might as well protect Unlock too, no?

This comment has been minimized.

Copy link
@tonistiigi

tonistiigi Sep 10, 2018

Author Member

It is hard to unlock unexpectedly to the user, what the other cases are protecting from.

id = DefaultID
}

go s.run(ctx, l, id)

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

error is ignored

return err
}

go Copy(ctx, conn, stream)

This comment has been minimized.

Copy link
@tiborvass

tiborvass Sep 10, 2018

Collaborator

error is ignored

tonistiigi added some commits Sep 6, 2018

sshprovider: allow keys from local files
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
vendor: add x/crypto
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

@tonistiigi tonistiigi force-pushed the tonistiigi:ssh-forwarding branch from 68502db to 1604b1b Sep 10, 2018

@tiborvass
Copy link
Collaborator

tiborvass left a comment

LGTM

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Sep 11, 2018

Can we merge?

@tonistiigi tonistiigi merged commit 1508ae0 into moby:master Sep 11, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Oct 1, 2018

@tonistiigi

Any ETA of Dockerfile frontend for this? ^^

@tonistiigi

This comment has been minimized.

Copy link
Member Author

tonistiigi commented Oct 1, 2018

@AkihiroSuda I'll work on the release/test stages for the experimental dockerfiles in next days so we can run them with travis and can probably do it after that. If you have cycles feel free to submit it yourself. The UI should be the same as the secrets with extra mount type and should automatically set up the environment variable if it is unset.

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Oct 2, 2018

AkihiroSuda added a commit to AkihiroSuda/cli that referenced this pull request Oct 5, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>

AkihiroSuda added a commit to AkihiroSuda/cli that referenced this pull request Oct 5, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>

docker-jenkins pushed a commit to docker/docker-ce that referenced this pull request Oct 10, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Upstream-commit: db7399a016bed833205a17129ed80fad4d15e48d
Component: cli

thaJeztah added a commit to thaJeztah/cli that referenced this pull request Oct 11, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit db7399a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah added a commit to thaJeztah/cli that referenced this pull request Oct 11, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit db7399a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah added a commit to thaJeztah/cli that referenced this pull request Oct 11, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit db7399a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

lifubang added a commit to lifubang/cli that referenced this pull request Oct 12, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>

docker-jenkins pushed a commit to docker/docker-ce that referenced this pull request Oct 12, 2018

build: add SSH agent socket forwarder (`docker build --ssh $SSHMOUNTI…
…D=$SSH_AUTH_SOCK`)

Unlike `docker build --secret`, `docker build --ssh` allows the build container to
use SSH keys with passphrases.

  $ eval $(ssh-agent)
  $ ssh-add ~/.ssh/id_rsa
  (Input your passphrase here)
  $ docker build --ssh default=$SSH_AUTH_SOCK ...

This feature requires the daemon with `CapExecMountSSH` build capability (moby/moby#37973) .

Currently, the official Dockerfile frontend does not provide the syntax for using the SSH forwarder.

However, the experimental `RUN --mount=type=ssh` syntax can be enabled by using
the Dockerfile frontend image built with the `BUILDTAGS="dfrunmount dfssh"`, via the `# syntax =` "shebang".

The Dockerfile for the Dockerfile frontend is available at  github.com/moby/buildkit/frontend/dockerfile/cmd/dockerfile-frontend)
The pre-built image is also available as `tonistiigi/dockerfile:ssh20181002` .

An example Dockerfile with `RUN --mount=type=ssh`:

  # syntax = tonistiigi/dockerfile:ssh20181002
  FROM alpine
  RUN apk add --no-cache openssh-client
  RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  RUN --mount=type=ssh ssh git@gitlab.com | tee /hello
  # "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here

More info available at moby/buildkit#608, moby/buildkit#655

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
(cherry picked from commit db7399a016bed833205a17129ed80fad4d15e48d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: e942084530002e5e02466b3f5941f0dc0136675e
Component: cli
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.