Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mounting kubectl from the host instead to installing in protokube #3550

Merged
merged 1 commit into from Oct 10, 2017
Merged

mounting kubectl from the host instead to installing in protokube #3550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 0 additions & 5 deletions images/protokube-builder/onbuild.sh
View file
Open in desktop
Expand Up @@ -32,9 +32,4 @@ cp /src/.build/local/protokube /src/.build/artifacts/
make channels
cp /src/.build/local/channels /src/.build/artifacts/

# channels uses protokube
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops - that comment wasn't right :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

cd /src/.build/artifacts/
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.6.6/bin/linux/amd64/kubectl
chmod +x kubectl

chown -R $HOST_UID:$HOST_GID /src/.build/artifacts
2 changes: 0 additions & 2 deletions images/protokube/Dockerfile
View file
Open in desktop
Expand Up @@ -21,8 +21,6 @@ RUN apt-get update && apt-get install --yes \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

COPY /.build/artifacts/kubectl /usr/bin/kubectl

COPY /.build/artifacts/protokube /usr/bin/protokube
COPY /.build/artifacts/channels /usr/bin/channels

Expand Down
12 changes: 12 additions & 0 deletions nodeup/pkg/model/context.go
View file
Open in desktop
Expand Up @@ -246,3 +246,15 @@ func (c *NodeupModelContext) UseSecureKubelet() bool {

return false
}

// KubectlPath returns distro based path for kubectl
func (c *NodeupModelContext) KubectlPath() string {
kubeletCommand := "/usr/local/bin"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename kubeletComamnd to kubectlPath

if c.Distribution == distros.DistributionCoreOS {
kubeletCommand = "/opt/bin"
}
if c.Distribution == distros.DistributionContainerOS {
kubeletCommand = "/home/kubernetes/bin"
}
return kubeletCommand
}
13 changes: 1 addition & 12 deletions nodeup/pkg/model/kubectl.go
View file
Open in desktop
Expand Up @@ -52,7 +52,7 @@ func (b *KubectlBuilder) Build(c *fi.ModelBuilderContext) error {
}

t := &nodetasks.File{
Path: b.kubectlPath(),
Path: b.KubectlPath() + "/" + assetName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use filepath.Join

Contents: asset,
Type: nodetasks.FileType_File,
Mode: s("0755"),
Expand Down Expand Up @@ -100,14 +100,3 @@ func (b *KubectlBuilder) Build(c *fi.ModelBuilderContext) error {

return nil
}

func (b *KubectlBuilder) kubectlPath() string {
kubeletCommand := "/usr/local/bin/kubectl"
if b.Distribution == distros.DistributionCoreOS {
kubeletCommand = "/opt/bin/kubectl"
}
if b.Distribution == distros.DistributionContainerOS {
kubeletCommand = "/home/kubernetes/bin/kubectl"
}
return kubeletCommand
}
19 changes: 17 additions & 2 deletions nodeup/pkg/model/protokube.go
View file
Open in desktop
Expand Up @@ -108,12 +108,27 @@ func (t *ProtokubeBuilder) buildSystemdService() (*nodetasks.Service, error) {
"-v", "/:/rootfs/",
"-v", "/var/run/dbus:/var/run/dbus",
"-v", "/run/systemd:/run/systemd",
"--net=host", "--privileged",
}

// add kubectl only if a master
// path changes depending on distro, and always mount it on /opt/kops/bin
// kubectl is downloaded an installed by other tasks
if t.IsMaster {
dockerArgs = append(dockerArgs, []string{
"-v", t.KubectlPath() + ":/opt/kops/bin:ro",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So docker can actually bind mount a file, which is nice because it avoids dragging in everything in the bin directory. Not sure whether I care, given we mount / -> /rootfs above...

A potential gotcha is that we have to be careful about shared libraries, but thankfully kubectl is statically linked.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but we have a chance that protokube will start before kubectl is installed ... async tasks ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point, and then we would hit the docker bug where it assumes a directory.

The tasks are technically ordered if we tell nodeup about our dependencies, but I do like your way better.

"--env", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/kops/bin",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Probably better to put /opt/kops/bin at the start, except that we know that most of these dirs don't exist because we're in a container...

I guess the problem here is that channels calls kubectl, so we have to make sure it's on the path? If that's the case, can we have a comment, as otherwise the temptation will be to just change kubectl calls to be a fully-qualified path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well ... I disagree. We need it to be at the end, so that other stuff does not get whacked. First, wins in bash?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK - we don't expect many copies of kubectl :-)

Can we have a comment though about why we don't just call kubectl with an absolute path, i.e. why we are messing with the PATH at all

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause we do not want to make changes to channels :P Yes I will make a comment.

}...)
}

dockerArgs = append(dockerArgs, []string{
"--net=host",
"--privileged",
"--env", "KUBECONFIG=/rootfs/var/lib/kops/kubeconfig",
t.ProtokubeEnvironmentVariables(),
t.ProtokubeImageName(),
"/usr/bin/protokube",
}
}...)

protokubeCommand := strings.Join(dockerArgs, " ") + " " + protokubeFlagsArgs

manifest := &systemd.Manifest{}
Expand Down