Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate protobuf-related files in Docker #22738

Merged
merged 1 commit into from
Mar 15, 2016
Merged

Generate protobuf-related files in Docker #22738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 0 additions & 1 deletion build/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ readonly DOCKER_MOUNT_ARGS_BASE=(
--volume "${OUTPUT_BINPATH}:${REMOTE_OUTPUT_BINPATH}"
--volume /etc/localtime:/etc/localtime:ro
)
# DOCKER_MOUNT_ARGS=("${DOCKER_MOUNT_ARGS_BASE[@]}" --volumes-from "${KUBE_BUILD_DATA_CONTAINER_NAME}")

# We create a Docker data container to cache incremental build artifacts. We
# need to cache both the go tree in _output and the go tree under Godeps.
Expand Down
24 changes: 24 additions & 0 deletions cmd/libs/go2idl/go-to-protobuf/build-image/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Copyright 2016 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This file creates a standard build environment for building Kubernetes
FROM gcr.io/google_containers/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG

# Mark this as a kube-build container
RUN touch /kube-build-image

WORKDIR /go/src/k8s.io/kubernetes

# Install goimports tool
RUN go get golang.org/x/tools/cmd/goimports
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't we just install this in kube-cross. Why here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can, but I tried to find a way to use that image to do what we want and didn't manage to do it.
And since we need another image anyway, why installing it there...

9 changes: 6 additions & 3 deletions hack/after-build/update-generated-protobuf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,17 @@ set -o pipefail
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
source "${KUBE_ROOT}/hack/lib/init.sh"

# TODO: preinstall protobuf
kube::golang::setup_env

hack/build-go.sh cmd/libs/go2idl/go-to-protobuf cmd/libs/go2idl/go-to-protobuf/protoc-gen-gogo
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not exactly how I would like this to look like, but when I build those binaries in the:
hack/update-generated-protobuf.sh
gotoprotobuf just doesn't work - I'm getting errors like these:

type checker error: k8s.io/kubernetes/pkg/api/resource/quantity.go:20:2: could not import errors (unable to *find*     "errors": cannot find package "errors" in any of:
    /usr/lib/google-golang/src/errors (from $GOROOT)
    /go/src/k8s.io/kubernetes/_output/local/go/src/errors (from $GOPATH)
    /go/src/k8s.io/kubernetes/Godeps/_workspace/src/errors)

If you have any suggestions how to fix it - I'm happy to change it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct to me.


if [[ -z "$(which protoc)" || "$(protoc --version)" != "libprotoc 3.0."* ]]; then
echo "Generating protobuf requires protoc 3.0.0-beta1 or newer. Please download and"
echo "install the platform appropriate Protobuf package for your OS: "
echo
echo " https://github.com/google/protobuf/releases"
echo
echo "WARNING: Protobuf changes are not being validated"
# TODO: make error when protobuf is installed
exit 0
fi

Expand All @@ -38,7 +40,8 @@ gotoprotobuf=$(kube::util::find-binary "go-to-protobuf")
# requires the 'proto' tag to build (will remove when ready)
# searches for the protoc-gen-gogo extension in the output directory
# satisfies import of github.com/gogo/protobuf/gogoproto/gogo.proto and the core Google protobuf types
PATH="${KUBE_ROOT}/_output/local/go/bin:${PATH}" "${gotoprotobuf}" \
PATH="${KUBE_ROOT}/_output/local/go/bin:${PATH}" \
"${gotoprotobuf}" \
--conditional="proto" \
--proto-import="${KUBE_ROOT}/Godeps/_workspace/src" \
--proto-import="${KUBE_ROOT}/third_party/protobuf"
30 changes: 27 additions & 3 deletions hack/update-generated-protobuf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,36 @@ set -o nounset
set -o pipefail

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
source "${KUBE_ROOT}/hack/lib/init.sh"
source "${KUBE_ROOT}/build/common.sh"

kube::golang::setup_env

"${KUBE_ROOT}/hack/build-go.sh" cmd/libs/go2idl/go-to-protobuf cmd/libs/go2idl/go-to-protobuf/protoc-gen-gogo
function prereqs() {
kube::log::status "Verifying Prerequisites...."
kube::build::ensure_docker_in_path || return 1
if kube::build::is_osx; then
kube::build::docker_available_on_osx || return 1
fi
kube::build::ensure_docker_daemon_connectivity || return 1

"${KUBE_ROOT}/hack/after-build/update-generated-protobuf.sh" "$@"
KUBE_ROOT_HASH=$(kube::build::short_hash "$KUBE_ROOT/go-to-protobuf")
KUBE_BUILD_IMAGE_TAG="build-${KUBE_ROOT_HASH}"
KUBE_BUILD_IMAGE="${KUBE_BUILD_IMAGE_REPO}:${KUBE_BUILD_IMAGE_TAG}"
KUBE_BUILD_CONTAINER_NAME="kube-build-${KUBE_ROOT_HASH}"
KUBE_BUILD_DATA_CONTAINER_NAME="kube-build-data-${KUBE_ROOT_HASH}"
DOCKER_MOUNT_ARGS=(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need these if we're doing a docker build?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically I want to reuse the existing docker_build commands (it's already tested, works on different platforms etc.) And this requires all of those to have this working.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem with our docker builds (not your problem to fix here) is that builds don't work from a local env to a remote docker vm. The volumes approach limits us - it would be better to do a pure docker build from the client than to do mounting. I'll open a separate issue for that.

--volume "${KUBE_ROOT}:/go/src/${KUBE_GO_PACKAGE}"
--volume /etc/localtime:/etc/localtime:ro
--volumes-from "${KUBE_BUILD_DATA_CONTAINER_NAME}"
)
LOCAL_OUTPUT_BUILD_CONTEXT="${LOCAL_OUTPUT_IMAGE_STAGING}/${KUBE_BUILD_IMAGE}"
}

prereqs
mkdir -p "${LOCAL_OUTPUT_BUILD_CONTEXT}"
cp "${KUBE_ROOT}/cmd/libs/go2idl/go-to-protobuf/build-image/Dockerfile" "${LOCAL_OUTPUT_BUILD_CONTEXT}/Dockerfile"
kube::build::update_dockerfile
kube::build::docker_build "${KUBE_BUILD_IMAGE}" "${LOCAL_OUTPUT_BUILD_CONTEXT}" 'false'
kube::build::run_build_command hack/after-build/update-generated-protobuf.sh "$@"

# ex: ts=2 sw=2 et filetype=sh