Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vendor gazelle and kazel #57600

Merged
merged 4 commits into from
Apr 2, 2018
Merged

Vendor gazelle and kazel #57600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3e583de
Vendor gazelle
thockin Dec 23, 2017
74451de
Vendor kazel
thockin Dec 23, 2017
f72484d
Remove kube::util::go_install_from_commit
thockin Dec 23, 2017
4a5e554
Don't change GOPATH or PATH in a script lib
thockin Feb 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 73 additions & 7 deletions Godeps/Godeps.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2,722 changes: 2,606 additions & 116 deletions Godeps/LICENSES
View file
Open in desktop

Large diffs are not rendered by default.

18 changes: 0 additions & 18 deletions api/BUILD
View file
Open in desktop

This file was deleted.

3 changes: 2 additions & 1 deletion build/root/BUILD.root
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ filegroup(
name = "all-srcs",
srcs = [
":package-srcs",
"//api:all-srcs",
"//api/openapi-spec:all-srcs",
"//api/swagger-spec:all-srcs",
"//build:all-srcs",
"//cluster:all-srcs",
"//cmd:all-srcs",
Expand Down
2 changes: 2 additions & 0 deletions hack/godep-save.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ REQUIRED_BINS=(
"github.com/jteeuwen/go-bindata/go-bindata"
"github.com/tools/godep"
"github.com/client9/misspell/cmd/misspell"
"github.com/bazelbuild/bazel-gazelle/cmd/gazelle"
"github.com/kubernetes/repo-infra/kazel"
"./..."
)

Expand Down
32 changes: 11 additions & 21 deletions hack/lib/util.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,12 +435,19 @@ kube::util::ensure_godep_version() {

kube::log::status "Installing godep version ${GODEP_VERSION}"
go install ./vendor/github.com/tools/godep/
GP="$(echo $GOPATH | cut -f1 -d:)"
hash -r # force bash to clear PATH cache
PATH="${GP}/bin:${PATH}"
if ! which godep >/dev/null 2>&1; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like that there is inconsistency between the way we handle godep, and the way we are handling gazelle/kazel/misspell.

It's not enough of a dislike to block this PR that we badly need, but I'd really like if we could unify on a single way (maybe via a helper util function) to install a specific version of a go app from vendor, use it in the context of the script, with as little pollution of the environment as possible. Maybe I'll look at this in a follow up PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could set GOBIN in init.sh or something, and then always call $GOBIN/godep

But as long as we're checking the version of godep, it shouldn't matter. I agree it would be cleaner to be consistent. Our scripts are a mess here.

function gobin() {
  echo "${KUBE_OUTPUT_BINPATH}"
}

function go_install() {
  GOBIN="$(gobin)" go install "$@"
}

function run_private() {
  PATH="$(gobin):${PATH}" "$@"
}

function run_global() {
  "$@"
}


go_install ./vendor/github.com/tools/godep/
run_private godep save

alternatively, make teh callers of ensure_godep_version pass the path to the godep they are using, and they can pass ${GOBIN}/godep ?

Agree it should be a followup

kube::log::error "Can't find godep - is your GOPATH 'bin' in your PATH?"
kube::log::error " GOPATH: ${GOPATH}"
kube::log::error " PATH: ${PATH}"
return 1
fi

if [[ "$(godep version 2>/dev/null)" != *"godep ${GODEP_VERSION}"* ]]; then
kube::log::error "Expected godep ${GODEP_VERSION}, got $(godep version)"
kube::log::error "Wrong godep version - is your GOPATH 'bin' in your PATH?"
kube::log::error " expected: godep ${GODEP_VERSION}"
kube::log::error " got: $(godep version)"
kube::log::error " GOPATH: ${GOPATH}"
kube::log::error " PATH: ${PATH}"
return 1
fi
}
Expand All @@ -466,23 +473,6 @@ kube::util::ensure_no_staging_repos_in_gopath() {
fi
}

# Installs the specified go package at a particular commit.
kube::util::go_install_from_commit() {
local -r pkg=$1
local -r commit=$2

kube::util::ensure-temp-dir
mkdir -p "${KUBE_TEMP}/go/src"
GOPATH="${KUBE_TEMP}/go" go get -d -u "${pkg}"
(
cd "${KUBE_TEMP}/go/src/${pkg}"
git checkout -q "${commit}"
GOPATH="${KUBE_TEMP}/go" go install "${pkg}"
)
PATH="${KUBE_TEMP}/go/bin:${PATH}"
hash -r # force bash to clear PATH cache
}

# Checks that the GOPATH is simple, i.e. consists only of one directory, not multiple.
kube::util::ensure_single_dir_gopath() {
if [[ "${GOPATH}" == *:* ]]; then
Expand Down
15 changes: 8 additions & 7 deletions hack/update-bazel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,14 @@ kube::util::ensure-gnu-sed
# TODO(spxtr): Remove this line once Bazel is the only way to build.
rm -f "${KUBE_ROOT}/pkg/generated/openapi/zz_generated.openapi.go"

# The git commit sha1s here should match the values in $KUBE_ROOT/WORKSPACE.
kube::util::go_install_from_commit \
github.com/kubernetes/repo-infra/kazel \
97099dccc8807e9159dc28f374a8f0602cab07e1
kube::util::go_install_from_commit \
github.com/bazelbuild/bazel-gazelle/cmd/gazelle \
0.10.1
# Ensure that we find the binaries we build before anything else.
export GOBIN="${KUBE_OUTPUT_BINPATH}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to export this? I don't like modifying a user's environment beyond the life of the script. There's a couple other cases that you're doing this too, so I'm not sure if it's strictly a requirement.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export exists for the life of the shell, not longer. Maybe I misunderstand. I did not put it in a generic place that gets sourced, for this reason.

$ echo $FOO; (export FOO=foo; echo $FOO; (export FOO=bar; echo $FOO); echo $FOO); echo $FOO

foo
bar
foo

PATH="${GOBIN}:${PATH}"

# Install tools we need, but only from vendor/...
go install ./vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle

go install ./vendor/github.com/kubernetes/repo-infra/kazel

touch "${KUBE_ROOT}/vendor/BUILD"

Expand Down
4 changes: 2 additions & 2 deletions hack/verify-bazel.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ mkdir -p "${_tmp_kuberoot}/.."
cp -a "${KUBE_ROOT}" "${_tmp_kuberoot}/.."

cd "${_tmp_kuberoot}"
GOPATH="${_tmp_gopath}" ./hack/update-bazel.sh
GOPATH="${_tmp_gopath}" PATH="${_tmp_gopath}/bin:${PATH}" ./hack/update-bazel.sh

diff=$(diff -Naupr "${KUBE_ROOT}" "${_tmp_kuberoot}" || true)
diff=$(diff -Naupr -x '_output' "${KUBE_ROOT}" "${_tmp_kuberoot}" || true)

if [[ -n "${diff}" ]]; then
echo "${diff}" >&2
Expand Down
1 change: 1 addition & 0 deletions hack/verify-godeps.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ _kubetmp="${_kubetmp}/kubernetes"

# Do all our work in the new GOPATH
export GOPATH="${_tmpdir}"
export PATH="${GOPATH}/bin:${PATH}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You had raised concerns about doing stuff like this here: #59048 (comment)

I'm wondering if a method like #59210 would be better. This approach uses the run-in-gopath method so that the utility is built hermetically, and doesn't A) overwrite a version of the tool already existing in the user's environment or B) pollute their $GOPATH/pkg

Let me know what you think!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue I took was changing global variables deep inside a function.

Changing them in a first-level script is OK.

Basically, any place that changes GOPATH should also change PATH to frontload the new GOPATH/bin

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a commit here that seems simple - PTAL?


pushd "${_kubetmp}" > /dev/null 2>&1
# Restore the Godeps into our temp directory
Expand Down
11 changes: 4 additions & 7 deletions hack/verify-spelling.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,12 @@ set -o pipefail
export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
source "${KUBE_ROOT}/hack/lib/init.sh"

# Ensure that we find the binaries we build before anything else.
export GOBIN="${KUBE_OUTPUT_BINPATH}"
PATH="${GOBIN}:${PATH}"

# Install tools we need, but only from vendor/...
cd ${KUBE_ROOT}
go install ./vendor/github.com/client9/misspell/cmd/misspell
if ! which misspell >/dev/null 2>&1; then
echo "Can't find misspell - is your GOPATH 'bin' in your PATH?" >&2
echo " GOPATH: ${GOPATH}" >&2
echo " PATH: ${PATH}" >&2
exit 1
fi

# Spell checking
# All the skipping files are defined in hack/.spelling_failures
Expand Down
1 change: 1 addition & 0 deletions pkg/generated/openapi/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ openapi_library(
"pkg/kubelet/apis/kubeletconfig/v1beta1",
"pkg/proxy/apis/kubeproxyconfig/v1alpha1",
"pkg/version",
"vendor/github.com/kubernetes/repo-infra/kazel",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

welp, the openapi generator in kazel is picking up itself, erroneously. I'll fix this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

],
tags = ["automanaged"],
vendor_prefix = openapi_vendor_prefix,
Expand Down
15 changes: 14 additions & 1 deletion vendor/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,19 @@ filegroup(
"//vendor/github.com/aws/aws-sdk-go/service/elbv2:all-srcs",
"//vendor/github.com/aws/aws-sdk-go/service/kms:all-srcs",
"//vendor/github.com/aws/aws-sdk-go/service/sts:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/config:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/label:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/merger:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/packages:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/pathtools:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/repos:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/resolve:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/rules:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/version:all-srcs",
"//vendor/github.com/bazelbuild/bazel-gazelle/internal/wspace:all-srcs",
"//vendor/github.com/bazelbuild/buildtools/build:all-srcs",
"//vendor/github.com/bazelbuild/buildtools/tables:all-srcs",
"//vendor/github.com/beorn7/perks/quantile:all-srcs",
"//vendor/github.com/blang/semver:all-srcs",
"//vendor/github.com/chai2010/gettext-go/gettext:all-srcs",
Expand Down Expand Up @@ -275,6 +288,7 @@ filegroup(
"//vendor/github.com/kr/pretty:all-srcs",
"//vendor/github.com/kr/pty:all-srcs",
"//vendor/github.com/kr/text:all-srcs",
"//vendor/github.com/kubernetes/repo-infra/kazel:all-srcs",
"//vendor/github.com/libopenstorage/openstorage/api:all-srcs",
"//vendor/github.com/libopenstorage/openstorage/pkg/parser:all-srcs",
"//vendor/github.com/libopenstorage/openstorage/pkg/units:all-srcs",
Expand Down Expand Up @@ -304,7 +318,6 @@ filegroup(
"//vendor/github.com/opencontainers/runtime-spec/specs-go:all-srcs",
"//vendor/github.com/opencontainers/selinux/go-selinux:all-srcs",
"//vendor/github.com/pborman/uuid:all-srcs",
"//vendor/github.com/pelletier/go-buffruneio:all-srcs",
"//vendor/github.com/pelletier/go-toml:all-srcs",
"//vendor/github.com/peterbourgon/diskv:all-srcs",
"//vendor/github.com/pkg/errors:all-srcs",
Expand Down
18 changes: 18 additions & 0 deletions vendor/github.com/bazelbuild/bazel-gazelle/AUTHORS
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions vendor/github.com/bazelbuild/bazel-gazelle/CONTRIBUTORS
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading