build/release.sh (hack/lib/golang.sh) requires minimum bash v4.4 #77521

cjschaef · 2019-05-06T21:47:45Z

What happened: building Kubernetes release and it failed due to a variable not being set (with set -u enabled). It appears a recent commit does not function as expected on bash v4.3 (used Ubuntu 16). Additional details will be provided in comment below

kubernetes/hack/lib/golang.sh: line 295: KUBE_SERVER_PLATFORMS[@]: unbound variable

What you expected to happen: successful build of Kuberentes release or documented minimum supported OS releases (i.e., bash versions)

How to reproduce it (as minimally and precisely as possible): using Ubuntu 16 (bash v4.3), run [build/release.sh] using release-1.14, v1.15.0-alpha.2 or master (containing dcee810 or subsequent cherry pick)

Anything else we need to know?: It appears the issue is with how bash v4.3 and bash v4.4 create readonly variables, where I believe v4.3 does not create the variables with the expected scope (created using local scope versus global scope). This creates a minimum supported OS version when running build/release.sh from the best I can tell.

Environment:

Kubernetes version (use kubectl version): v1.15.0-alpha.2 release-1.14
Cloud provider or hardware configuration: N/A
OS (e.g: cat /etc/os-release):

release 
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a): Linux armada-lb-build 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools: build/release.sh
Network plugin and version (if this is a network-related bug): N/A
Others:

/sig release

The text was updated successfully, but these errors were encountered:

cjschaef · 2019-05-06T21:52:22Z

Some of the details I mentioned above, modified the hack/lib/golang.sh file to output some helpful information

the diff of changes I made
https://pastebin.com/zrU28PRQ
the results when running on Ubuntu 16 (bash 4.3), the variable KUBE_SERVER_PLATFORMS appears to be locally scoped inside the function kube::golang::setup_platforms
https://pastebin.com/ph1648Ui
the results when running on Ubuntu 16 (bash 4.3) without the readonly variable attribute (you can now see the variable is a global variable, not locally scoped to the function kube::golang::setup_platforms)
https://pastebin.com/yfhXmANC

And on Ubuntu 18 (bash 4.4) the variable is scoped globally with the readonly attribute as you would expect and build/release.sh runs as expected.

nikhita · 2019-05-07T11:23:27Z

I guess #77362 fixes this problem.

cc @cblecker

cjschaef · 2019-05-07T14:38:51Z

I don't think the mapfile is/was the only issue here. It was obviously readonly as I was testing with KUBE_FASTBUILD=true, so only readonly variables were being used and not scoped outside the function (in bash 4.3).

kubernetes/hack/lib/golang.sh

Lines 204 to 220 in eaafa59

    
           elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then 
        
             readonly KUBE_SERVER_PLATFORMS=(linux/amd64) 
        
             readonly KUBE_NODE_PLATFORMS=(linux/amd64) 
        
             if [[ "${KUBE_BUILDER_OS:-}" == "darwin"* ]]; then 
        
               readonly KUBE_TEST_PLATFORMS=( 
        
                 darwin/amd64 
        
                 linux/amd64 
        
               ) 
        
               readonly KUBE_CLIENT_PLATFORMS=( 
        
                 darwin/amd64 
        
                 linux/amd64 
        
               ) 
        
             else 
        
               readonly KUBE_TEST_PLATFORMS=(linux/amd64) 
        
               readonly KUBE_CLIENT_PLATFORMS=(linux/amd64) 
        
             fi 
        
           else

adisky · 2019-05-08T05:21:48Z

@cjschaef we are also getting the same error on bash version 4.3
cc @h00130372 @mrhillsman

adisky · 2019-05-13T05:35:02Z

@cjschaef any update on this, are you fixing it??

kiwik · 2019-05-14T03:39:51Z

Same question with @adisky , this issue impact OpenLab cloud-provider-openstack E2E testing job.

As per the issue kubernetes/kubernetes#77521 (comment), issue seems to be happen only when KUBE_FASTBUILD=true. Changing the same to check if it fixes the e2e job.

dims · 2019-05-14T13:43:05Z

@kiwik @adisky can you please try this? #77867

aojea · 2019-05-14T14:20:31Z

@dims it works with your fix https://circleci.com/gh/aojea/kubernetes/53 ,
previously jobs were failing https://circleci.com/gh/aojea/kubernetes/50
--- EDIT ---
Hold on, I think I was too fast, it failed in a later stage don't know if is a kind problem or related to the script changes. Should we have the tar files as an output ?

circleci@default-c236894c-8a28-48dc-9670-a8a3a15b6682:~/go/src/k8s.io/kubernetes$ export KUBE_FASTBUILD=true
circleci@default-c236894c-8a28-48dc-9670-a8a3a15b6682:~/go/src/k8s.io/kubernetes$ make quick-release-images
+++ [0514 14:27:17] Verifying Prerequisites....
+++ [0514 14:27:18] Building Docker image kube-build:build-49a05260c2-5-v1.12.5-1
+++ [0514 14:27:20] Syncing sources to container
+++ [0514 14:27:22] Running build command...
+++ [0514 14:27:54] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0514 14:27:55] Building go targets for linux/amd64:
    cmd/cloud-controller-manager
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kube-scheduler
    cmd/kube-proxy
    cmd/hyperkube
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubectl
+++ [0514 14:37:18] Syncing out of container
circleci@default-c236894c-8a28-48dc-9670-a8a3a15b6682:~/go/src/k8s.io/kubernetes$ find _output/
_output/
_output/images
_output/images/kube-build:build-49a05260c2-5-v1.12.5-1
_output/images/kube-build:build-49a05260c2-5-v1.12.5-1/localtime
_output/images/kube-build:build-49a05260c2-5-v1.12.5-1/rsyncd.password
_output/images/kube-build:build-49a05260c2-5-v1.12.5-1/rsyncd.sh
_output/images/kube-build:build-49a05260c2-5-v1.12.5-1/Dockerfile
_output/dockerized
_output/dockerized/bin
_output/dockerized/bin/linux
_output/dockerized/bin/linux/amd64
_output/dockerized/bin/linux/amd64/defaulter-gen
_output/dockerized/bin/linux/amd64/openapi-gen
_output/dockerized/bin/linux/amd64/kubectl
_output/dockerized/bin/linux/amd64/cloud-controller-manager
_output/dockerized/bin/linux/amd64/go-bindata
_output/dockerized/bin/linux/amd64/ginkgo
_output/dockerized/bin/linux/amd64/kube-proxy
_output/dockerized/bin/linux/amd64/e2e.test
_output/dockerized/bin/linux/amd64/go2make
_output/dockerized/bin/linux/amd64/deepcopy-gen
_output/dockerized/bin/linux/amd64/kube-scheduler
_output/dockerized/bin/linux/amd64/hyperkube
_output/dockerized/bin/linux/amd64/conversion-gen
_output/dockerized/bin/linux/amd64/kube-apiserver
_output/dockerized/bin/linux/amd64/kube-controller-manager
_output/dockerized/go
circleci@default-c236894c-8a28-48dc-9670-a8a3a15b6682:~/go/src/k8s.io/kubernetes$

cjschaef · 2019-05-14T15:00:25Z

Sorry I missed this yesterday. I will try #77867 as well. Otherwise, my assumption was to revert the use of a function, as that creates the scoping problem we've been seeing with readonly variables in bash 4.3.

dims · 2019-05-14T15:14:31Z

@aojea i think that would be make quick-release not make quick-release-images

dims · 2019-05-14T15:22:45Z

Also both make quick-release not make quick-release-images should work now.

cjschaef · 2019-05-14T18:38:12Z

Looks like #77867 worked for me. Was able to run a KUBE_FASTBUILD using bash 4.3 successfully.

GOPATH=/home/vagrant/go KUBE_GIT_VERSION=v1.15.0-alpha.3 KUBE_DOCKER_IMAGE_TAG=v1.15.0-alpha.3 KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true KUBE_VERBOSE=0 ./build/run.sh make hyperkube
...
cmd/hyperkube
+++ [0514 18:28:04] Syncing out of container

VERSION="v1.15.0-alpha.3" ARCH="amd64" make -C cluster/images/hyperkube/ build
...
Successfully built 09f441c067bb
Successfully tagged hyperkube-amd64:v1.15.0-alpha.3

spiffxp · 2019-05-14T21:55:17Z

ref: #76974 as the pr that appears to have caused this

cjschaef added the kind/bug Categorizes issue or PR as related to a bug. label May 6, 2019

k8s-ci-robot added the sig/release Categorizes an issue or PR as relevant to SIG Release. label May 6, 2019

ramineni mentioned this issue May 8, 2019

CSI Manila driver kubernetes/cloud-provider-openstack#536

Merged

adisky mentioned this issue May 13, 2019

KUBE_SERVER_PLATFORMS[@]: unbound variable building images on CircleCI #77799

Closed

kiwik mentioned this issue May 14, 2019

cloud-provider-openstack-acceptance-test-e2e-conformance job failed with "KUBE_SERVER_PLATFORMS[@]: unbound variable" error theopenlab/openlab#254

Closed

ramineni mentioned this issue May 14, 2019

workaround to fix cpo e2e job theopenlab/openlab-zuul-jobs#515

Merged

dims mentioned this issue May 14, 2019

Fix make quick-release failures under bash 4.3 #77867

Closed

adisky mentioned this issue May 15, 2019

fix hack/lib/golang.sh for bash 4.3 #77910

Merged

k8s-ci-robot closed this as completed in #77910 May 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build/release.sh (hack/lib/golang.sh) requires minimum bash v4.4 #77521

build/release.sh (hack/lib/golang.sh) requires minimum bash v4.4 #77521

cjschaef commented May 6, 2019

cjschaef commented May 6, 2019 •

edited

Loading

nikhita commented May 7, 2019

cjschaef commented May 7, 2019

adisky commented May 8, 2019

adisky commented May 13, 2019

kiwik commented May 14, 2019

dims commented May 14, 2019

aojea commented May 14, 2019 •

edited

Loading

cjschaef commented May 14, 2019

dims commented May 14, 2019

dims commented May 14, 2019

cjschaef commented May 14, 2019 •

edited

Loading

spiffxp commented May 14, 2019 •

edited

Loading

build/release.sh (hack/lib/golang.sh) requires minimum bash v4.4 #77521

build/release.sh (hack/lib/golang.sh) requires minimum bash v4.4 #77521

Comments

cjschaef commented May 6, 2019

cjschaef commented May 6, 2019 • edited Loading

nikhita commented May 7, 2019

cjschaef commented May 7, 2019

adisky commented May 8, 2019

adisky commented May 13, 2019

kiwik commented May 14, 2019

dims commented May 14, 2019

aojea commented May 14, 2019 • edited Loading

cjschaef commented May 14, 2019

dims commented May 14, 2019

dims commented May 14, 2019

cjschaef commented May 14, 2019 • edited Loading

spiffxp commented May 14, 2019 • edited Loading

cjschaef commented May 6, 2019 •

edited

Loading

aojea commented May 14, 2019 •

edited

Loading

cjschaef commented May 14, 2019 •

edited

Loading

spiffxp commented May 14, 2019 •

edited

Loading