Replacing Volume brick failed - repliacte:2 - heketi v9.0.0

[amgads]

Kind of issue

Bug
We're upgrading from heketi v5 to heketi v9.0.0

Observed behavior

Replicated volume - replica_count=2:

We're trying to do a recovery of a node. The node has one of the bricks for a volume (replica: 2) by doing:
a) removing the device on the failed node from heketi topology
b) allocate new brick on another node
c) replace old brick by new one.

We get error " ERROR : Command (/usr/bin/heketi-cli device remove ed0659cd69eb3d1a3c7b0f4c584ff6cf) failed: (Error: Failed to remove device, error: Cannot replace brick 0a7a5821de0e7a8ebbd6cb9568c0c00b as only 1 of 2 required peer bricks are online"

gluster cli allows that, but there is no way to sync heketi db with gluster state (to remove old bricks, add new)

Are there any possible solutions for these cases in heketi?

Expected/desired behavior

Details on how to reproduce (minimal and precise)

1. Topology of 3 nodes
2. Create a storage class with replicate:2
   volumetype: "replicate:2"
3. create a pod with volume (has two bricks)
4. delete a device or node that has one of the replica
   " ERROR : Command (/usr/bin/heketi-cli device remove ed0659cd69eb3d1a3c7b0f4c584ff6cf) failed: (Error: Failed to remove device, error: Cannot replace brick 0a7a5821de0e7a8ebbd6cb9568c0c00b as only 1 of 2 required peer bricks are online"

Information about the environment:

• Heketi version used: v9.0.0
• Operating system used: CentOS
• Heketi compiled from sources, as a package (rpm/deb), or container: package is used to create a container
• If container, which container image: built from using tar package
• Using kubernetes install
• If kubernetes: running using kubernetes storageclass
  kubectl apply -f .yml

Other useful information

Appreciate urgent support as this is blocking recovery of critical infrastructure

[sbias]

as far i can see, should be fixed in #1653 (which is not released yet)

[amgads]

is it part of v9.0.0 or it's going to be v10.0.0? where and when I can get the tar file with the fix
can you as well point to the build process/procedure to have a private build if needed?

[phlogistonjohn]

It will likely be part of the next release whatever version number we decide upon. The change in question is on the master branch. Refer to our
contributing.md doc for information on how to build from source.

[amgads]

According to: https://github.com/heketi/heketi/blob/master/docs/contributing.md,
I followed the steps but got errors (see below) -- help is appreciated

did the following:

yum -y install go glide

... installed

echo $HOME

/root

pwd

/root

mkdir golang

cd golang

export GOPATH=$HOME/golang

export PATH=$GOPATH/bin:$PATH

mkdir -p $GOPATH/src/github.com/heketi

cd $GOPATH/src/github.com/heketi

git clone https://github.com/heketi/heketi.git

.... cloned

make

fatal: --points-at option is only allowed with -l.
go build -ldflags "-X main.HEKETI_VERSION=v9.0.0-137-g7340ee4 -extldflags '-z relro -z now'" -o heketi
middleware/jwt.go:23:2: cannot find package "github.com/auth0/go-jwt-middleware" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/auth0/go-jwt-middleware (vendor tree)
/usr/lib/golang/src/github.com/auth0/go-jwt-middleware (from $GOROOT)
/root/golang/src/github.com/auth0/go-jwt-middleware (from $GOPATH)
apps/glusterfs/app.go:20:2: cannot find package "github.com/boltdb/bolt" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/boltdb/bolt (vendor tree)
/usr/lib/golang/src/github.com/boltdb/bolt (from $GOROOT)
/root/golang/src/github.com/boltdb/bolt (from $GOPATH)
apps/glusterfs/app_middleware.go:16:2: cannot find package "github.com/dgrijalva/jwt-go" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/dgrijalva/jwt-go (vendor tree)
/usr/lib/golang/src/github.com/dgrijalva/jwt-go (from $GOROOT)
/root/golang/src/github.com/dgrijalva/jwt-go (from $GOPATH)
pkg/glusterfs/api/types.go:24:2: cannot find package "github.com/go-ozzo/ozzo-validation" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/go-ozzo/ozzo-validation (vendor tree)
/usr/lib/golang/src/github.com/go-ozzo/ozzo-validation (from $GOROOT)
/root/golang/src/github.com/go-ozzo/ozzo-validation (from $GOPATH)
pkg/glusterfs/api/types.go:25:2: cannot find package "github.com/go-ozzo/ozzo-validation/is" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/go-ozzo/ozzo-validation/is (vendor tree)
/usr/lib/golang/src/github.com/go-ozzo/ozzo-validation/is (from $GOROOT)
/root/golang/src/github.com/go-ozzo/ozzo-validation/is (from $GOPATH)
apps/glusterfs/app_middleware.go:17:2: cannot find package "github.com/gorilla/context" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/gorilla/context (vendor tree)
/usr/lib/golang/src/github.com/gorilla/context (from $GOROOT)
/root/golang/src/github.com/gorilla/context (from $GOPATH)
main.go:24:2: cannot find package "github.com/gorilla/mux" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/gorilla/mux (vendor tree)
/usr/lib/golang/src/github.com/gorilla/mux (from $GOROOT)
/root/golang/src/github.com/gorilla/mux (from $GOPATH)
pkg/idgen/id.go:20:2: cannot find package "github.com/lpabon/godbc" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/lpabon/godbc (vendor tree)
/usr/lib/golang/src/github.com/lpabon/godbc (from $GOROOT)
/root/golang/src/github.com/lpabon/godbc (from $GOPATH)
pkg/metrics/metrics.go:17:2: cannot find package "github.com/prometheus/client_golang/prometheus" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/prometheus/client_golang/prometheus (vendor tree)
/usr/lib/golang/src/github.com/prometheus/client_golang/prometheus (from $GOROOT)
/root/golang/src/github.com/prometheus/client_golang/prometheus (from $GOPATH)
pkg/metrics/metrics.go:18:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
/usr/lib/golang/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
/root/golang/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
main.go:25:2: cannot find package "github.com/spf13/cobra" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/spf13/cobra (vendor tree)
/usr/lib/golang/src/github.com/spf13/cobra (from $GOROOT)
/root/golang/src/github.com/spf13/cobra (from $GOPATH)
apps/glusterfs/app_middleware.go:18:2: cannot find package "github.com/urfave/negroni" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/github.com/urfave/negroni (vendor tree)
/usr/lib/golang/src/github.com/urfave/negroni (from $GOROOT)
/root/golang/src/github.com/urfave/negroni (from $GOPATH)
pkg/remoteexec/ssh/ssh.go:23:2: cannot find package "golang.org/x/crypto/ssh" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/golang.org/x/crypto/ssh (vendor tree)
/usr/lib/golang/src/golang.org/x/crypto/ssh (from $GOROOT)
/root/golang/src/golang.org/x/crypto/ssh (from $GOPATH)
pkg/remoteexec/ssh/ssh.go:24:2: cannot find package "golang.org/x/crypto/ssh/agent" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/golang.org/x/crypto/ssh/agent (vendor tree)
/usr/lib/golang/src/golang.org/x/crypto/ssh/agent (from $GOROOT)
/root/golang/src/golang.org/x/crypto/ssh/agent (from $GOPATH)
pkg/remoteexec/ssh/ssh.go:25:2: cannot find package "golang.org/x/crypto/ssh/knownhosts" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/golang.org/x/crypto/ssh/knownhosts (vendor tree)
/usr/lib/golang/src/golang.org/x/crypto/ssh/knownhosts (from $GOROOT)
/root/golang/src/golang.org/x/crypto/ssh/knownhosts (from $GOPATH)
pkg/kubernetes/backupdb.go:21:2: cannot find package "k8s.io/api/core/v1" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/api/core/v1 (vendor tree)
/usr/lib/golang/src/k8s.io/api/core/v1 (from $GOROOT)
/root/golang/src/k8s.io/api/core/v1 (from $GOPATH)
pkg/kubernetes/backupdb.go:22:2: cannot find package "k8s.io/apimachinery/pkg/api/errors" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/apimachinery/pkg/api/errors (vendor tree)
/usr/lib/golang/src/k8s.io/apimachinery/pkg/api/errors (from $GOROOT)
/root/golang/src/k8s.io/apimachinery/pkg/api/errors (from $GOPATH)
pkg/kubernetes/pv.go:18:2: cannot find package "k8s.io/apimachinery/pkg/api/resource" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/apimachinery/pkg/api/resource (vendor tree)
/usr/lib/golang/src/k8s.io/apimachinery/pkg/api/resource (from $GOROOT)
/root/golang/src/k8s.io/apimachinery/pkg/api/resource (from $GOPATH)
pkg/remoteexec/kube/target.go:16:2: cannot find package "k8s.io/apimachinery/pkg/apis/meta/v1" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 (vendor tree)
/usr/lib/golang/src/k8s.io/apimachinery/pkg/apis/meta/v1 (from $GOROOT)
/root/golang/src/k8s.io/apimachinery/pkg/apis/meta/v1 (from $GOPATH)
pkg/kubernetes/backupdb.go:23:2: cannot find package "k8s.io/client-go/kubernetes" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/client-go/kubernetes (vendor tree)
/usr/lib/golang/src/k8s.io/client-go/kubernetes (from $GOROOT)
/root/golang/src/k8s.io/client-go/kubernetes (from $GOPATH)
pkg/remoteexec/kube/exec.go:19:2: cannot find package "k8s.io/client-go/kubernetes/scheme" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/client-go/kubernetes/scheme (vendor tree)
/usr/lib/golang/src/k8s.io/client-go/kubernetes/scheme (from $GOROOT)
/root/golang/src/k8s.io/client-go/kubernetes/scheme (from $GOPATH)
pkg/kubernetes/backupdb.go:24:2: cannot find package "k8s.io/client-go/rest" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/client-go/rest (vendor tree)
/usr/lib/golang/src/k8s.io/client-go/rest (from $GOROOT)
/root/golang/src/k8s.io/client-go/rest (from $GOPATH)
pkg/remoteexec/kube/exec.go:20:2: cannot find package "k8s.io/client-go/tools/remotecommand" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/client-go/tools/remotecommand (vendor tree)
/usr/lib/golang/src/k8s.io/client-go/tools/remotecommand (from $GOROOT)
/root/golang/src/k8s.io/client-go/tools/remotecommand (from $GOPATH)
pkg/remoteexec/kube/exec.go:21:2: cannot find package "k8s.io/client-go/util/exec" in any of:
/root/golang/src/github.com/heketi/heketi/vendor/k8s.io/client-go/util/exec (vendor tree)
/usr/lib/golang/src/k8s.io/client-go/util/exec (from $GOROOT)
/root/golang/src/k8s.io/client-go/util/exec (from $GOPATH)
make: *** [heketi] Error 1

[amgads]

I need to build the server .tgz file

[raghavendra-talur]

@amgads try running "glide install -v" before make.

[amgads]

Thanks a lot! It did build the "heketi" executable. How do I build the tarball "heketi-v9.0.0.linux.amd64.tar.gz" ?

[amgads]

I guess the question: is there a make option to make it? or it's just a tarball of the files:
"heketi" -- under $HOME/golang/src/github.com/heketi/heketi
"heketi-cli" -- under $HOME/....../heketi/client/cli/go
"heketi.json" -- under $HOME/ ....../heketi/etc

[raghavendra-talur]

You are probably looking for "make release" https://github.com/heketi/heketi/blob/master/Makefile#L162

[amgads]

Thanks a lot ..

[amgads]

I tested the original problem “remove replicate:2 bricks” after building from master and it failed the same way as reported

[amgads]

one thing I realised -- heketi and heketi-cli sizes are smaller than the original "v9.0.0". Does that mean something wasn't picked up or newer code is cleaned up?

[amgads]

The code is showing the changes that was done
func (r *VolumeReplicaDurability) QuorumBrickCount() int {
if r.BricksInSet() < 3 {
return 1
} else {
return r.BricksInSet()/2 + 1
}
}
.....

[amgads]

OK -- found the issue. It was an enviroment parameter in the container. So the fix works.
However, I'm facing a different issue as some of the topology/device info fields have changed. My code is broken if I use the latest master copy. I tried to build one v9.0.0 and the fix on top, but didn't compile and stuck with the following error: any idea to help

[INFO] --> Fetching k8s.io/kubernetes.
[WARN] Unable to checkout bitbucket.org/ww/goautoneg
[ERROR] Update failed for bitbucket.org/ww/goautoneg: Cannot detect VCS

[fkpwolf]

I just encountered same issue. After build, need copy binary output to server end. Using new heketi-cli is not enough.

@amgads
[ERROR] Update failed for bitbucket.org/ww/goautoneg: Cannot detect VCS.
It is a network issue.

[phlogistonjohn]

Real issue was resolved. Issue in build is a how glide-uses-bitbucket bug, but glide is a moribund project AFAICT.
IIRC we hacked something in to work around this bug.
Closing issue.