New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iSCSI mount fails in 1.12.x/1.13.x #73858

Open
MansM opened this Issue Feb 8, 2019 · 5 comments

Comments

Projects
None yet
2 participants
@MansM
Copy link

MansM commented Feb 8, 2019

What happened:
iscsi volumes fail to mount

What you expected to happen:
mounted volume
How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mediasync-data
  namespace: mediasync
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  persistentVolumeReclaimPolicy: Retain
  iscsi:
    iqn: iqn.2000-01.com.synology:DiskStation.Target-1.9cbabe8c8b
    targetPortal: 10.0.1.2
    lun: 3

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mediasync-data
  namespace: mediasync
spec:
  volumeName: mediasync-data
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Anything else we need to know?:
worked on 1.11.3

Environment:

  • Kubernetes version (use kubectl version):
    kubectl version
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm"}
    Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm"}
  • Cloud provider or hardware configuration: rpi 3+
  • OS (e.g. from /etc/os-release):
    PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
  • Kernel (e.g. uname -a):
    Linux k8s-node-1 4.14.34-hypriotos-v7+ #1 SMP Sun Apr 22 14:57:31 UTC 2018 armv7l GNU/Linux
  • Install tools: kubeadm
  • Others:
    open-iscsi/stable,now 2.0.874-3~deb9u1 armhf [installed]

Feb 08 18:31:17 k8s-node-1 kubelet[310]: E0208 18:31:17.704144 310 nestedpendingoperations.go:267] Operation for ""kubernetes.io/iscsi/10.0.1.2:iqn.2000-01.com.synology:DiskStation.Target-1.9cbabe8c8b:3"" failed. No retries permitted until 2019-02-08 18:33:19.703838972 +0000 UTC m=+1353.824233274 (durationBeforeRetry 2m2s). Error: "recovered from panic "runtime error: index out of range". (err=) Call stack:\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158\n/usr/local/go/src/runtime/asm_arm.s:519\n/usr/local/go/src/runtime/panic.go:513\n/usr/local/go/src/runtime/panic.go:44\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/keymutex/hashed.go:48\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/iscsi/iscsi_util.go:317\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/iscsi/attacher.go:85\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:503\n/workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:143\n/usr/local/go/src/runtime/asm_arm.s:867\n"

@MansM MansM added the kind/bug label Feb 8, 2019

@MansM

This comment has been minimized.

Copy link
Author

MansM commented Feb 8, 2019

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage and removed needs-sig labels Feb 8, 2019

@MansM

This comment has been minimized.

Copy link
Author

MansM commented Feb 9, 2019

did some research, doesnt work on 1.12.1/1.12.5/1.13.3

only working option for now is 1.11.3

@MansM

This comment has been minimized.

Copy link
Author

MansM commented Feb 9, 2019

@MansM MansM changed the title iscsi mount fails iSCSI mount fails in 1.12.x/1.13.x Feb 11, 2019

@MansM

This comment has been minimized.

Copy link
Author

MansM commented Feb 17, 2019

Brian Hand ( slack/sig-storage) came to the conclusion it had to do with the way the hash are calculated. He created the following snippet:

package main

import (
	"fmt"
	"hash/fnv"
	"runtime"
)

func main() {
	// this results in a negative number and causes K8s to fail
	// recommend that the devs adopt adding some code to ensure that a positive number is always returned like wrapping in an abs function or the like
	fmt.Println(hash("iqn.2000-01.com.synology:DiskStation.Target-1.9cbabe8c8b"))
	fmt.Println(hash("iqn.2000-01.com.synology:DiskStation.Target-1.9cbabe8c8b") % runtime.NumCPU())

	// this below was me changing the iqn string until I got a positive number.  Then I used that iqn on my deployment.
	fmt.Println(hash("iqn.2000-01.com.synology:DiskStation.Target-1.u") % runtime.NumCPU())
}

// This was the hash function in the source
func hash(id string) int {
	h := fnv.New32a()
	h.Write([]byte(id))
	return int(h.Sum32())
}

which uses same method as in k8s ( https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/utils/keymutex/hashed.go#L54 )

output on arm:

root@ba278112ec42:/usr/src/myapp# go build && ./myapp
-1920022901
-1
3

output on amd64:

root@ed6e4e31a707:/usr/src/myapp# go build && ./myapp
2374944395
3
3
@MansM

This comment has been minimized.

Copy link
Author

MansM commented Feb 17, 2019

@danielqsj you added keymutex to the iscsi plugin, can you take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment