"timed out locking store" leads to "unexpected end of JSON input" on v1.0.16 #289
Description
Is this an ISSUE or FEATURE REQUEST? (choose one):
Issue
Which release version?:
v1.0.16
Which component (CNI/IPAM/CNM/CNS):
CNI
Which Operating System (Linux/Windows):
Windows
For windows: provide output of "$(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion"
Windows Server 2019 / 10.0.17763.194
Which Orchestrator and version (e.g. Kubernetes, Docker)
Kubernetes
What happened:
Trying to start a group of pods at the same time will fail.
Azure-vnet.log:
2019/01/14 22:38:50 [cni] Failed to lock store: timed out locking store.
2019/01/14 22:38:50 Failed to initialize key-value store of network plugin, err:timed out locking store.
2019/01/14 22:38:50 Report plugin error
2019/01/14 22:38:50 [cni] Failed to lock store: timed out locking store.
2019/01/14 22:38:50 Failed to initialize key-value store of network plugin, err:timed out locking store.
2019/01/14 22:38:50 Report plugin error
2019/01/14 22:38:51 [cni] Failed to lock store: timed out locking store.
2019/01/14 22:38:51 Failed to initialize key-value store of network plugin, err:timed out locking store.
2019/01/14 22:38:51 Report plugin error
2019/01/14 22:38:52 [cni] Failed to lock store: timed out locking store.
2019/01/14 22:38:52 Failed to initialize key-value store of network plugin, err:timed out locking store.
2019/01/14 22:38:52 Report plugin error
2019/01/14 22:38:52 [cni] Failed to lock store: timed out locking store.
2019/01/14 22:38:52 Failed to initialize key-value store of network plugin, err:timed out locking store.
2019/01/14 22:38:52 Report plugin error
Kubelet.err.log:
E0114 22:38:53.490441 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-vhjqm/5c23ee042e342f0e1386c861179ea29cabfb40e50fb17aa9b294a0e55876047e to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
E0114 22:38:53.495500 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-dxzbp/d90312b6cfee53edcd0cff74bb26e6bcfc7aea14ab91aa511bb4316bd84b9d6e to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
E0114 22:38:55.860006 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-tdg8w/9244da2c082350729b28dc097e4505677acf510c0d373421b1f51c9cde40181e to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
E0114 22:38:56.061026 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-nmmcs/f34a9fa829518df1ed36d4ecbe5278581d0f100a0667f8f14f1ac99351ad5708 to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
E0114 22:38:56.102027 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-bkl6p/ca9316dc5e204a6aa663520b2a94716e9d17aa0fbdc7e991dbe0fe19d4119273 to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
E0114 22:38:59.749474 5580 cni.go:324] Error adding default_iis-2019-c5d48d8c-7xj8g/93040c2380203ad60183c9004cc079d99c41d6150e613828eefd1e4aecf050e3 to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
What you expected to happen:
No failures
How to reproduce it (as minimally and precisely as possible):
kubectl create -f https://raw.githubusercontent.com/PatrickLang/Windows-K8s-Samples/master/iis/iis-2019.yaml
kubectl scale deploy iis-2019 --replicas=20
Anything else we need to know:
#251 probably fixes this but has been stuck in the PR queue for some months
#242 and #186 are past user reports on this
Metadata
Metadata
Assignees
Labels
No labels