PrestopHook configured for zookeeper cluster always fails

[pbelgundi]

This hook is invoked by kubernetes when we try to gracefuly delete a pod in the cluster.
Exception noticed:
"
Warning FailedPreStopHook 11s kubelet, 12bb6b0d-8b46-4a58-bd42-5051288f62d5 Exec lifecycle hook ([zookeeperTeardown.sh]) for Container "zookeeper" in Pod "example-2_default(2e958a3d-ec21-11e9-8087-005056b6cc60)" failed - error: command 'zookeeperTeardown.sh' exited with 137: + source /conf/env.sh
++ DOMAIN=example-headless.default.svc.cluster.local
++ QUORUM_PORT=2888
++ LEADER_PORT=3888
++ CLIENT_HOST=example-client
++ CLIENT_PORT=2181

• source /usr/local/bin/zookeeperFunctions.sh
  ++ set -ex
• DATA_DIR=/data
• MYID_FILE=/data/myid
• LOG4J_CONF=/conf/log4j-quiet.properties
• set +e
  ++ zkConnectionString
  ++ set +e
  ++ nslookup example-client
  ++ [[ 0 -eq 1 ]]
  ++ set -e
  ++ echo example-client:2181
• ZKURL=example-client:2181
• set -e
  ++ cat /data/myid
• MYID=3
• java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar remove example-client:2181 3
  Connecting to Zookeeper example-client:2181
  ++ echo cons
  ++ nc localhost 2181
  ++ grep -v '^$'
  ++ wc -l
  ++ grep -v /127.0.0.1:
• CONN_COUNT=0
• (( i = 0 ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ nc localhost 2181
  ++ grep -v '^$'
  ++ grep -v /127.0.0.1:
  ++ wc -l
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ nc localhost 2181
  ++ grep -v /127.0.0.1:
  ++ grep -v '^$'
  ++ wc -l
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ grep -v '^$'
  ++ nc localhost 2181
  ++ grep -v /127.0.0.1:
  ++ wc -l
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ nc localhost 2181
  ++ grep -v /127.0.0.1:
  ++ grep -v '^$'
  ++ wc -l
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ grep -v '^$'
  ++ nc localhost 2181
  ++ wc -l
  ++ grep -v /127.0.0.1:
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  ++ echo cons
  ++ grep -v /127.0.0.1:
  ++ wc -l
  ++ nc localhost 2181
  ++ grep -v '^$'
• CONN_COUNT=0
• (( i++ ))
• (( i < 36 ))
• echo '0 non-local connections still connected.'
• sleep 5
  , message: "0 non-local connections still connected.\n0 non-local connections still connected.\n0 non-local connections still connected.\n0 non-local connections still connected.\n0 non-local connections still connected.\n0 non-local connections still connected.\n0 non-local connections still connected.\n+ source /conf/env.sh\n++ DOMAIN=example-headless.default.svc.cluster.local\n++ QUORUM_PORT=2888\n++ LEADER_PORT=3888\n++ CLIENT_HOST=example-client\n++ CLIENT_PORT=2181\n+ source /usr/local/bin/zookeeperFunctions.sh\n++ set -ex\n+ DATA_DIR=/data\n+ MYID_FILE=/data/myid\n+ LOG4J_CONF=/conf/log4j-quiet.properties\n+ set +e\n++ zkConnectionString\n++ set +e\n++ nslookup example-client\n++ [[ 0 -eq 1 ]]\n++ set -e\n++ echo example-client:2181\n+ ZKURL=example-client:2181\n+ set -e\n++ cat /data/myid\n+ MYID=3\n+ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar remove example-client:2181 3\nConnecting to Zookeeper example-client:2181\n++ echo cons\n++ nc localhost 2181\n++ grep -v '^$'\n++ wc -l\n++ grep -v /127.0.0.1:\n+ CONN_COUNT=0\n+ (( i = 0 ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ nc localhost 2181\n++ grep -v '^$'\n++ grep -v /127.0.0.1:\n++ wc -l\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ nc localhost 2181\n++ grep -v /127.0.0.1:\n++ grep -v '^$'\n++ wc -l\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ grep -v '^$'\n++ nc localhost 2181\n++ grep -v /127.0.0.1:\n++ wc -l\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ nc localhost 2181\n++ grep -v /127.0.0.1:\n++ grep -v '^$'\n++ wc -l\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ grep -v '^$'\n++ nc localhost 2181\n++ wc -l\n++ grep -v /127.0.0.1:\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n++ echo cons\n++ grep -v /127.0.0.1:\n++ wc -l\n++ nc localhost 2181\n++ grep -v '^$'\n+ CONN_COUNT=0\n+ (( i++ ))\n+ (( i < 36 ))\n+ echo '0 non-local connections still connected.'\n+ sleep 5\n"
  "

[pbelgundi]

Need to check if this is still reproducible and do RCA