Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID #53466

davidz627 · 2017-10-04T22:37:41Z

Mount SCSI local SSD by UUID in /mnt/disks/by-uuid/, also allows for users to request and mount NVMe disks. Both types of disks will be accessable either through block or filesystem

To see code in progress for NVMe and block support see working branch: https://github.com/davidz627/kubernetes/tree/localExt

davidz627 · 2017-10-04T22:40:31Z

cc @msau42

msau42 · 2017-10-04T22:42:45Z

/ok-to-test

msau42 · 2017-10-04T22:42:58Z

/release-note-none

msau42 · 2017-10-04T22:44:42Z

cluster/common.sh

@@ -679,6 +679,8 @@ PROMETHEUS_TO_SD_ENDPOINT: $(yaml-quote ${PROMETHEUS_TO_SD_ENDPOINT:-})
 PROMETHEUS_TO_SD_PREFIX: $(yaml-quote ${PROMETHEUS_TO_SD_PREFIX:-})
 ENABLE_PROMETHEUS_TO_SD: $(yaml-quote ${ENABLE_PROMETHEUS_TO_SD:-false})
 ENABLE_POD_PRIORITY: $(yaml-quote ${ENABLE_POD_PRIORITY:-})
+NODE_LOCAL_SSDS: $(yaml-quote ${NODE_LOCAL_SSDS:-0})


Hm is this the right place to add this? This is a GCE specific variable, but the file is not GCE specific

I'm not actually sure. I can't find any other scripts that seem to set env variables on the node, this looks like the only one to me. Could you point me to an alternative or something else I should be looking for?

Oh I think I see now. Before, the configure-vm script didn't need NODE_LOCAL_SSDS, but now it does. Yes, this is the right place to copy over env vars

msau42 · 2017-10-04T22:45:07Z

cluster/gce/config-default.sh

@@ -36,6 +36,7 @@ MASTER_ROOT_DISK_SIZE=${MASTER_ROOT_DISK_SIZE:-$(get-master-root-disk-size)}
 NODE_DISK_TYPE=${NODE_DISK_TYPE:-pd-standard}
 NODE_DISK_SIZE=${NODE_DISK_SIZE:-100GB}
 NODE_LOCAL_SSDS=${NODE_LOCAL_SSDS:-0}
+NODE_LOCAL_SSDS_EXT=${NODE_LOCAL_SSDS_EXT:-""}


Can you add comments to describe what the variable means, and the format of the values

msau42 · 2017-10-04T23:00:09Z

cluster/gce/gci/configure-helper.sh

+    chmod +w ${ssdmap}
+  fi
+
+  # each line looks like "${device} persistant-uuid"


msau42 · 2017-10-04T23:06:02Z

cluster/gce/gci/configure-helper.sh

+  # Get the real UUID of the device
+  item=$(readlink -f ${device} | cut -d '/' -f 3)
+  myuuid=$(ls -l /dev/disk/by-uuid/ | grep ${item} | tr -s ' ' | cut -d ' ' -f 9) 
+  mountpoint="/mnt/disks/by-uuid/google-local-ssds-${interface,,}-fs/local-ssd-${myuuid}"


Can you make this path a constant since we use it a few times in this file

do you mean just make "/mnt/disks/by-uuid/google-local-ssds" a const var like mntprefix or something?

msau42 · 2017-10-04T23:07:04Z

cluster/gce/gci/configure-helper.sh

  mkdir -p "${mountpoint}"
  echo "Mounting '${device}' at '${mountpoint}'"
  mount -o discard,defaults "${device}" "${mountpoint}"
+  chmod a+w ${mountpoint}


Do we need to do this considering that it wasn't done previously?

I believe this was done previously on a mount:

kubernetes/cluster/gce/gci/configure-helper.sh

Line 95 in fa916c1

chmod a+w ${ssdmount}

msau42 · 2017-10-04T23:12:07Z

cluster/gce/gci/configure-helper.sh

+  format=$3  
+
+  if [[ ${format,,} == "fs" ]]; then
+    safe-format-and-mount-by-interface ${ssd} ${interface,,}


What if we call safe-format-and-mount like before, and then a new function that does the bind mount to the uuid path?

this method is more natural for if we plan to eventually deprecate the "old" mount points. We would just have to delete the bind-mounting section as opposed to reversing the logic when the backwards compatibility is removed.

I don't think we'll be able to deprecate it anytime soon. We'll have to support both types of users. Those that want to use hostpath and needs the paths to be consistent across nodes, and those that want Local PVs which needs all the paths to be unique.

davidz627 · 2017-10-05T00:10:58Z

/retest

msau42 · 2017-10-05T01:06:46Z

cluster/common.sh

@@ -679,6 +679,8 @@ PROMETHEUS_TO_SD_ENDPOINT: $(yaml-quote ${PROMETHEUS_TO_SD_ENDPOINT:-})
 PROMETHEUS_TO_SD_PREFIX: $(yaml-quote ${PROMETHEUS_TO_SD_PREFIX:-})
 ENABLE_PROMETHEUS_TO_SD: $(yaml-quote ${ENABLE_PROMETHEUS_TO_SD:-false})
 ENABLE_POD_PRIORITY: $(yaml-quote ${ENABLE_POD_PRIORITY:-})
+NODE_LOCAL_SSDS: $(yaml-quote ${NODE_LOCAL_SSDS:-0})


Oh I think I see now. Before, the configure-vm script didn't need NODE_LOCAL_SSDS, but now it does. Yes, this is the right place to copy over env vars

msau42 · 2017-10-05T01:09:22Z

cluster/gce/gci/configure-helper.sh

+  format=$2
+
+  total=0
+  if [[ ! -z ${NODE_LOCAL_SSDS_EXT} ]]; then


Can you add a comment describe the format of the value?

msau42 · 2017-10-05T01:09:45Z

cluster/gce/gci/configure-helper.sh

+  # Get the real UUID of the device
+  item=$(readlink -f ${device} | cut -d '/' -f 3)
+  myuuid=$(ls -l /dev/disk/by-uuid/ | grep ${item} | tr -s ' ' | cut -d ' ' -f 9) 
+  mountpoint="/mnt/disks/by-uuid/google-local-ssds-${interface,,}-fs/local-ssd-${myuuid}"


msau42 · 2017-10-05T01:12:11Z

/assign

davidz627 · 2017-10-05T23:59:15Z

addressed comments and added tests.

msau42 · 2017-10-06T00:17:05Z

cluster/gce/gci/configure-helper.sh

@@ -91,7 +93,7 @@ function get-local-disk-num() {
 function safe-block-symlink(){
  device=$1
  interface=$2
-  mkdir -p "/mnt/disks/by-uuid/google-local-ssds-${interface,,}-block"
+  mkdir -p "${UUID_MNT_PREFIX}-${interface,,}-block"


can you handle lowercasing $interface when we're parsing the args?

i mean very technically none of these ever need to be lower cased because they just come from some hardcoded "scsi" "fs" strings. I'll remove them all.

msau42 · 2017-10-06T00:20:00Z

test/e2e/storage/persistent_volumes-local.go

@@ -54,6 +55,8 @@ const (
 	DirectoryLocalVolumeType LocalVolumeType = "dir"
 	// creates a tmpfs and mounts it
 	TmpfsLocalVolumeType LocalVolumeType = "tmpfs"
+	// tests based on local ssd at /mnt/disks/by-uuid/
+	GCELocalSSDVolumeType LocalVolumeType = "gcessd"


maybe call it "gce-localssd-scsi-fs"

msau42 · 2017-10-06T00:20:50Z

test/e2e/storage/persistent_volumes-local.go


 	Context("when two pods mount a local volume at the same time", func() {
+		BeforeEach(func() {
+			framework.SkipUnlessLocalSSDExists("scsi", "fs", config.node0)


We only want to skip if the type == GCELocalSSDVolumeType.

Ditto everywhere else

In that case if we want to keep the tests as they are they will not "skip" the test perse, just skip the part that involves localSSD. The test will still show up as "passed" when running because it just wont do the localSSD part.

Alternatively I could split out the tests so that it does localSSD seperately and actually "skips" the test so it is more clear as to what is happening when you run them.

Hmmm I wonder if it's possible to restructure the tests so that there is an It() for each type.

msau42 · 2017-10-06T00:23:07Z

test/e2e/storage/persistent_volumes-local.go

-	testDirName := "local-volume-test-" + string(uuid.NewUUID())
-	hostDir := filepath.Join(hostBase, testDirName)
+	var hostDir string
+	if localVolumeType == GCELocalSSDVolumeType {


Maybe the setup and cleanup methods can be abstracted out into an interface for each supported LocalVolumeType

Im not sure what you mean by abstracting them out into an interface. Do you mean to have multiple methods for setup and cleanup (one for each volume type)?

Yes, I mean define an interface with Setup and Teardown methods. And have each volume type implement that interface with their own methods.

I'm actually not really sure how to do that since localVolumeType aren't actually different "types" but are actually just strings so you can't really distinguish them for interface methods. They are all of type "LocalVolumeType". Any documentation on what you're suggesting here?

You could have a map with key = local volume type string, value = setup and teardown methods

msau42 · 2017-10-06T00:25:56Z

cluster/gce/gci/configure-helper.sh

+  fi
+
+  sym="${UUID_MNT_PREFIX}-${interface,,}-block/local-ssd-${myuuid}"
+  # dont make the exact directory first or else symlink goes 1 directory down


can you explain this comment? i'm not understanding it

If you make the directory with "mkdir" before doing a symlink to that directory, it actually symlinks to "inside" the directory instead of "to" the directory. I've updated the comment to:

Do not "mkdir -p ${sym}" as that will cause unintended symlink behavior

Ok yeah that's correct. Symlinks are files, not directories.

that explains a lot

msau42 · 2017-10-06T00:30:25Z

cluster/gce/gci/configure-helper.sh

@@ -140,6 +241,8 @@ function find-master-pd {
  MASTER_PD_DEVICE="/dev/disk/by-id/${relative_path}"
 }

+


msau42 · 2017-10-06T00:32:36Z

cluster/gce/util.sh

+    for ssdgroup in "${ssdgroups[@]}"
+    do
+      IFS="," read -r -a ssdopts <<< ${ssdgroup}
+      for i in $(seq ${ssdopts[0]}); do


What happens if the user misformats these variables? (like maybe only includes 1 out of the 3 values)

it just kinda "goes with it" and tries its best, an invalid array access here will basically just return the empty string. Should I add some validation?

I think some validation would be good, so that we don't accidentally create more local SSDs than the user intended. Actually, there was a bug a few months ago where "seq 0" returns "1 0" on mac, so it actually created 2 local SSDs instead of 0. We should make sure we don't hit the same issue here.

I guess if the local ssd interface is malformed, then the instance creation should fail with invalid argument.

msau42 · 2017-10-06T00:35:44Z

cluster/gce/gci/configure-helper.sh

@@ -66,8 +68,57 @@ function create-dirs {
  fi
 }

-# Formats the given device ($1) if needed and mounts it at given mount point
-# ($2).
+function get-local-disk-num() {


Can you add a comment as to what this function does

msau42 · 2017-10-06T00:38:57Z

cluster/gce/gci/configure-helper.sh

 function ensure-local-ssds() {
+  get-local-disk-num "scsi" "fs"
+  scsifsnum=${total}


you can combine these two lines

msau42 · 2017-10-06T00:41:09Z

cluster/gce/gci/configure-helper.sh

 function ensure-local-ssds() {
+  get-local-disk-num "scsi" "fs"
+  scsifsnum=${total}
+  scsifsnum=$((scsifsnum+NODE_LOCAL_SSDS))


I think this is going to break GKE because GKE doesn't set NODE_LOCAL_SSDS, so scsifsnum will be 0.

do they still provision localssds without setting that env var?

Yes, GKE creates the instances differently than the cluster/gce/util.sh script.

We may need to do something like:

First setup all the block devices

The remainder is for filesystem

msau42 · 2017-10-06T00:42:21Z

cluster/gce/gci/configure-helper.sh

-# Formats the given device ($1) if needed and mounts it at given mount point
-# ($2).
+function get-local-disk-num() {
+  interface=$1


Can you prefix all local variables with "local"

davidz627 · 2017-10-09T21:41:27Z

addressed all comments.

msau42 · 2017-10-09T21:46:31Z

cluster/gce/gci/configure-helper.sh

+  local interface=$1
+  local format=$2
+
+  total=0


Can you give this a more distinct name to minimize variable name collisions

msau42 · 2017-10-09T21:51:52Z

cluster/gce/gci/configure-helper.sh

+  fi
+
+  # each line looks like "${device} persistent-uuid"
+  if [[ ! -z $(cat ${ssdmap} | grep ${device}) ]]; then


You could just do grep $device $ssdmap.

I would add a space after device, the handle the case where you have 2 devices and the first is a substring of the second.

msau42 · 2017-10-09T21:53:50Z

cluster/gce/gci/configure-helper.sh

+  udevadm settle
+
+  # Get the real UUID of the device
+  local item=$(readlink -f ${device} | cut -d '/' -f 3)


Can you add a comment at each step what the expected output format is?

msau42 · 2017-10-09T21:54:53Z

cluster/gce/gci/configure-helper.sh

+  if [[ ${format} == "fs" ]]; then
+    if [[ ${interface} == "scsi" ]]; then
+      local scsinum=`echo ${ssd} | sed -e 's/\/dev\/disk\/by-id\/google-local-ssd-\([0-9]*\)/\1/'`
+      local mountpoint="/mnt/disks/ssd${scsinum}"


This is only going to be local to this if block?

I think local restricts only to function blocks. So this is technically valid. Would it look better in some other way?

I dont know if bash has the concept of variable declaration without initialization so i couldn't do:

local scsinum;

at some earlier point.

msau42 · 2017-10-09T21:58:05Z

cluster/gce/gci/configure-helper.sh

 function ensure-local-ssds() {
+  get-local-disk-num "scsi" "block"
+  local scsiblocknum=${total}
+  local i=0
  for ssd in /dev/disk/by-id/google-local-ssd-*; do


Should we sort this list to ensure that we process the disks in the same order?

The ${ssd} will already be sorted by default. I do remember why i decided to do "fs" disks first now though, now the disk numbers will make even less sense because they would start with the numbers after block storage.

Ex. We have 2 block and 2 fs devices
google-local-ssd-[0,1] are block,
then when we mount the fs devices we will have them mounted at:
/mnt/disks/ssd2 /mnt/disks/ssd3

And if we include nvme devices in the mix they might not even be increasing by 1 each time...

Can be solved by re-indexing them to start at 0 and monotonically increment. But then the numbers are no longer 1-1 from /dev/disks to /mnt/disks

davidz627 · 2017-10-10T00:15:27Z

all comments addressed and nvme support added

davidz627 · 2017-10-12T23:06:18Z

Comments from other PR addressed.

davidz627 · 2017-10-13T17:21:27Z

/retest

zmerlynn · 2017-11-14T22:34:22Z

LGTM except a couple of real comments and some nits.

…disks/by-uuid/, set up ability to have unformatted disk symlinked in /dev/disk/by-uuid/. Added tests. Preserved backwards compatibility.

davidz627 · 2017-11-15T01:15:07Z

addressed your comments @zmerlynn

davidz627 · 2017-11-15T18:38:41Z

/retest

davidz627 · 2017-11-15T21:25:35Z

/retest

zmerlynn · 2017-11-16T00:55:31Z

/lgtm /approve

…

On Wed, Nov 15, 2017 at 1:26 PM, David Zhu ***@***.***> wrote: /retest — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#53466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEtqgMpoOoNJbCbc-kuzdif_YADF3oVCks5s21bxgaJpZM4PuVU3> .

k8s-github-robot · 2017-11-16T00:56:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davidz627, msau42, zmerlynn

Associated issue: 51431

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cluster/OWNERS~~ [zmerlynn]
~~test/e2e/storage/OWNERS~~ [msau42,zmerlynn]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-11-16T01:42:04Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-11-16T02:25:29Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

rohitagarwal003 · 2017-11-16T15:45:06Z

This seems to have broken all GKE tests.

https://k8s-testgrid.appspot.com/google-gke#gci-gke

Last success: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/18380

First failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/18381

6f35d49...c3ed0f2

mikedanese · 2017-11-16T15:55:09Z

I'm running gke-gci on the revert. #55889

msau42 · 2017-11-16T17:28:38Z

I'm curious why this is affecting gke and not gce.

msau42 · 2017-11-16T17:30:43Z

cluster/gce/config-default.sh

+# Format of this variable will be "#,scsi/nvme,block/fs" you can specify multiple
+# configurations by seperating them by a semi-colon ex. "2,scsi,fs;1,nvme,block"
+# is a request for 2 SCSI formatted and mounted SSDs and 1 NVMe block device SSD.
+NODE_LOCAL_SSDS_EXT=${NODE_LOCAL_SSDS_EXT:-""}


I wonder if it's because on gce it's defaulted to "", but on GKE it's unset.

Yeah I found this error in the master logs:

/home/kubernetes/bin/configure-helper.sh: line 98: NODE_LOCAL_SSDS_EXT: unbound variable

We keep running into this every time a new var is added, because GKE does not use config-default.sh at all, and configure-helper.sh uses nounset.

@enisoc how has this been fixed in the past? I was just going to add a 'NODE_LOCAL_SSDS_EXT=${NODE_LOCAL_SSDS_EXT:-""}' to configure-helper.sh but it feels like maybe there might be a better solution

@davidz627

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Revert "Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID" Reverts #53466 @mindprince @davidz627

mikedanese · 2017-11-16T21:10:00Z

cluster/gce/gci/configure-helper.sh

+  local format="${2}"
+
+  localdisknum=0
+  if [[ ! -z "${NODE_LOCAL_SSDS_EXT}" ]]; then


add a :- to this, e.g. "${NODE_LOCAL_SSDS_EXT:-}"

mikedanese · 2017-11-16T21:10:24Z

cluster/gce/gci/configure-helper.sh

+    safe-format-and-mount "${ssd}" "${mountpoint}"
+    # We only do the bindmount if users are using the new local ssd request method
+    # see https://github.com/kubernetes/kubernetes/pull/53466#discussion_r146431894
+    if [[ ! -z "${NODE_LOCAL_SSDS_EXT}" ]]; then


:- here as well

mikedanese · 2017-11-16T21:10:52Z

cluster/gce/util.sh

@@ -37,6 +39,11 @@ else
  exit 1
 fi

+if [[ ${NODE_LOCAL_SSDS} -ge 1 ]] && [[ ! -z ${NODE_LOCAL_SSDS_EXT} ]] ; then


:- here as well

mikedanese · 2017-11-16T21:11:56Z

cluster/gce/util.sh

@@ -587,6 +617,19 @@ function create-node-template() {
  fi

  local local_ssds=""
+  local_ssd_ext_count=0
+  if [[ ! -z ${NODE_LOCAL_SSDS_EXT-""} ]]; then


instead of -"", use :-

Automatic merge from submit-queue (batch tested with PRs 54824, 55911, 55730, 55979, 55961). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID Fixes: #51431 Fixed version of: #53466 Mount SCSI local SSD by UUID in /mnt/disks/by-uuid/, also allows for users to request and mount NVMe disks. Both types of disks will be accessible either through block or file-system. I have confirmed that it is no longer crashing when nodes are initialized on GKE.

k8s-github-robot assigned zmerlynn and gmarek Oct 4, 2017

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 4, 2017

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Oct 4, 2017

msau42 reviewed Oct 4, 2017

View reviewed changes

msau42 reviewed Oct 5, 2017

View reviewed changes

k8s-ci-robot assigned msau42 Oct 5, 2017

msau42 reviewed Oct 6, 2017

View reviewed changes

msau42 reviewed Oct 9, 2017

View reviewed changes

davidz627 changed the title ~~Mount SCSI local SSD by UUID in /mnt/disks/by-uuid/~~ Add options for mounting SCSI or NVMe local SSD though Block or Filesystem Oct 9, 2017

davidz627 mentioned this pull request Oct 9, 2017

Added bind-mounting to the by-uuid directory for local SSD. Refactored for easy addition of block devices and NVMe devices. #53623

Closed

davidz627 force-pushed the localSSDUUID branch from ab91a2e to 0215c4c Compare October 12, 2017 23:07

davidz627 changed the title ~~Add options for mounting SCSI or NVMe local SSD though Block or Filesystem~~ Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID Oct 13, 2017

Set up alternate mount point for SCSI/NVMe local SSD by UUID in /mnt/…

0282582

…disks/by-uuid/, set up ability to have unformatted disk symlinked in /dev/disk/by-uuid/. Added tests. Preserved backwards compatibility.

davidz627 force-pushed the localSSDUUID branch from a76df6d to 0282582 Compare November 15, 2017 01:14

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 15, 2017

roberthbailey unassigned pipejakob, mikedanese and roberthbailey Nov 15, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 16, 2017

k8s-github-robot merged commit c3ed0f2 into kubernetes:master Nov 16, 2017

msau42 mentioned this pull request Nov 16, 2017

[local-volume] Update gce configmap with new local ssd naming scheme kubernetes-retired/external-storage#464

Closed

mikedanese mentioned this pull request Nov 16, 2017

Revert "Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID" #55889

Merged

msau42 reviewed Nov 16, 2017

View reviewed changes

mikedanese reviewed Nov 16, 2017

View reviewed changes

davidz627 mentioned this pull request Nov 17, 2017

Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID #55911

Merged

		@@ -140,6 +241,8 @@ function find-master-pd {
		MASTER_PD_DEVICE="/dev/disk/by-id/${relative_path}"
		}

Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID #53466

Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID #53466

Conversation

davidz627 commented Oct 4, 2017 • edited Loading

davidz627 commented Oct 4, 2017

msau42 commented Oct 4, 2017

msau42 commented Oct 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 commented Oct 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Oct 5, 2017

davidz627 commented Oct 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Do not "mkdir -p ${sym}" as that will cause unintended symlink behavior

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 commented Oct 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 commented Oct 10, 2017

davidz627 commented Oct 12, 2017

davidz627 commented Oct 13, 2017

zmerlynn commented Nov 14, 2017

davidz627 commented Nov 15, 2017 • edited Loading

davidz627 commented Nov 15, 2017

davidz627 commented Nov 15, 2017

zmerlynn commented Nov 16, 2017 via email

k8s-github-robot commented Nov 16, 2017

k8s-github-robot commented Nov 16, 2017

k8s-github-robot commented Nov 16, 2017

rohitagarwal003 commented Nov 16, 2017

mikedanese commented Nov 16, 2017

msau42 commented Nov 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 Nov 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 commented Oct 4, 2017 •

edited

Loading

davidz627 commented Nov 15, 2017 •

edited

Loading

davidz627 Nov 16, 2017 •

edited

Loading