Support new NODE_OS_DISTRIBUTION 'custom' on GCE

[yguo0905]

What this PR does / why we need it:

This PR allows us to run e2e tests against arbitrary OS images on GCE.

It will be cherry picked into 1.8, 1.9 and 1.10.

Release note:

Support new NODE_OS_DISTRIBUTION 'custom' on GCE.

/assign @dashpole

[dashpole]

do we need to allow this for master os distribution as well, or do those tests still use GCI for the master?

[yguo0905]

All tests use cos as master image. BTW, we should probably also remove ubuntu from master os some time later.

[dashpole]

Ok, great.

[yguo0905]

I might need to change this from CUSTOM to custom.

[dashpole]

/lgtm

[yguo0905]

@zmerlynn, please take a look. Thanks!

[yguo0905]

The new GKE tests (example) are failing because

W0319 13:42:13.835] 2018/03/19 13:42:13 process.go:150: Running: bash -c 
W0319 13:42:13.835] function log_dump_custom_get_instances() {
W0319 13:42:13.836]   if [[ $1 == "master" ]]; then
W0319 13:42:13.836]     return 0
W0319 13:42:13.836]   fi
W0319 13:42:13.836] 
W0319 13:42:13.837]   gcloud compute instances list '--project=gke-up-c1-4-glat-up-mas' '--filter=(metadata.created-by:*zones/us-central1-f/instanceGroupManagers/gke-test-9f571eac2b-default-pool-ab3bb5f7-grp)' '--format=get(name)'
W0319 13:42:13.837] }
W0319 13:42:13.837] export -f log_dump_custom_get_instances
W0319 13:42:13.837] # Set below vars that log-dump.sh expects in order to use scp with gcloud.
W0319 13:42:13.837] export PROJECT=gke-up-c1-4-glat-up-mas
W0319 13:42:13.838] export ZONE='us-central1-f'
W0319 13:42:13.838] export KUBERNETES_PROVIDER=gke
W0319 13:42:13.838] ./cluster/log-dump/log-dump.sh '/workspace/_artifacts'
W0319 13:42:13.838] Cannot operate on cluster using node os distro: custom

So this PR is needed for both GCE and GKE tests.

[yguo0905]

/assign @dchen1107

[dchen1107]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dashpole, dchen1107, yguo0905

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster/OWNERS [dchen1107]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yguo0905]

/retest

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 61124, 59537, 61235, 61258, 61114). If you want to cherry-pick this change to another branch, please follow the instructions here.