packages: update to Golang 1.8.5; docs: add runbook of distinguish lo…

…gs; integration: comment flavor stuffs
cloudfoundry · Nov 28, 2017 · 05a4b0b · 05a4b0b
1 parent f79f2fe
commit 05a4b0b
Show file tree

Hide file tree

Showing 10 changed files with 142 additions and 73 deletions.
diff --git a/config/blobs.yml b/config/blobs.yml
@@ -1,9 +1,8 @@
----
-golang_1.7.1/go1.7.1.linux-amd64.tar.gz:
-  object_id: 48ee2c2c-c7eb-4c31-ab67-c59ad4a4a478
-  sha: 919ab01305ada0078a9fdf8a12bb56fb0b8a1444
-  size: 81618401
-golang_1.7.1/go1.7.1.darwin-amd64.tar.gz:
-  object_id: f3007357-2404-4749-a83f-7273c208536a
-  sha: f33d5bacf2c35fe59ef76e8fc0d87d27ba0432a3
-  size: 81509752
+golang/go1.8.5.darwin-amd64.tar.gz:
+  size: 97396903
+  object_id: bea7897b-3554-4e92-7012-70a66c351081
+  sha: 16f4b97fe7232525ab1c2ee5ce595c56014a65b2
+golang/go1.8.5.linux-amd64.tar.gz:
+  size: 99077377
+  object_id: 201c94b8-019c-4a73-79d7-f3f097d7406f
+  sha: 71d2e29b7ea053e411ed4ba6200db967481ddead
diff --git a/docs/distinguish_logs_thru_thread_ID.md b/docs/distinguish_logs_thru_thread_ID.md
@@ -0,0 +1,70 @@
+Distinguish logs thru thread ID
+
+# Motivation
+
+Currently log format is `[packageName] date time message` in the existing CPI. It is difficult to track and locate problems in CPI log file when multi-actions execute at the same time.
+For example A log snippet is below. There were 3 'create_vm' actions running and it was difficult to distinguish their process.
+
+```
+[softlayerGo] 2017/11/27 06:48:08 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Virtual_Guest_Block_Device_Template_Group/22345678.json?objectMask=mask%5Bid%2C+name%2C+globalIdentifier%2C+imageType%2C+accountId%5D
+[softlayerGo] 2017/11/27 06:48:08 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":1234567,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/32345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":1234567,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/42345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":1234567,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/42345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":42345678,"primarySubnetId":52345678,"networkSpace":"PUBLIC"}
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/62345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":42345678,"primarySubnetId":52345678,"networkSpace":"PUBLIC"}
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/62345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Parameters:
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":32345678,"primarySubnetId":62345678,"networkSpace":"PRIVATE"}
+[softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  POST https://api.softlayer.com/rest/v3.1/SoftLayer_Virtual_Guest.json
+```
+
+The softlayer CPI supports a function to distinguish every actions by thread id.
+
+# Details
+
+The thread_id was generated in 9 digits when invoke cpi action and added to the log prefix. So the log format in softlayer_ng is `[thread_id:packageName] date time message`.
+So it could be used to distinguish in each actions. For example, in this picture, some logs could be gathered by the number beginning with 9383 in a ‘create_vm’ action.
+We can get rid of logs generated by others actions.
+
+```
+[938349194:softlayerGo] 2017/11/27 06:48:08 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Virtual_Guest_Block_Device_Template_Group/22345678.json?objectMask=mask%5Bid%2C+name%2C+globalIdentifier%2C+imageType%2C+accountId%5D
+[938349194:softlayerGo] 2017/11/27 06:48:08 [DEBUG] Parameters:
+[810766244:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":12345678,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[810766244:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/32345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[810766244:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[906216319:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":12345678,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[906216319:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/42345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[906216319:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":12345678,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/42345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":42345678,"primarySubnetId":52345678,"networkSpace":"PUBLIC"}
+[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/62345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Parameters:
+[906216319:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":42345678,"primarySubnetId":52345678,"networkSpace":"PUBLIC"}
+[906216319:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/62345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+[906216319:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Parameters:
+[810766244:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":32345678,"primarySubnetId":72345678,"networkSpace":"PRIVATE"}
+```
+
+And we could use `grep -nr cpi.stderr.log -e 938349194 > 938349194.log` to grep specified CPI process log.
+
+```
+1:[938349194:softlayerGo] 2017/11/27 06:48:08 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Virtual_Guest_Block_Device_Template_Group/22345678.json?objectMask=mask%5Bid%2C+name%2C+globalIdentifier%2C+imageType%2C+accountId%5D
+2:[938349194:softlayerGo] 2017/11/27 06:48:08 [DEBUG] Parameters:
+9:[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Response:  {"accountId":12345678,"id":22345678,"name":"light-bosh-stemcell-3468.11-softlayer-xen-ubuntu-trusty-go_agent","globalIdentifier":"557ba6ac-6e5d-4052-bb9f-a732067bdefc","imageType":{"description":"a disk that may be replaced on upgrade","keyName":"SYSTEM","name":"System"}}
+10:[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/42345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+11:[938349194:softlayerGo] 2017/11/27 06:48:09 [DEBUG] Parameters:
+12:[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Response:  {"id":42345678,"primarySubnetId":52345678,"networkSpace":"PUBLIC"}
+13:[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Request URL:  GET https://api.softlayer.com/rest/v3.1/SoftLayer_Network_Vlan/62345678.json?objectMask=mask%5Bid%2CprimarySubnetId%2CnetworkSpace%5D
+14:[938349194:softlayerGo] 2017/11/27 06:48:10 [DEBUG] Parameters:
+```
diff --git a/packages/bosh_softlayer_cpi/packaging b/packages/bosh_softlayer_cpi/packaging
@@ -5,7 +5,7 @@ set -u # report the usage of uninitialized variables
 PACKAGES_DIR=${BOSH_PACKAGES_DIR:-/var/vcap/packages}
 
 # Set Golang dependency
-export GOROOT=$(cd "${PACKAGES_DIR}/golang_1.7.1" && pwd -P)
+export GOROOT=$(cd "${PACKAGES_DIR}/golang" && pwd -P)
 export PATH=${GOROOT}/bin:${PATH}
 
 # Build BOSH SoftLayer CPI package

diff --git a/packages/bosh_softlayer_cpi/spec b/packages/bosh_softlayer_cpi/spec
@@ -2,7 +2,7 @@
 name: bosh_softlayer_cpi
 
 dependencies:
-- golang_1.7.1
+- golang
 
 files:
 - bosh-softlayer-cpi/**/*
diff --git a/packages/golang_1.7.1/packaging → packages/golang/packaging b/packages/golang_1.7.1/packaging → packages/golang/packaging
@@ -3,10 +3,10 @@ set -u # report the usage of uninitialized variables
 
 # Grab the latest versions that are in the directory
 PLATFORM=`uname | tr '[:upper:]' '[:lower:]'`
-GOLANG_VERSION=`ls -r golang_1.7.1/go*.${PLATFORM}-* | sed "s/golang_1.7.1\/go\(.*\)\.${PLATFORM}-amd64.tar.gz/\1/" | head -1`
+GOLANG_VERSION=`ls -r golang/go*.${PLATFORM}-* | sed "s/golang\/go\(.*\)\.${PLATFORM}-amd64.tar.gz/\1/" | head -1`
 
 # Extract Go Programming Language package
-tar xzvf ${BOSH_COMPILE_TARGET}/golang_1.7.1/go${GOLANG_VERSION}.${PLATFORM}-amd64.tar.gz
+tar xzvf ${BOSH_COMPILE_TARGET}/golang/go${GOLANG_VERSION}.${PLATFORM}-amd64.tar.gz
 
 # Copy Go Programming Language package
 mkdir -p ${BOSH_INSTALL_TARGET}/bin

diff --git a/packages/golang/spec b/packages/golang/spec
@@ -0,0 +1,8 @@
+---
+name: golang
+
+dependencies:
+
+files:
+  - golang/go1.8.5.linux-amd64.tar.gz
+  - golang/go1.8.5.darwin-amd64.tar.gz
diff --git a/packages/golang_1.7.1/spec b/packages/golang_1.7.1/spec
diff --git a/src/bosh-softlayer-cpi/integration/config.go b/src/bosh-softlayer-cpi/integration/config.go
@@ -96,10 +96,10 @@ var (
 	outLogger    = log.New(multiWriter, "", log.LstdFlags)
 	errLogger    = log.New(os.Stderr, "", log.LstdFlags)
 
-	cpiLogger   = cpiLog.New(boshlog.LevelDebug, "Integration", outLogger, errLogger)
-	multiLogger = api.MultiLogger{Logger: cpiLogger, LogBuff: &logBuffer}
-	uuidGen     = uuid.NewGenerator()
-	sess        *session.Session
+	cpiLogger       = cpiLog.New(boshlog.LevelDebug, "Integration", outLogger, errLogger)
+	multiLogger     = api.MultiLogger{Logger: cpiLogger, LogBuff: &logBuffer}
+	uuidGen         = uuid.NewGenerator()
+	sess            *session.Session
 	softlayerClient = &client.ClientManager{}
 )
 

diff --git a/src/bosh-softlayer-cpi/integration/vm_test.go b/src/bosh-softlayer-cpi/integration/vm_test.go
@@ -128,52 +128,52 @@ var _ = Describe("VM", func() {
 		assertSucceeds(request)
 	})
 
-	It("can create a VM by flavor", func() {
-		var vmCID string
-		By("creating a VM by flavor")
-		request := fmt.Sprintf(`{
-		  "method": "create_vm",
-		  "arguments": [
-		    "45632666-9fb1-422a-af35-2ab6102c5c1b",
-		    "%v",
-		    {
-		      "hostname_prefix": "bluebosh-slcpi-integration-test",
-		      "domain": "softlayer.com",
-		      "flavor_key_name": "B1_1X2X25",
-		      "ephemeral_disk_size": 25,
-		      "max_network_speed": 100,
-		      "hourly_billing_flag": true,
-		      "dedicated_account_host_only_flag": false,
-		      "datacenter": "lon02"
-		    },
-		    {
-		      "default": {
-		        "type": "dynamic",
-		        "dns": [
-			      "8.8.8.8"
-		        ],
-		        "default": [
-		          "dns",
-		          "gateway"
-		        ],
-		        "cloud_properties": {
-		          "vlan_ids": [1292653, 1292651]
-		        }
-		      }
-		    },
-		    null,
-		    {}
-		  ]
-		}`, existingStemcellId)
-		vmCID = assertSucceedsWithResult(request).(string)
-
-		By("deleting the VM")
-		request = fmt.Sprintf(`{
-		  "method": "delete_vm",
-		  "arguments": ["%v"]
-		}`, vmCID)
-		assertSucceeds(request)
-	})
+	//XIt("can create a VM by flavor", func() {
+	//	var vmCID string
+	//	By("creating a VM by flavor")
+	//	request := fmt.Sprintf(`{
+	//	  "method": "create_vm",
+	//	  "arguments": [
+	//	    "45632666-9fb1-422a-af35-2ab6102c5c1b",
+	//	    "%v",
+	//	    {
+	//	      "hostname_prefix": "bluebosh-slcpi-integration-test",
+	//	      "domain": "softlayer.com",
+	//	      "flavor_key_name": "B1_1X2X25",
+	//	      "ephemeral_disk_size": 25,
+	//	      "max_network_speed": 100,
+	//	      "hourly_billing_flag": true,
+	//	      "dedicated_account_host_only_flag": false,
+	//	      "datacenter": "lon02"
+	//	    },
+	//	    {
+	//	      "default": {
+	//	        "type": "dynamic",
+	//	        "dns": [
+	//		      "8.8.8.8"
+	//	        ],
+	//	        "default": [
+	//	          "dns",
+	//	          "gateway"
+	//	        ],
+	//	        "cloud_properties": {
+	//	          "vlan_ids": [1292653, 1292651]
+	//	        }
+	//	      }
+	//	    },
+	//	    null,
+	//	    {}
+	//	  ]
+	//	}`, existingStemcellId)
+	//	vmCID = assertSucceedsWithResult(request).(string)
+	//
+	//	By("deleting the VM")
+	//	request = fmt.Sprintf(`{
+	//	  "method": "delete_vm",
+	//	  "arguments": ["%v"]
+	//	}`, vmCID)
+	//	assertSucceeds(request)
+	//})
 
 	It("can create a VM with static ip", func() {
 		By("creating a VM with static ip")
@@ -199,7 +199,7 @@ var _ = Describe("VM", func() {
 		      "default": {
 				"cloud_properties": {
 				  "vlan_ids": [
-					524954
+					1292651
 				  ]
 				},
 				"dns": [

diff --git a/src/bosh-softlayer-cpi/softlayer/virtual_guest_service/softlayer_ubuntu_net_manager.go b/src/bosh-softlayer-cpi/softlayer/virtual_guest_service/softlayer_ubuntu_net_manager.go
@@ -116,7 +116,7 @@ func (u *Softlayer_Ubuntu_Net) NormalizeDynamics(networkComponents datatypes.Vir
 		}
 	}
 
-	if publicDynamic == nil && networkComponents.PrimaryNetworkComponent.NetworkVlan.Id != nil {
+	if publicDynamic == nil && networkComponents.PrimaryNetworkComponent.NetworkVlan != nil && networkComponents.PrimaryNetworkComponent.NetworkVlan.Id != nil {
 		networks["generated-public"] = Network{
 			Type: "dynamic",
 			IP:   *(networkComponents.PrimaryNetworkComponent.PrimaryIpAddress),