Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scale for Azure Stack Kubernetes node pool fails #4

Closed
ned1313 opened this Issue Feb 20, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@ned1313
Copy link

ned1313 commented Feb 20, 2019

Is this a request for help?: Yes


Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE


What version of acs-engine?: Version: canary
GitCommit: 0d15f88
GitTreeState: clean


Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened: When attempting to scale the node pool from 3 to 4 worker nodes, the scale action fails with error: panic: runtime error: invalid memory address or nil pointer dereference

What you expected to happen: I expected a fourth worker node to be added to the node pool

How to reproduce it (as minimally and precisely as possible): Deploy the Azure Stack Kubernetes template. Once deployment is complete, ssh into the vmd deployment machine and run the following for the acs-engine

#Values will come from the deployment log, which should be here
LOG_FILE=/var/log/azure/acsengine-kubernetes-dvm.log
ACS_ENGINE_BIN=/var/lib/waagent/custom-script/download/0/acs-engine/bin

RESOURCE_GROUP=$(grep "RESOURCE_GROUP_NAME" $LOG_FILE | awk '{print $2}')
SUBSCRIPTION_ID=$(grep "TENANT_SUBSCRIPTION_ID" $LOG_FILE | awk '{print $2}')
LOCATION=$(grep "REGION_NAME" $LOG_FILE | awk '{print $2}')
DEPLOYMENT_DIR=$(grep "Templates output directory is" $LOG_FILE | awk '{print $NF}')
NODE_POOL=$(jq -r .properties.agentPoolProfiles[0].name $DEPLOYMENT_DIR/apimodel.json)
NEW_NODE_COUNT=$(grep "AGENT_COUNT" $LOG_FILE | awk '{print $2}')
DNS_PREFIX=$(jq -r .properties.masterProfile.dnsPrefix $DEPLOYMENT_DIR/apimodel.json)
VM_DNS_SUFFIX=$(jq -r .properties.cloudProfile.resourceManagerVMDNSSuffix $DEPLOYMENT_DIR/apimodel.json)
MASTER_FQDN="$DNS_PREFIX.$LOCATION.$VM_DNS_SUFFIX"
CLIENT_ID=$(jq -r .parameters.servicePrincipalClientId.value $DEPLOYMENT_DIR/azuredeploy.parameters.json)
CLIENT_SECRET=$(jq -r .parameters.servicePrincipalClientSecret.value $DEPLOYMENT_DIR/azuredeploy.parameters.json)
AZURE_ENV=$(jq -r .parameters.targetEnvironment.value $DEPLOYMENT_DIR/azuredeploy.parameters.json)

let "NEW_NODE_COUNT++"

#Verify properties
echo "RESOURCE_GROUP: $RESOURCE_GROUP"
echo "SUBSCRIPTION_ID: $SUBSCRIPTION_ID"
echo "LOCATION: $LOCATION"
echo "DEPLOYMENT_DIR: $DEPLOYMENT_DIR"
echo "NODE_POOL: $NODE_POOL"
echo "NEW_NODE_COUNT: $NEW_NODE_COUNT"
echo "DNS_PREFIX: $DNS_PREFIX"
echo "VM_DNS_SUFFIX: $VM_DNS_SUFFIX"
echo "MASTER_FQDN: $MASTER_FQDN"
echo "CLIENT_ID: $CLIENT_ID"
echo "CLIENT_SECRET: $CLIENT_SECRET"
echo "AZURE_ENV: $AZURE_ENV"

#Invoke the acs-engine scale
$ACS_ENGINE_BIN/acs-engine scale --subscription-id $SUBSCRIPTION_ID \
    --resource-group $RESOURCE_GROUP  --location $LOCATION \
    --deployment-dir $DEPLOYMENT_DIR --new-node-count $NEW_NODE_COUNT \
    --node-pool $NODE_POOL --master-FQDN $MASTER_FQDN \
    --azure-env $AZURE_ENV \
    --auth-method client_secret \
    --client-id $CLIENT_ID \
    --client-secret $CLIENT_SECRET

Anything else we need to know: This is using the Azure AD auth method. I have not tested with ADFS. Here is a full error log when I enable debugging:

INFO[0000] validating...
INFO[0000] Writing cloud profile to: /tmp/azurestackcloud.json068758029
INFO[0000] AzureEnvironment: AzureStackCloud
DEBU[0000] Resolving tenantID for subscriptionID: 39c86ab5-8ac8-422b-b570-5fc6092c8702

DEBU[0000] Already registered for "Microsoft.Compute"
DEBU[0000] Already registered for "Microsoft.Storage"
DEBU[0000] Already registered for "Microsoft.Network"
INFO[0001] Name suffix: 34239724
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1269d4d]

goroutine 1 [running]:
github.com/Azure/acs-engine/pkg/armhelpers/azurestack/converter.ConvertVirtualMachine(0x0, 0x0, 0xc420152050, 0xc420020980, 0x0, 0x0, 0xc4200223c0, 0xc4200223d0, 0xc420022570, 0xc420022300, ...)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/pkg/armhelpers/azurestack/converter/compute.go:79 +0x2cd
github.com/Azure/acs-engine/pkg/armhelpers/azurestack/converter.ConvertVirtualMachineSlice(0xc420174700, 0x7, 0x9, 0x0, 0x0, 0x0)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/pkg/armhelpers/azurestack/converter/compute.go:110 +0x146
github.com/Azure/acs-engine/pkg/armhelpers/azurestack.VirtualMachineListResultPageClient.Values(0xc42052d5e0, 0xc4200ea090, 0xc4200201c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/pkg/armhelpers/azurestack/compute.go:36 +0x52
github.com/Azure/acs-engine/cmd.(*scaleCmd).run(0xc420383ce0, 0xc4203a8900, 0xc4203b2000, 0x0, 0x13, 0x0, 0x0)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/cmd/scale.go:230 +0x254f
github.com/Azure/acs-engine/cmd.newScaleCmd.func1(0xc4203a8900, 0xc4203b2000, 0x0, 0x13, 0x0, 0x0)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/cmd/scale.go:70 +0x52
github.com/Azure/acs-engine/vendor/github.com/spf13/cobra.(*Command).execute(0xc4203a8900, 0xc420359e00, 0x13, 0x14, 0xc4203a8900, 0xc420359e00)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/vendor/github.com/spf13/cobra/command.go:647 +0x3e4
github.com/Azure/acs-engine/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420363b00, 0xc4203a86c0, 0xc4203a8480, 0xc4203a8240)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/vendor/github.com/spf13/cobra/command.go:726 +0x2d4
github.com/Azure/acs-engine/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420363b00, 0xc4200ba008, 0x12b1c03)
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/vendor/github.com/spf13/cobra/command.go:685 +0x2b
main.main()
        /home/honcaou2/acs-engine/src/github.com/Azure/acs-engine/main.go:12 +0x74
@jadarsie

This comment has been minimized.

Copy link
Member

jadarsie commented Feb 20, 2019

Hello Ned, scale and upgrade are not supported on Azure Stack yet.
And that error message could be better ...

We are actively working on adding support for those two commands. Do not have an ETA though.

@ned1313

This comment has been minimized.

Copy link
Author

ned1313 commented Feb 20, 2019

Thanks @jadarsie! That is good to know. If you need some help working through it, or a guinea pig to test builds, let me know. I don't know much Go, but if you point me in the right direction, I can try.

@jadarsie jadarsie closed this Feb 21, 2019

@jadarsie

This comment has been minimized.

Copy link
Member

jadarsie commented Feb 21, 2019

Sorry, I did not mean to close before replying.
Because of the acs-engine to aks-engine migration, we are not quite ready yet to take contributions.
The work we are doing now should hopefully simplify contributions.
Thanks for posting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.