Question: Kubernetes and a custom VNET

[mogthesprog]

Hi Guys,

Great to see this open sourced (and in go)! I'm looking at using this package to deploy production kubernetes clusters, however the inability to deploy to a custom VNET in another resource group is a bit of a problem for us. I was wondering why this isn't possible? is it a constraint inherited from the resources deployed by the generated template, or a feature that needs adding in code?

If it's a feature request in acs-engine then i'm happy to muck in and add this. If you can point me at the problem.

Cheers,

Morgan

[colemickens]

There's actually an example cluster definition that enables it: https://github.com/Azure/acs-engine/blob/master/examples/vnet/kubernetesvnet.json

However, I've not tested it and I've heard there might be a problem with it. If you want to give it a shot we can troubleshoot it if the cluster's not working at the end.

[mogthesprog]

Thanks for the speedy response!

Your response made me go back and have another look - it works!

So the reason i'd given up so easily was going off the warning on this page. I got an error, complaining that the resource (VNET) couldn't be found, even thought the IDs were correct, even though looking at the IDs, they were correct.

For me the issue was that i was deploying the cluster in a different region to the original vnet - oops.

I'll issue a pull request with the updated doc in a bit.

I'll close this now, thanks for the help. Do you have a roadmap or a list of tasks that need implementing? Happy to contribute.

Thanks again

[colemickens]

Can you please run some containers and make sure they start properly and let me know?

And docs PRs are hugely appreciated. We should get some sort of roadmap up for features we want to build. I'll make a note and try to add something soon.

[mogthesprog]

Yeah sure, i'll double check and post back here. Reopening...

[mogthesprog]

Yep, all seems to work.

I first deployed to one of our dev environments, which has a locked down vnet/subnets, that failed. I then created a temporary vnet to test this out with no custom NSG rules, all was good. :) Ran a few pods fine.

Would you guy be interested in http_proxy support? I may spend some time tomorrow looking at how to add proxy support here. Looking at the templates, some script actions are called, plus the docker config would need to include the proxy... Also depends how etcd cluster is initiated too, if that calls out to a registration service then that would need it too. I'll have a good dig in the morning.

Thanks again for your help

[colemickens]

Is http_proxy meaning something that docker knows about to be able to pull through a restrictive firewall?

I don't think we have any need for it, but I think it's something we would consider merging, depending on the size of the change, how much it would take to support it, etc.

In the current kubernetes configuration, there is a single master that has etcd running as a systemd unit.

[SorraTheOrc]

+1000 for docs patches :-)

---

Twitter: @rgardler

---

From: Cole Mickens notifications@github.com
Sent: Tuesday, November 15, 2016 6:08:14 PM
To: Azure/acs-engine
Subject: Re: [Azure/acs-engine] Question: Kubernetes and a custom VNET (#99)

Can you please run some containers and make sure they start properly and let me know?

And docs PRs are hugely appreciated. We should get some sort of roadmap up for features we want to build. I'll make a note and try to add something soon.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fissues%2F99%23issuecomment-260832524&data=02%7C01%7Cross.gardler%40microsoft.com%7Cac652325818d4f7391c208d40dc56d3f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636148588970668229&sdata=A3vFTvpv0xn9keV1xw0I0eZzTB412LGUYU1OnEXZby8%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAPRgDk9Wub8Ym-Dy0mIUcWK-w_lSi62ks5q-mWOgaJpZM4KzK0e&data=02%7C01%7Cross.gardler%40microsoft.com%7Cac652325818d4f7391c208d40dc56d3f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636148588970668229&sdata=SDc8KwoB88x46hESUd77ju7Iwi4LONCp5fTc7i8o%2F00%3D&reserved=0.

[mogthesprog]

Yeah as in proxy support for clusters that have to run in locked down networks, the same way we run our cluster on premise today. I wouldn't be surprised if we aren't the only ones interested in that support.

Thanks for the etcd notes. I only asked since on premise we run an etcd instance on each master (3 node masters) which runs in containers managed by the kubelet, so docker on those nodes have the proxy set. I guess in this case it's not needed :) We do this as the hyperkube instances for controller-manager/api-server/scheduler also run in containers managed by the kubelet too. I'll have a dig around this cluster in the morning to see how it's currently configured.

[mogthesprog]

Will make a PR for docs tomorrow. :)

[colemickens]

Reopening this, I've had multiple reports that things aren't actually working right. Another person reported the same issue, the cluster might appear to be okay, but containers aren't actually getting scheduled. There was also an assertion that the route table wasn't getting updated as expected.

[mogthesprog]

No worries, yeah so i had a strange issue whereby the agent VMs weren't
being created properly - complaining that the storage account didn't exist.
So i thought there may have been a dependson statement missing from the
VMAS but that wasn't the issue. Still not sure what is casuing it but it's
intermittent. For the moment i'm putting it down to DNS updates in
repeatedly creating/destroying storage accounts. When the VMs are created
successfully it all seems to work fine for me.

I'll repeat the test in a new empty VNET in a different RG and report back
on what i see.

On Sat, Nov 19, 2016 at 1:38 AM, Cole Mickens notifications@github.com
wrote:

Reopened #99 #99.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#99 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH0ngeHLLhQb216LUDYdgZKRM4FwngGPks5q_lMwgaJpZM4KzK0e
.

[anhowe]

VNET support is being fixed in #172

[colemickens]

This is now fixed by the merging of #172.