Template Refactor

[benofben]

The templates have over a year of history and during that time have accumulated a lot of cruft. I'm working on a major rewrite with the goal of compartamentalizing logic so it is reusable. With that complete, we'll have a resvised simple template and then begin work on a multi data center template (capable of deploying n nodes in each of m data centers/regions).

To reduce parameters passed between sub-templates, I'm making a lot of assumptions. I'm going to use this issue to document that logic and thought process.

The first step is to create a template that is simple and creates an OpsCenter instance. To remove dependences, opsCenterNode.json will create its own storage account for the OpsCenter node (possibly nodes at a later date if we add HA). It will also always bind to the private IP 10.0.0.6. The vnet will be passed in as a parameter and the region to deploy to will be determined from where the vnet is.

So, we're presuming the existence of a vnet and associated subnets. Besides that, I think the only other parameter we need is username/pw.

[benofben]

I think I'm going to predeclare subnets 10.x.y.0/24 for y=0...9.

10.x.0.6 = OpsCenter Node
10.x.{1,2,3,4,5,6,7,8,9}.{6-45} = DSE nodes

Each time we call the node template I think we're going to specify x, y and some number of nodes to create which is less than 40. Multiple calls can then be made to the template (and want to wrap/hide that).

At any rate, don't have to worry about that yet. Better to just concentrate on the OpsCenter node question for now.

[benofben]

Just rewrote the opscenter.json template. I think all external dependencies are now gone with the exception of the subnet/vnet. I'm using resourceid and effectively treating subnet0 as a global variable. It's not great, but given that the subnet is going to need to be shared with the nodes, I think this is the best way.

There is an alternative that's sort of interesting.... Create a vnet for opscenter, another for each datacenter and then use gateways to link those together. It benefits from symmetry. Going to think about that a bit tonight and come back to it in the morning. The downside is more moving parts (lots of gateways).

It's also a bit tempting to create one great big subnet and stick everything in that, but then the creating of the nodes with the 10.0.0.6-40 like notation becomes difficult/impossible.

[benofben]

Would work like this:
opscenter vnet with 10.0.0.0/16 -- use only 10.0.0.6 for opscenter. Maybe add 10.0.0.7 for HA later. Gateways in this subnet too.

datacenter1 vnet with 10.1.0.0/16
10.1.0.0/24 would be subnet0 with gateways
10.1.1.0/24 would be subnet1 with 40 nodes
10.1.2.0/24 would be subnet2 with 40 nodes
...

datacenter2 vnet with 10.2.0.0/16
10.2.0.0/24 would be subnet0 with gateways
10.2.1.0/24 would be subnet1 with 40 nodes
10.2.2.0/24 would be subnet2 with 40 nodes
...

and so on with support for 256 datacenters and lots of nodes per dc (40*255=10200)

This way there's nice symmetry and OpsCenter is separate from the cluster nodes. Each datacenter has a single vnet and multiple subnets.

I know that on Azure 10.0.0.0 through 10.0.0.5 are reserved. I wonder if 10.1.1.1 is usable? It'd be nice to have these things start counting at .1 rather than .6

[benofben]

Just realized my idea of specifying numberNodes and numberDataCenters isn't going to work. The node count works fine, but you need to somehow specify the location for each DC. You effectively need to pass in a list of dc locations with len(list) determining number of DCs. I don't know of any way to do that in the ARM language. We may end up just hard coding two DCs for now.

[benofben]

I'd also really like to move from having a fixed number of nodes (4,12,36,90) to being able to specify an arbitrary number. However, that requires either fixing the storage account limitation or having the ability to do some modulo logic, ie I request 100 nodes, so I provision subnet1 with 40, subnet2 with 40 and subnet3 with 20. In the previous version of the template the equivalent logic was hard coded, hence the fixed node counts.

One solution would be to write an ARM template generator in a language like Perl. This would be sufficiently expressive, but I can't thing of a way that a solution like that could work with the Azure Marketplace or the HTML Deploy to Azure buttons.

Another solution would be to further extend the ARM language to include a more complete set of control functions. However, a markup language like json probably isn't the best place for control functions.

The best solution I can think of would be to use an existing programming language and either expose an API it can call (ie a C# .Net API or Python or...) or expose Azure as a REST API that could be orchestrated in a programming language. In either case, Marketplace would need new hooks (unless something exists that I'm unaware of) to allow a use interaction like deploying a template to call the code.

It seems like what I'm trying to do might not be possible in the ARM language as it exists today. I think to get things going, I'm just going to write a sh script that invokes the ARM templates the correct number of times and wraps all this. We can then introduce multi-dc to the Marketplace when alternative support is there.

[benofben]

copyIndex gets you part way there. There is a good example of it here (we also use it in node.json in the simple template): https://alexandrebrisebois.wordpress.com/2015/06/03/use-the-azure-resource-manager-copy-operation-to-deploy-duplicates/

However, I think you need conditional logic as well to make this work, as well as modulo arithmetic.

Then, beyond that you need the ability to pass in lists or arrays.

[benofben]

I think I know how to deal with this now. We're going to ignore the marketplace for the time being, and assume that an appropriate UI can be built once the template is in place. Further, we're going to add a python preprocessor. This python will take a json as input which defines the cluster. I'm going to take a top down approach and define that input json initially.

The python will then generate a largely static ARM template using a variety of subscripts and json files. There will be no need for copy index or anything similar and arbitrary control structures can be added because python is Turing complete.

[benofben]

I can't quite decide if this is going to output one big json template or a tree of them. In favor of one big one --- the syntax is very verbose and putting it all in one avoids the verbose function call syntax. Against that idea --- compartmentalization.

[benofben]

Looks like I'm going the generate one big template route. That makes handling the python for this a lot simpler.

One pain point I noticed --- the scripts seem to be referenced using a URL. As such, I believe they're going to need to be hosted somewhere. I'm going to try using a relative ./scripts/dsenode.sh like reference, but I suspect it's not going to work.

[benofben]

Confirmed that relative paths do not work for the scripts. Other than that, everything seems to be working as expected. I think I'm on track to get this done in another 10 hours or so. A type checker would make this work a lot easier.

[benofben]

Multi DC seems to have come up successfully. I'm now concentrating on the scripts (what Azure terms an extension). I'd broken them into a bunch of granular scripts, but invocation doesn't seem to work. Not sure what's going on there.

Also got some info from the OpsCenter team on json format for the rest provision. There's an example here: https://datastax.jira.com/wiki/pages/viewpage.action?spaceKey=~ryan.springer&title=MultiDcProvisioningRequestJSON

[benofben]

We're a lot further along. The nodes all seem to provision most of the time, though we are seeing intermittent errors.

After that, it's possible to ssh to the OpsCenter node and run a wget to get the python script. Running that script generates a provision.json that can be passed to OpsCenter. However, OpsCenter is hanging on the configure. Looking at the logs shows repeated connection attempts.

I'm pursuing the issue with engineering. Here's a screenshot:

There's an additional intermittent issue --- the Azul repo is now timing out intermittently. We've previous tried the Canonical and Oracle repos as well. One solution would be nightly builds of VM images. We are open to other suggestions. Downloading 10s-100s of JVM simultaneously seems an unreliable proposition.

[benofben]

The configuring issue turns out to be down to a missing "rack" tag in the json for provision.json. That is resolved now and we are well situated to add rack awareness.

The intermittent Azul issue seems to have gone away.

Canonical seems to have changed something with their repos and we have apt-get commands failing. In the last day packages to be updated went from 0 to 70 some. Mahesh and I had a long talk about using Docker containers and Docker Hub to work around this in the future. We're going to want to reconnect with the Docker team on this.

We are also working on resolving issues in both the Azure connection and Azure extension.

Finally, network performance seems highly latent and unreliable between DCs (250ms ping). We need to better understand that.

[benofben]

From a coding perspective, the refactor is mostly complete. I still need to add dependencies for the OpsCenter extension, so it doesn't run before all VMs and connections are complete.

We have open issues with:

• Custom extension provisioning
• Gateway provisioning
• Provisioning large numbers of connections in parallel

[benofben]

I've added the connection and VM dependencies. Full end to end seems to work now. We may still have an intermittent issue with extensions, but I'm having trouble recreating it right now. Likewise, there is an intermittent connection issue.

That said, the refactor work is done and only debugging and building a UI remains.

Here's a screenshot: