Discussion - Compositional Limits / Improvements

[robrankin]

Limitations

The Module pattern allows for a certain amount of code reuse, but there are still limits, specifically surrounding sub-resources and resource configuration blocks and changing/overriding those.

Assume a setup as such:

├── modules
│   └── virtualMachine
│       └── linux
│           ├── output.tf
│           ├── variables.tf
│           └── virtualMachine.tf
└── profiles
    └── vmtype
        └── vm.tf

Example 1 - Base VM Resource, Linux, no data disks

/modules/virtualMachine/linux/virtualMachine.tf:

resource "azurerm_virtual_machine" "virtualMachine" {
    count                         = "${var.count}"
    name                          = "${var.vmName}${format("%02d",count.index)}"
    location                      = "${var.location}"
    resource_group_name           = "${var.resourceGroupName}"
    availability_set_id           = "${var.availabilitySetId}"
    network_interface_ids         = ["${element(var.networkInterfaceIds, count.index)}"]
    vm_size                       = "${var.vmSize}"
    delete_os_disk_on_termination = "true"

    storage_image_reference {
        publisher = "${var.imagePublisher}"
        offer     = "${var.imageOffer}"
        sku       = "${var.imageSku}"
        version   = "${var.imageVersion}"
    }

    storage_os_disk {
        name          = "${var.vmName}${format("%02d",count.index)}-os"
        vhd_uri       = "${var.vmName}${format("%02d",count.index)}-os.vhd"
        caching       = "ReadWrite"
        create_option = "FromImage"
    }

    os_profile {
        computer_name  = "${var.vmName}${format("%02d",count.index)}"
        admin_username = "${var.adminUsername}"
        admin_password = "${var.adminPassword}"
    }

    os_profile_linux_config {
        disable_password_authentication = true
        ssh_keys {
            path     = "/home/${var.adminUsername}/.ssh/authorized_keys"
            key_data = "${file("~/.ssh/id_rsa.pub")}"
        }
    }
}

This resource is only usable as a Linux VM with no data disks attached.

You cannot use an override file here, as that override file is applied whenever you use this base resource in a module (i.e. the override file exists in the base resource directory, therefore it is applied when the module sources the base resource directory)

I’d like to be able to use the following base VM template, and then add to that using modules (or some other agreed approach), as detailed in Example 2.1 / 2.2

Example 2.1: Base VM Resource, no data disks, no specific OS config

Setup now more generic (no linux specificity), as such:

├── modules
│   └── virtualMachine
│           ├── output.tf
│           ├── variables.tf
│           └── virtualMachine.tf
└── profiles
    └── vmtype
        └── vm.tf

/modules/virtualMachine/virtualMachine.tf:

resource "azurerm_virtual_machine" "virtualMachine" {
    count                         = "${var.count}"
    name                          = "${var.vmName}${format("%02d",count.index)}"
    location                      = "${var.location}"
    resource_group_name           = "${var.resourceGroupName}"
    availability_set_id           = "${var.availabilitySetId}"
    network_interface_ids         = ["${element(var.networkInterfaceIds, count.index)}"]
    vm_size                       = "${var.vmSize}"
    delete_os_disk_on_termination = "true"

    storage_image_reference {
        publisher = "${var.imagePublisher}"
        offer     = "${var.imageOffer}"
        sku       = "${var.imageSku}"
        version   = "${var.imageVersion}"
    }

    storage_os_disk {
        name          = "${var.vmName}${format("%02d",count.index)}-os"
        vhd_uri       = "${var.vmName}${format("%02d",count.index)}-os.vhd"
        caching       = "ReadWrite"
        create_option = "FromImage"
    }

    os_profile {
        computer_name  = "${var.vmName}${format("%02d",count.index)}"
        admin_username = "${var.adminUsername}"
        admin_password = "${var.adminPassword}"
    }
 }

Example 2.2 - Extract Linux OS Config with multiple data disks below.

I’d then like to be able to alter the underlying template sourced by a module. This example can add the OS specific os_profile_linux_config, and it can add additional config for the data disks.

/profiles/vmtype/vm.tf:

module "specificVMType" {
    source  = "../../modules/virtualMachine"

    resource "azurerm_virtual_machine" "virtualMachine" {
        os_profile_linux_config {
            disable_password_authentication = true
            ssh_keys {
                path     = "/home/${var.adminUsername}/.ssh/authorized_keys"
                key_data = "${file("~/.ssh/id_rsa.pub")}"
            }
        }
        storage_data_disk {
          name          = "datadisk0"
          vhd_uri       = "datadisk0.vhd"
          disk_size_gb  = "1023"
          create_option = "empty"
          lun           = 0
        }
        storage_data_disk {
          name          = "datadisk1"
          vhd_uri       = "datadisk1.vhd"
          disk_size_gb  = "1023"
          create_option = "empty"
          lun           = 1
        }
    }
}

Basically, I’d like modules to be able to apply some form of overrides to the resource they are sourcing. Syntax and approach completely open for debate, above examples simply used to try and highlight what I’d like to see.

Is there anything like this available today, or being planned?

[jen20]

Hi @robrankin! The [list and map] types which are now available as module-level variables are the start of work to support situations similar to this one, however I think they will need more work to complete. Examples such as this are useful for guiding this so thank you for opening this issue!

Something that is perhaps not well documented, and is probably unclear unless you are familiar with the source for HCL, the language from which it is derived (UCL), or the source for the specific resource in question is that multiple storage_data_disk blocks actually define a list of maps, but where the schema of the maps is known ahead of time. There was a discussion here about the idea of nestable resources, at which point we could probably achieve this using a combination of count and interpolation. Once we iron out issues coming from the upgrade from 0.6.x to 0.7 we'll start to look more into being able to implement this.

Consequently I'm going to leave this issue open - if others have additional examples of how they would like to use the module system we are certainly open to hearing those and considering them as part of future design. Thanks!

[robrankin]

In the Azure RM world storage_data_disk is just a configuration argument of the virtualMachine resource, whereas (for example) a NIC is actually a separate resource (and is treated as a sub-resource once connected to a VM).

Does that distinction exist in TF at all? Or are the various configuration arguments of a resource essentially the equivalent of a sub-resource (or nested resource)?

Currently some of the configuration override/adding/changing I want to accomplish can be done with sub resources if they're exposed as top level resources (i.e. see virtual networks and subnets), but even then the normal resource arguments of that aren't exposed.

If the nested resources work would expose the resources arguments as resources that can managed separately, that might do the trick for me.

p.s I tried passing lists of maps into various resources and using them with arguments, and it causes all sorts of funky error messages 😄

i.e. storage_data_disk = "${var.listOfMapsOfDisks}"

[mildwonkey]

Terraform 0.12 is the start of having a more composable language and we'll continue to improve it over time.
I am going to close this particular issue as we have addressed at least some of the concerns. If you (or anyone!) have any specific use-cases not covered, please open a new feature request!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.