feature request: new built-in function gzip(string)

[glerchundi]

Maybe this is a little bit opinionated but here it is.

Recently I run into an issue with cloud-config (yes, again (: ), and I solved it with a gzip function. I use GCE for deploying my machines and the maximum size for metadata is 32KB. I was deploying a whole Kubernetes cluster using CoreOS and cloud-config but when I tried to create the kube-master (which requires a bunch of files -certs,keys,cfgs-) it suddenly failed: fileSizeTooLarge.

Probably the best way to create this kind of clusters is to refer to a google cloud storage which contains everything I need and load them when booting but:

• It's not as portable as a unique cloud-config file
• and, it has dependencies I would like to avoid

Having a gzip built-in interpolation function solved my problem, I'm using it like this:

resource "template_file" "kube-master-userdata" {
    filename = "${format("%s/%s", path.module, "files/kube-master.yml")}"
    vars {

        #
        # parameters
        #

        dns-service-ip   = "10.3.0.10"
        etcd-endpoints   = "${var.etcd-endpoints}"
        service-ip-range = "10.3.0.0/24"
        pod-network      = "10.2.0.0/16"
        fleet-metadata   = "${var.kube-master-tags}"

        #
        # files
        #

        ca-cert               = "${base64encode(gzip(file(var.file-ca-cert)))}"
        api-server-cert       = "${base64encode(gzip(file(var.file-kube-api-server-cert)))}"
        api-server-key        = "${base64encode(gzip(file(var.file-kube-api-server-key)))}"
        etcd-ca-cert-file     = "/etc/ssl/etcd/ca.pem"
        etcd-ca-cert          = "${base64encode(gzip(file(var.file-kube-master-etcd-ca-cert)))}"
        etcd-client-cert-file = "/etc/ssl/etcd/client.pem"
        etcd-client-cert      = "${base64encode(gzip(file(var.file-kube-master-etcd-client-cert)))}"
        etcd-client-key-file  = "/etc/ssl/etcd/client.key.pem"
        etcd-client-key       = "${base64encode(gzip(file(var.file-kube-master-etcd-client-key)))}"
    }
}

And then in the cloud-config template like this:

[...]

  - path: "${etcd-client-key-file}"
    permissions: "0644"
    encoding: "gzip+base64"
    content: |
      ${etcd-client-key}

[...]

WDYT, is it worth it?

[sachincab]

This is definitely needed especially with Kubernetes TLS assets.
Anyone able to workaround it using template_cloudinit_config which support gzip and base64_encode?

I could not able to get working with template_cloudinit_config.

Please let me know your thoughts or suggestions.

[sachincab]

Any plan to incorporate gzip support in template_file for upcoming terraform release?

[cemo]

We unfortunately need for TLS too. What is the current workaround without using storages?

[apparentlymart]

Hi all! Sorry for the long silence here.

Since Terraform v0.10.3 we have had the base64gzip function, which gzips the given data and returns it base64 encoded. Base64 encoding is the conventional way to deal with small binary data blobs in Terraform config, since strings in Terraform are required to be UTF-8 strings and so cannot safely transfer arbitrary binary data.

When a resource type in a provider expects a small binary data blob to be provided in configuration, it must be written to accept it in base64 encoding. For example, the aws_instance resource type has both user_data for UTF-8 text and user_data_base64 for raw binary data, so users can choose which one is most appropriate.

Since we now have a gzip function, I'm going to close this out. Sorry we didn't catch this back when preparing the v0.10.3 release.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.