Etcdv3 backend. Send message larger than max.

[aneroid13]

Terraform Version

Terraform v0.12.20

Terraform Configuration Files

terraform {
  required_version = ">=0.12.20"
  required_providers {
    vault = ">=2.4"
    vsphere = ">=1.14"
    foreman = ">=1.0"
  }
  backend "etcdv3" {
    endpoints = ["etcd-srv.domain.local:2379"]
    lock = true
    prefix = "/dev/terraform/testzones/state/"
}
}

Debug Output

TF_LOG=1 terraform state push errored.tfstate
2020/08/04 13:37:19 [WARN] Invalid log level: "1". Defaulting to level: TRACE. Valid levels are: [TRACE DEBUG INFO WARN ERROR]
2020/08/04 13:37:19 [INFO] Terraform version: 0.12.20
2020/08/04 13:37:19 [INFO] Go runtime version: go1.12.13
2020/08/04 13:37:19 [INFO] CLI args: []string{"/usr/local/bin/terraform", "state", "push", "errored.tfstate"}
2020/08/04 13:37:19 [DEBUG] Attempting to open CLI config file: /home/gitlab-runner/.terraformrc
2020/08/04 13:37:19 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/08/04 13:37:19 [INFO] CLI command args: []string{"state", "push", "errored.tfstate"}
2020/08/04 13:37:19 [TRACE] Meta.Backend: BackendOpts.Config not set, so using settings loaded from init.tf:14,3-19
2020/08/04 13:37:19 [TRACE] Meta.Backend: built configuration for "etcdv3" backend with hash value 1761853381
2020/08/04 13:37:19 [TRACE] Preserving existing state lineage "eddefc0a-07b6-297f-16a2-190281c12128"
2020/08/04 13:37:19 [TRACE] Preserving existing state lineage "eddefc0a-07b6-297f-16a2-190281c12128"
2020/08/04 13:37:19 [TRACE] Meta.Backend: working directory was previously initialized for "etcdv3" backend
2020/08/04 13:37:19 [TRACE] Meta.Backend: using already-initialized, unchanged "etcdv3" backend configuration
2020/08/04 13:37:20 [TRACE] Meta.Backend: instantiated backend of type etcd.Backend
2020/08/04 13:37:20 [DEBUG] checking for provider in "."
2020/08/04 13:37:20 [WARN] found legacy provider "terraform-provider-foreman.log"
2020/08/04 13:37:20 [DEBUG] checking for provider in "/usr/local/bin"
2020/08/04 13:37:20 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2020/08/04 13:37:20 [DEBUG] found valid plugin: "foreman.log", "0.0.0", "/home/gitlab-runner/tmp/tf-allzonesconfig/terraform-provider-foreman.log"
2020/08/04 13:37:20 [DEBUG] checking for provisioner in "."
2020/08/04 13:37:20 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/08/04 13:37:20 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2020/08/04 13:37:20 [TRACE] Meta.Backend: backend *etcd.Backend does not support operations, so wrapping it in a local backend
Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)

Crash Output

Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)

Expected Behavior

Update current etcd state.

Actual Behavior

Throw error:
Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)

Steps to Reproduce

terraform init -get-plugins=false -plugin-dir /data/TF/plugins
terraform workspace select Testzone
terraform plan -input=false -out=plan.tfplan
terraform apply -input=false -auto-approve plan.tfplan

Additional Context

After issue with consul KV max field 512Kb. #24394
I try to switch my backend to etcdv3. And deploy 200 VMs in my test workspace.
VMs deployed successfully, but configuration save failed.
Terraform generate errored.tfstate file and throw error.

My etcd version is:
etcd Version: 3.4.10
Git SHA: 18dfb9cca
Go Version: go1.12.17
Go OS/Arch: linux/amd64

and etcd configuration has option --max-request-bytes 15728640
that increase client request size up to 15Mb

But command "terraform state push errored.tfstate" don't allow to do that either, because of grpc limits:
trying to send message larger than max (2800411 vs. 2097152)

errored.tfstate size is 2,7Mb.

It looks like terraform's grpc client not allow to send more than 2097152 bytes of data.
But may be there another reason.

References

[remilapeyre]

Hi @aneroid13, I hope it's not too late but we are currently working to remove the 512 KB limit in the Consul backend: #25856

[jasons42]

We are running into this issue as well.

I've opened a PR with a potential fix here:
#28078

When attempting to test my changes I ran into a number of issues with the etcdv3 acceptance tests. I was able to work through a couple of them but I don't know that I can easily resolve the final one involving the locking feature of the backend.

Links to the new issues:
#28096
#28098
#28100

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.