"Unexpected end of JSON input" error locking s3 backend

[petur]

Terraform Version

Terraform v1.3.8
on linux_amd64

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "<redacted>"
    key = "bug/unexpected-end-of-json-input/terraform.tfstate"
    encrypt = true
    kms_key_id = "<redacted>"
    dynamodb_table = "<redacted>"
  }
}

Debug Output

https://gist.github.com/petur/b66b6c7fce877ddb1bf3cc7629101de0

Expected Behavior

When terraform is run with -lock-timeout, it should either acquire the lock, or retry until the timeout expires.

Actual Behavior

Terraform fails with the error:

╷
│ Error: Error acquiring the state lock
│ 
│ Error message: 2 errors occurred:
│       * ConditionalCheckFailedException: The conditional request failed
│       * unexpected end of JSON input
│ 
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Steps to Reproduce

1. terraform init
2. Duplicate the folder containing the configuration
3. Run while TF_LOG=trace terraform plan -lock-timeout=60s; do : ; done in both folders (needs two shells).

(This is an artificial way to reproduce the issue. It was found in CI where multiple people push changes and plans get run in parallel.)

Additional Context

No response

References

No response

[petur]

This looks like a simple race condition in the locking code:

In s3/client.go:

	_, err := c.dynClient.PutItem(putParams)

	if err != nil {
		lockInfo, infoErr := c.getLockInfo()
		if infoErr != nil {
			err = multierror.Append(err, infoErr)
		}

		lockErr := &statemgr.LockError{
			Err:  err,
			Info: lockInfo,
		}
		return "", lockErr
	}

PutItem fails with a ConditionalCheckFailedException when another user has the state locked. The other user then unlocks the state before c.getLockInfo manages to fetch the lock.

In locker.go:

		if le == nil || le.Info == nil || le.Info.ID == "" {
			// If we don't have a complete LockError then there's something
			// wrong with the lock.
			return "", err
		}

Here le.Info.ID is presumably empty because the lock ID was gone before getLockInfo could read it from DynamoDB, and the locker exits the wait loop.

[skp33]

I am also facing the same. Is there any update on this issue?

[zhiweio]

same problem

[Amarsali01]

╷
│ Error: Error acquiring the state lock
│
│ Error message: 2 errors occurred:
│ * ConditionalCheckFailedException: The conditional request failed
│ * unexpected end of JSON input
│
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

[samaiyayashraj]

Getting exactly the same issue. Any update ?