Terraform apply failed due to mismatch time

[joonhwa]

Terraform Version

hashicorp/terraform:0.11.10

Terraform Configuration Files

module.ecs_singapore.network.aws_nat_gateway.main: Still creating... (20s elapsed)
module.ecs_singapore.aws_alb.main: Still creating... (20s elapsed)
module.ecs_singapore.network.aws_nat_gateway.main: Still creating... (8h0m7s elapsed) <<<============= Time went wrong here
module.ecs_singapore.aws_alb.main: Still creating... (8h0m8s elapsed) 
Failed to save state: failed to upload state: RequestTimeTooSkewed: The difference between the request time and the current time is too large.
	status code: 403, request id: 0B0026E03C66248C, host id: n4DuSL4lWCUY0uXCLecwNtOyszV/Pr6ucFMZYxr4TjMuPWNHz2UYtuziy5dysFpJncWdwss1XTo=


Error releasing the state lock!

Error message: failed to retrieve lock info: InvalidSignatureException: Signature not yet current: 20181206T223022Z is still later than 20181206T144544Z (20181206T143044Z + 15 min.)
	status code: 400, request id: G5CBO6G31177HOF9IB22HKEQDBVV4KQNSO5AEMVJF66Q9ASUAAJG

Terraform acquires a lock when accessing your state to prevent others
running Terraform to potentially modify the state at the same time. An
error occurred while releasing this lock. This could mean that the lock
did or did not release properly. If the lock didn't release properly,
Terraform may not be able to run future commands since it'll appear as if
the lock is held.

Expected Behavior

Continue to wait for resource to be created then move on to other resource

Actual Behavior

Operation failed with an error message stating mismatch time.

Steps to Reproduce

1. terraform init
2. terraform plan
3. terraform apply

Additional Context

I am running this with terraform's docker image from my CI pipeline using gitlab-ci. I kind of suspect some kind of time drift somewhere? I have tried setting timezone and installing NTP and synching the timezone. I verified that the time and TZ has changed from the default UTC to my local timezone GMT+8. However, the error still occurs.

I am kind of confused if this is a terraform issue or docker issue or both?
Any help will be greatly appreciated.

[apparentlymart]

Hi @joonhwa,

Terraform computes that 8h0m7s elapsed readout by comparing the current system time when the message is printed to the system time that was recorded when the apply operation began. Assuming the operation did not take 8 hours to complete (which seems unlikely for an aws_alb object), that indeed suggests that your system time was somehow reset to eight hours later while Terraform was running.

This seems confirmed by the final error that is complaining that the timestamp used in the signature is "not yet current".

Terraform itself does not make any changes to your host system clock, so something outside of Terraform must be doing it. Since both the AWS SDK and Terraform itself are using UTC timestamps for this comparision, this suggests some confusion on your system about what timezone the system clock is in.

On Unix systems, the system clock is usually in UTC and then offset to local time only for display purposes, while on Windows systems the system clock may be in local time and require an offset to be applied to recover UTC. It seems like something else on your system switched the system clock from UTC to local or vice-versa while Terraform was running.

When Docker is running on Linux it lives inside the same kernel as the "host" and so there should only be one system clock. But when Docker runs on non-Linux platforms it runs in a virtual machine that may virtualize the system clock on the host by applying an offset to it. I don't know why this offset would change during runtime; I can only assume there's some other software running either on the host or in the container that reset the system time.

I can't think of any way this could be a Terraform issue, since both Terraform and the AWS provider are just asking the host OS for the current time and trusting what they are told.

[joonhwa]

Thanks alot for the detailed explanation @apparentlymart

The root cause for me is an incorrect time setting in my ESXi host. That wrong time is being synched to all the VM running in it, which resulted in this issue. Everything work fine once i set it back to the right UTC time.

[apparentlymart]

Thanks for following up, @joonhwa! I'm glad you found the answer. Since this wasn't a Terraform issue, I'm going to close this out.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.