Reading plan file from other than file

[stazz]

Terraform Version

Terraform v0.13.0-rc1

But also seems to happen with 0.12.xyz versions.

Terraform Configuration Files

resource "null_resource" "resource" {
}

After this, run: terraform init, then terraform plan -out=plan.bin, and finally cat plan.bin | terraform apply /dev/stdin.

Debug Output

N/A.

Crash Output

N/A.

Expected Behavior

I would expect for the cat plan.bin | terraform apply /dev/stdin command to behave like terraform apply plan.bin does: it reads the file produced by plan phase, and applies the changes therein.

Actual Behavior

The following error message was printed: zip: not a valid zip file.
If I try command cat plan.bin | terraform apply - (in a spirit of many other *nix commands accepting - as alias for stdin), I will get: stat -: no such file or directory.

Steps to Reproduce

1. terraform init
2. terraform plan -out=plan.bin
3. cat plan.bin | terraform apply /dev/stdin

Additional Context

I understand that zip files are structured in such way that reading them from non-seekable stream is not suitable for many libraries, and I guess Go's zip library is one of them.
Could Terraform maybe detect when the user tries to specify non-seekable stream (stdin/pipe/etc) as apply file, read it to memory in full, and then pass it to the zip library?

The exact usecase I have for this is when I am storing encrypted plan file as pipeline artifact in our DevOps pipeline (since storing it unencrypted would expose all the secret stuff in plaintext).
During the apply phase, I tried to make a fifo and pass it through openssl decrypting it, before the contents finally reach terraform apply command, so the unencrypted file would never be stored on the disk wherever the pipeline agent is running.
However, this bug prevents such workflow.

FWIW, the approach works in the plan phase:

# Create fifo, instruct openssl to read password from environment variable, and fork it to proceed further to actual TF command
mkfifo plan.bin && openssl enc -e -aes-256-ctr -iter 100 -in plan.bin -out plan.bin.encrypted -pass env:TF_PLAN_ENCRYPTION_PW &
set +e # Let's not abort on non-zero exit codes
terraform plan -out=plan.bin -detailed-exitcode # Pass fifo to TF plan command. The unencrypted plan will not be written to disk, insted it will go directly to openssl, which will write the encrypted output to disk.
TF_EXIT_CODE=$? # Save TF exit code.
rm plan.bin # Remove fifo, but keep the encrypted file
set -e # Abort on non-zero exit codes again.
exit ${TF_EXIT_CODE} # The plan.bin.encrypted file is left to be uploaded as pipeline artifact

References

I searched issues with the error message I got but anything even close to this was #22396 , which is just about unhelpful error messages when passing badly formatted plan file or command line parameters.

Workaround

Currently, during apply phase, I do the following:

openssl enc -d -aes-256-ctr -iter 100 -in plan.bin.encrypted -out plan.bin -pass 'env:TF_PLAN_ENCRYPTION_PW' # Decrypt file and store it to disk 
set +e # Let's not abort on non-zero exit codes
terraform apply plan.bin # Pass the plan file to TF apply command and wait for it to complete
TF_EXIT_CODE=$? # Save TF exit code.
rm -rf plan.bin # Remember to remove unencrypted plan file
set -e # Abort on non-zero exit codes again.
exit ${TF_EXIT_CODE}

This of course has the problem of storing the unencrypted plan file on disk during the apply.

[danieldreier]

@stazz I'm going to recategorize this as an enhancement request, because I don't think the Terraform docs describe that reading a plan from stdin should work. This seems like a reasonable enough request, but since I don't think we ever said this would work, I think that it's more accurate to categorize it as an enhancement request than as a bug.

I didn't quite follow - did you say that the FIFO approach worked?

[stazz]

Hi @danieldreier , you're right that TF docs do not mention stdin. I do a lot of Linux command-line usage, and most of the tools there support stdin as alternative for file, so I admit I did an assumption there. :)

The FIFO approach works for writing the plan file (during terraform plan). But reading the plan file (during terraform apply), however, fails from both FIFO and stdin, because neither is seekable stream. So there is a small inconsistency there, especially for a person who is unaware that plan files are zip files, and TF + Go combination in its current state requires seekable stream ( = file ) to be passed to apply in order to successfully read the zip file.