Sporadic issues installing Ubuntu packages after apt-get update on AWS

[yakneens]

I'm trying to install Saltstack packages using apt-get and am getting very inconsistent behaviour. About 60% of my provisioning runs fail complaining about not being able to find the salt packages. Others finish with no problems. This is deploying to AWS US-East with a Ubuntu 14.04 AMI.

My .tf file:

https://gist.github.com/llevar/b9a2bd8823d8bd0f2993

A typical error looks like this:

aws_instance.salt_master (remote-exec): Connecting to remote host via SSH...
aws_instance.salt_master (remote-exec):   Host: 54.205.141.103
aws_instance.salt_master (remote-exec):   User: ubuntu
aws_instance.salt_master (remote-exec):   Password: false
aws_instance.salt_master (remote-exec):   Private key: true
aws_instance.salt_master (remote-exec):   SSH Agent: true
aws_instance.salt_master (remote-exec): Connected!
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): Some packages could not be installed. This may mean that you have
aws_instance.salt_master (remote-exec): requested an impossible situation or if you are using the unstable
aws_instance.salt_master (remote-exec): distribution that some required packages have not yet been created
aws_instance.salt_master (remote-exec): or been moved out of Incoming.
aws_instance.salt_master (remote-exec): The following information may help to resolve the situation:

aws_instance.salt_master (remote-exec): The following packages have unmet dependencies:
aws_instance.salt_master (remote-exec):  salt-common : Depends: python-dateutil but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-jinja2 but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-croniter but it is not installable
aws_instance.salt_master (remote-exec): E: Unable to correct problems, you have held broken packages.
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): Some packages could not be installed. This may mean that you have
aws_instance.salt_master (remote-exec): requested an impossible situation or if you are using the unstable
aws_instance.salt_master (remote-exec): distribution that some required packages have not yet been created
aws_instance.salt_master (remote-exec): or been moved out of Incoming.
aws_instance.salt_master (remote-exec): The following information may help to resolve the situation:

aws_instance.salt_master (remote-exec): The following packages have unmet dependencies:
aws_instance.salt_master (remote-exec):  salt-master : Depends: salt-common (= 2015.5.3+ds-1trusty1) but it is not going to be installed
aws_instance.salt_master (remote-exec):                Depends: python-m2crypto but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-crypto but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-msgpack but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-zmq (>= 13.1.0) but it is not installable
aws_instance.salt_master (remote-exec):                Recommends: python-git but it is not installable
aws_instance.salt_master (remote-exec): E: Unable to correct problems, you have held broken packages.
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): Some packages could not be installed. This may mean that you have
aws_instance.salt_master (remote-exec): requested an impossible situation or if you are using the unstable
aws_instance.salt_master (remote-exec): distribution that some required packages have not yet been created
aws_instance.salt_master (remote-exec): or been moved out of Incoming.
aws_instance.salt_master (remote-exec): The following information may help to resolve the situation:

aws_instance.salt_master (remote-exec): The following packages have unmet dependencies:
aws_instance.salt_master (remote-exec):  salt-minion : Depends: salt-common (= 2015.5.3+ds-1trusty1) but it is not going to be installed
aws_instance.salt_master (remote-exec):                Depends: python-m2crypto but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-crypto but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-msgpack but it is not installable
aws_instance.salt_master (remote-exec):                Depends: python-zmq (>= 13.1.0) but it is not installable
aws_instance.salt_master (remote-exec):                Depends: dctrl-tools but it is not installable
aws_instance.salt_master (remote-exec):                Recommends: debconf-utils but it is not installable
aws_instance.salt_master (remote-exec): E: Unable to correct problems, you have held broken packages.
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): Some packages could not be installed. This may mean that you have
aws_instance.salt_master (remote-exec): requested an impossible situation or if you are using the unstable
aws_instance.salt_master (remote-exec): distribution that some required packages have not yet been created
aws_instance.salt_master (remote-exec): or been moved out of Incoming.
aws_instance.salt_master (remote-exec): The following information may help to resolve the situation:

aws_instance.salt_master (remote-exec): The following packages have unmet dependencies:
aws_instance.salt_master (remote-exec):  salt-syndic : Depends: salt-master (= 2015.5.3+ds-1trusty1) but it is not going to be installed
aws_instance.salt_master (remote-exec): E: Unable to correct problems, you have held broken packages.
aws_instance.salt_master (remote-exec): /tmp/terraform_2019727887.sh: 6: /tmp/terraform_2019727887.sh: cannot create /etc/salt/minion: Directory nonexistent
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): E: Unable to locate package python-pip
aws_instance.salt_master (remote-exec): Reading package lists... 0%
aws_instance.salt_master (remote-exec): Reading package lists... 100%
aws_instance.salt_master (remote-exec): Reading package lists... Done
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 0%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree... 50%
aws_instance.salt_master (remote-exec): Building dependency tree
aws_instance.salt_master (remote-exec): Reading state information... 0%
aws_instance.salt_master (remote-exec): Reading state information... 8%
aws_instance.salt_master (remote-exec): Reading state information... Done
aws_instance.salt_master (remote-exec): Package python-git is not available, but is referred to by another package.
aws_instance.salt_master (remote-exec): This may mean that the package is missing, has been obsoleted, or
aws_instance.salt_master (remote-exec): is only available from another source

aws_instance.salt_master (remote-exec): E: Package 'python-git' has no installation candidate
aws_instance.salt_master (remote-exec): sudo: salt-master: command not found
aws_instance.salt_master (remote-exec): sudo: salt-minion: command not found
Error applying plan:

1 error(s) occurred:

* Script exited with non-zero exit status: 1

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[catsby]

Hey @llevar – are you able to run the same provisioniong scripts outside of Terraform with no issue? This appears to be an error situation outside of Terraform itself, e.g. Terraform is running the scripts, but they themselves are not completing successfully.

Looking at your .tf file, I see ingress rules but no egress rule. Terraform destroys the default Security Group Egress Rule, so you need to add it to the resource... perhaps that will help?

See our docs here:

• https://terraform.io/docs/providers/aws/r/security_group.html

Try adding this to your security group resource:

egress {
  from_port = 0
  to_port = 0
  protocol = "-1"
  cidr_blocks = ["0.0.0.0/0"]
}

[yakneens]

Hi @catsby. Thanks for your response. I tried running the commands manually and they do succeed. I also tried running the shell scripts that Terraform generates and they also succeed. Of course my sample is small. I've done many more terraform apply's than manual script runs.

When it comes to egress rules, my deployment is not into a VPC and egress rules are not supported in EC2 Classic. The default all enabled Egress rule is applied instead, so that shouldn't be an issue.

[clstokes]

@llevar the provisioning issue you're having may be due to some timing issues between cloud-init completing and Terraform running it's provisioners. hashicorp/packer#2639 and hashicorp/packer#41 (comment) provide some more context and workarounds that may be worth trying.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.