Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt-cloud does not recognize terminated instances #27532

Closed
centromere opened this issue Sep 30, 2015 · 8 comments

Comments

Projects
None yet
3 participants
@centromere
Copy link

commented Sep 30, 2015

When trying to start a group of servers like so:
salt-cloud -m /srv/config/cloud.maps.d/all.map -P
all servers fail to launch with the following output:

app01:
    ----------
    Message:
        Already running

These servers appear in the EC2 instance list, but they are in the "Terminated" state, so I see no reason why salt should consider them to be already running.

@rallytime

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2015

This is a funny thing that happens from EC2. They don't run their clean-up functions anywhere from 30-60 mins after you've "deleted" a machine via salt-cloud. To get around this, you can set rename_on_destroy: True in your cloud provider config and you shouldn't have problems anymore. You can read about that in our docs, here, for more information.

Also, starting with 2015.8.0, this rename_on_destroy parameter should be set to True by default, so you don't run into this problem. Therefore, if you're using that version, there might be a bug that we should look into. @centromere Can you post the output of salt-cloud --versions-report so we can confirm you're not bumping into a bug?

@rallytime rallytime added this to the Blocked milestone Sep 30, 2015

@centromere

This comment has been minimized.

Copy link
Author

commented Sep 30, 2015

Salt Version:
            Salt: 2015.8.0

Dependency Versions:
 Apache Libcloud: 0.15.1
          Jinja2: 2.7.2
        M2Crypto: Not Installed
            Mako: 0.9.1
          PyYAML: 3.10
           PyZMQ: 14.4.0
          Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
            RAET: Not Installed
         Tornado: 4.2.1
             ZMQ: 4.0.4
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 1.5
           gitdb: 0.5.4
       gitpython: Not Installed
           ioflo: Not Installed
         libnacl: Not Installed
    msgpack-pure: Not Installed
  msgpack-python: 0.3.0
    mysql-python: Not Installed
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
    python-gnupg: Not Installed
           smmap: 0.8.2
         timelib: Not Installed

System Versions:
            dist: Ubuntu 14.04 trusty
         machine: x86_64
         release: 3.13.0-65-generic
          system: Ubuntu 14.04 trusty
@rallytime

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2015

Ok, so the fix I was mentioning above is in #26479. How did you delete/terminate the instances in question that are marked as Already Running? Did you do it from the AWS Console? Or from Salt-Cloud?

Salt-Cloud can't create instances of the same name, so the messaging here can be a little misleading.

@centromere

This comment has been minimized.

Copy link
Author

commented Sep 30, 2015

I deleted them from the AWS console. I purposely do not allow the IAM role for the salt-master to delete instances for security reasons. I've manually renamed the instances upon termination and everything behaves as I would expect.

@rallytime

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2015

Ah, perfect. Then that makes sense. I'll see if we can at least clean up the messaging here so that it doesn't say "Already Running" when it is in fact Terminated. Thanks!

@rallytime

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2015

@centromere I've fixed this issue in #27575 by amending the message if the instance is on EC2:

root@rallytime:~# salt-cloud -m /etc/salt/map
[INFO    ] salt-cloud starting
[INFO    ] Applying map from '/etc/salt/map'.
[WARNING ] 'CLOUD-TEST-KRMJVL' already exists, removing from the create map.
CLOUD-TEST-KRMJVL:
    ----------
    Message:
        Instance already exists, or is terminated and has the same name.

This should help reduce confusion in the future.

@centromere

This comment has been minimized.

Copy link
Author

commented Oct 1, 2015

Thank you!

@rallytime rallytime closed this Oct 2, 2015

@DanyC97

This comment has been minimized.

Copy link

commented Oct 8, 2015

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.