@ec2instance decorator executed too early?

[curiousleo]

Hi there, I'm trying to solve the following problem with awsfabrictasks: I would like to first start a certain number of instances, and then execute tasks on them.

I wrote a small task setup that creates a list of Ec2LaunchInstances, starts them using Ec2LaunchInstance.run_many_instances, and calls Ec2InstanceWrapper(instance.instance).add_instance_to_env() on each of them.

The idea was to then call awsfabric setup task1 task2, where task1 and task2 would then be executed on the instances started from setup. Unfortunately add_instance_to_env() does not seem to actually add the instance to env, and so this approach doesn't work.

Now I was wondering

1. Is there a nicer way of doing this? Starting a number of instances and then executing tasks on them seems like a fairly standard thing to do, and a task that manipulates the env feels a little hacky.
2. What's going wrong in the code explained above -- why does add_instance_to_env() not result in that instance being added to env?

[espenak]

As far as I can tell, you assume env is persistent between tasks? This is not the case. You need to save state somewhere else.

add_instance_to_env is used by awsfabrictasks.main.get_hosts_supporting_aws to register autogenerated Ec2InstanceWrapper instances based on command line options (-E, --ec2tags or --ec2ids).

One solution is to use Ec2 tags to save state, and use --ec2tags to perform actions on all instances with the same tag. We also provide a few utilities and helpers for tags.

To sum this up, you will most likely want to do something like this:

@tasks
def setup():
    a = Ec2LaunchInstance(extra_tags={'Name': 'nginxserver', 'Environment': 'my_super_webapp'})
    b = Ec2LaunchInstance(extra_tags={'Name': 'djangoserver', 'Environment': 'my_super_webapp'})
    Ec2LaunchInstance.confirm_many([a, b])
    Ec2LaunchInstance.run_many_instances([a, b])
    # Note: that we can start doing stuff with ``a`` and ``b`` that does not
    # require the instances to be running, such as setting tags.
    Ec2LaunchInstance.wait_for_running_state_many([a, b])

@task
def dosomething():
    """
    Typically called with ``awsfab --ec2tags my_super_webapp dosomething``.
    """
    run('uname -a')

@task
@ec2instance(tags={'Environment': 'my_super_webapp'})
def dosomething_for_my_super_webapp():
    """
    Called with ``awsfab dosomething_for_my_super_webapp``.

    """
    run('whoami')

You should also take a look at #11.

[curiousleo]

Ah okay, thank you very much for the explanation! So this is not a bug, I just had a different mental model of what was going on ... Doing this via tags seems pretty neat.

Just to make sure I got this right: Using the example fabfile you provided (thanks!), calling awsfab setup dosomething_for_my_super_webapp should launch two instances named nginxserver and djangoserver, and then execute whoami on both?

I copied the code from your comment into a fabfile.py, added the required imports, fixed a typo (@tasks -> @task in the first line) and ran awsfab setup dosomething_for_my_super_webapp. The setup part launches the instances just fine (they show up as "running" on the EC2 Management Console), but then dosomething_for_my_super_webapp prints

No hosts found. Please specify (single) host string for connection: 

-- is that the expected behaviour? My understanding from your explanation was that at this point, the ec2instances decorator should feed the two instances created in setup to dosomething_for_my_super_webapp because they were both tagged with the my_super_webapp environment.

[espenak]

As far as I remember it should work like https://fabric.readthedocs.org/en/1.3.0/usage/parallel.html, but I have not used it for a couple of months and I created the code over a year ago so I may be remembering it wrong. If the tag matching does not work like specifying multiple hosts, we should provide another way of doing what you want to do using tags.

[curiousleo]

Sorry -- could you explain what you mean by "it should work like @parallel"?

[espenak]

Sorry that was not explained clearly! It should not work like @parallel, but it should work like specifying multiple hosts. I.E.: awsfab --ec2tags mytag should work like specifying fab -H a,b as long as a and b is tagged with mytag. I guess this is not the case, and I or someone willing to make a patch will have to look into fixing it, or creating an alternative command line option for matching multiple hosts with a single tag.

[curiousleo]

Cheers! By sneaking in a couple of print statements, I found a possible explanation for this behaviour: It seems like the @ec2instance decorator is called when awsfab starts up, as opposed to when the task is actually executed.

This means that in the particular case of the example code above, in decorator.py, line 34, instancewrappers is assigned to the list of instance wrappers for instances with the tag 'Environment':'my_super_webapp' that exist when the script is called. When the task dosomething_for_my_super_webapp is finally executed, it is run on this list of instances, which does not include the ones launched in the setup task.