Timer triggers not firing or taking a long time to start

[AnthonyDewhirst]

I have been trying to use the Timer Triggers for 3 webbjobs that live on the same instance. When running locally, they fired the first time no problem. Then I deployed to my text env and everything was just fine. Then they started to not work at all locally and then when deployed, 2 webjobs started to fire after 20 minutes and the 3rd never fired. I have them set at "00:00:30" which I realise doesn't create a monitor by default. I am using the same dashboard credentials for local and for my test environments.
I decided to download the code from github and I have seen the same results when running locally. I commented all the code out of the sample program except the call to config.UseTimers and all the sample example except the 30s timer example. Again, I am using the same dashboard credentials as for my real project and the results are also intermittent. I have gone back to using a loop and thread sleep which is really not ideal so any help would be appreciated.

[mathewc]

This is likely due to the lock behavior that TimerTrigger employs to ensure that only a single instance of your function is running across scaled out instances. Please see this wiki page for more details and recommendations on how to facilitate local debugging/development.

[AnthonyDewhirst]

Hi Mathew,

Thanks for the pointer.
I can understand now why I am seeing issues. I had somehow used the same storage account in each of my environments, this will be fixed immediately.
Just to be sure, the page is saying that I don't need a storage account per web job, just per environment running the same web job, is this correct?
Out of curiosity, is this trigger name based on the name and namespace of the function?
How do you propose to deal with local devs running the webjobs where we all share the same storage account? (I.e., the string in the web.config is checked in and always pointing to the same place for local dev)

[mathewc]

Yes, if you have multiple developers all attempting to run the same job sharing a storage account, there will be lock contention. The lock ID is based the fully qualified function name. You can see the lock blobs in the azure-webjobs-hosts blob container, under path "locks/". The scenario at that point is the same as if the function is running in production across multiple scaled out instances - there are multiple instances of the job host running, but only one will get the lock and run the function schedule. That's the behavior you want in production.

This issue is related to this issue where we added a new JobHostConfiguration.IsDevelopment property and corresponding JobHostConfiguration.UseDevelopmentSettings() method to configure things optimally for local development.

We are working on enabling a way for multiple developers to all share the same storage account and still be able to run concurrently. I'll keep you posted - we'll get a new release out with the updates in week or two.

[mathewc]

We've made changes that are available in the latest prerelease builds to address the issues you pointed out. You can pull our latest prerelease package to try it out (1.1.1-alpha-10216). We've made changes to ensure that for a particular HostId (which is 1:1 with your WebJobs project / fully qualified assembly name) all Singleton locks are isolated within that scope. Now normally you don't worry about the HostId - the SDK manages that for you and assigns one for you. This means that when doing local development, all devs are running with the same HostId and hence the lock contention.

The proposal is for devs to use their own HostId during local development. They can do this easily by adding a command line argument to their VS project (which will be stored in their csproj.user file). You can then modify your WebJob startup code to use this HostId if specified (but in production use the default ID):

if (config.IsDevelopment)
{
    config.UseDevelopmentSettings();
    if (args.Length > 0)
    {
        config.HostId = args[0];
    }
}

You don't have to use the command line approach - you can set it however you want conditionally. With this approach there will no longer be any locking issues, and you'll also have Dashboard separation of invocations for each HostId. There are some restrictions on HostIds - see JobHost.HostId for details. However you can use descriptive values like "jsmith-dev", etc.