Job execution not working in dynamically loaded assemblies

[thnk2wn]

I have a Console App / Windows Service that uses a plugin style model which dynamically loads plugin assemblies (using MEF) containing code to schedule jobs in Hangfire as well as the jobs themselves.

At runtime I get the following error ("hidden" but visible via hangfire dashboard under recurring jobs):

Could not load file or assembly 'Cyrus.ClientMergeEmails, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

I can set breakpoints and see that assembly loaded, and see the job scheduled in Hangfire but the job class never gets created.

I can do the same style of dynamically loaded jobs in Quartz.net without issue. I can also do the same thing in Hangfire if I use static references instead of dynamically loading the assemblies.

I created this repo to demonstrate the issue: https://github.com/thnk2wn/WindowsServicePluginJobDiscovery

There are 3 proof of concepts here:

• WindowsService.Hangfire - demonstrates the issue (note the readme on testing/reproducing)
• WindowsService.Quartz - demonstrates the desired functionality working in Quartz (note the readme on testing)
• WindowsService.HangfireStatic - demonstrates Hangfire working when the assembly is statically referenced / loaded

Should this work in Hangfire? Is there anything I can do to make it work?

[odinserj]

Hi @thnk2wn, I haven't tested this. But if it works in Quartz (however, it uses the same Type.GetType method to load job types), it should work in Hangfire. I've cloned your repo, but unfortunately the following packages aren't available, and I get the compile-time errors:

• CyberCoders.Background
• CyberCoders.Core
• NLog version 4.1.1

Would you like to reduce the code to simplify testing?

[thnk2wn]

@odinserj Sorry about that. I've removed our internal packages from those projects. NLog is public on nuget but the other two are now removed in my sample repo

[gandarez]

I've been getting the same error. I'm using the same concept of @thnk2wn is using.

[gandarez]

Maybe implementing AssemblyResolve?
http://stackoverflow.com/a/4009428/833531

[gandarez]

I've done PR #486 to resolve this issue

[sacerdotu]

@odinserj I have the same problem like @thnk2wn. @gandarez mentioned that the problem is solved with PR #486. Do you have any idea if/when this PR will be merged?

[ofnhkb1]

I have this problem, but I checked the latest code, and did not solve this problem, is there any way to solve

[ghost]

Any progress in this?

[kenlacoste843]

I'm also watching/waiting 👍

[Trazarw]

+1

[Fl4v10]

Anyone has an awnser to this?

[natiki]

This is also something that would be awesome for us. We also use a console application to host the job code but have had to keep releasing the console app each time we have a change :-(. So I too am looking for a solution to this one.

[odinserj]

Try to reproduce the issue using Hangfire 1.7.0-beta1 released yesterday – serialised type information doesn't contain assembly version anymore, may be new format will help. If not, I'll add the TypeResolver extension point, that will be used to find types based on their serialised information.

[odinserj]

Thanks for your patience. Default type resolver in the new 1.7.0 version dynamically loads the required assemblies, so there will be no exceptions, when assembly is on disk, but not loaded into the application. Also, if type resolver was unable to load the assembly using its strong name, it will retry with a partial name, so there's less need in binding redirects.

New version is already released, please see the upgrade guide: https://docs.hangfire.io/en/latest/upgrade-guides/upgrading-to-hangfire-1.7.html.

[mkgn]

This issue still remains in version 1.7.33. I am on .Net 6 and running a hosted service to execute jobs. Jobs are build as plugins and dropped to a plugin directory inside the hosted service with all it's dependancies. To properly load the plugins with it's dependancies I use https://github.com/natemcmaster/DotNetCorePlugins nuget. All dependencies get properly loaded to DI. However when the job gets executed it gives the same error Could not load file or assembly 'MyCompany.Services.Messaging, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

May be Hangfire searches only it's parent folder and not sub folders for the assemblies? Since my plugins are in
\plugins\MyPlugin[files]

[odinserj]

Hangfire doesn't store custom paths for such assemblies, but you can override the type resolver in the following way:

TypeHelper.CurrentTypeResolver += typeName =>
{
    if (typeName.Contains("MyCompany.Services.Messaging"))
    {
        // Load assembly from the specific place and use it to find the type
        return Assembly.GetType(typeName);
    }

    return TypeHelper.DefaultTypeResolver(typeName);
};

[mkgn]

Thanks, so the name I get to my custom resolver is something like below

MyCompany.Messaging.IMessageSender`1[[MyCompany.Messaging.Email.EMailMessage, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]], MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

Just to be sure that my AppDomain has the required types registered; I scanned AppDomain.CurrentDomain.GetAssemblies(), and printed AssemblyQualifiedName property. it shows below types are loaded

MyCompany.Messaging.IEMailMessage`1, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.IMessage`1, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.IMessageSender`1, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.ISMSMessage`1, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.MessagingServiceCollectionExtentions, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.Register, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.SMS.SMSMessage, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.SMS.SMSMessageSender, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.SMS.SMSSettings, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.Email.EMailMessage, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.Email.EMailMessageSender, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.Email.EMailSettings, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.SMS.SMSMessageSender+<SendAsync>d__8, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
MyCompany.Messaging.Email.EMailMessageSender+<SendAsync>d__5, MyCompany.Messaging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null  

I also used;

    options.UseIgnoredAssemblyVersionTypeResolver();
    options.UseSimpleAssemblyNameTypeSerializer();

to make it a bit simpler hoping this will also make it a bit easy to get the job done. This simplifed the type name without version, culture etc... as below.

MyCompany.Messaging.IMessageSender`1[[MyCompany.Messaging.Email.EMailMessage, MyCompany.Messaging]], MyCompany.Messaging

Then, the quickly hacked type resolver code is below.

options.UseTypeResolver((name) =>
{
      try
      {
          return TypeHelper.DefaultTypeResolver(name); //try using default resolver
      }
            catch (Exception) //if couldn't find
      {
             //get it from AppDomain loaded assemblies.
             var reqType = AppDomain.CurrentDomain.GetAssemblies().SelectMany(x => x.GetTypes().Where(x=> name.StartsWith(x.FullName))).FirstOrDefault();

             if (reqType != null)
                   return reqType;

             throw;
       }
   });

This resolver gets hit twice, one for MyCompany.Messaging.IMessageSender`1 and MyCompany.Messaging.Email.EMailMessage

Since both of the above types are now in AppDomain, I return them and this throws two exceptions which I can't figure out the cause.

Exception thrown: 'System.ArgumentOutOfRangeException' in System.Private.CoreLib.dll
Exception thrown: 'Hangfire.Common.JobLoadException' in Hangfire.Core.dll

All of this works fine if I statically link my plugins to hangfire server project and we will have lot of different jobs and linking all of them to hangfire server project is not the right way to go I think.

[mkgn]

Ok, so got this to work. I will try to brief the issue so that someone else will find this helpful if they are also trying to use plugins with hangfire.

Key points to note is that my jobs are in separate dlls and to load them properly with dependencies I use https://github.com/natemcmaster/DotNetCorePlugins

The way it works, my custom resolver will not be able to locate the types properly. See Reflection. Therefore, my custom type resolver now looks like below;

options.UseTypeResolver((name) => 
{
var foundType = PluginLoaderExtention.FindTypeInPluginContexts(name) ?? TypeHelper.DefaultTypeResolver(name);

if (foundType == null)
{
	//TODO:Log
	throw new JobTypeNotFoundException($"Type: {name} not found");
}

return foundType;
});

Above will call a method in plugin loader like below. I cache the types in a dictionary to make it a bit efficient in subsequent calls.

    public static Type? FindTypeInPluginContexts(string name)
    {
        if (jobTypeCache.ContainsKey(name))
            return jobTypeCache[name];

        foreach (var loader in loaders)
        {
            using (loader.EnterContextualReflection())
            {
                var jobType = Type.GetType(name);
                if (jobType != null)
                {
                    jobTypeCache.TryAdd(name, jobType);
                    return jobType;
                }
            }
        }
        return null;
    } 

Check the plugin library I use for more details.