New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deserialization doesn't work when invoked from Azure Function #2407

Closed
tomaszkiewicz opened this Issue Mar 9, 2017 · 16 comments

Comments

Projects
@tomaszkiewicz
Copy link

tomaszkiewicz commented Mar 9, 2017

Hi,

I setup a bot on Azure Bot Service, downloaded the template, made it work with VS2015 and I've moved all code to separate class library to run it as precompiled function.

You can find whole code here: https://github.com/tomaszkiewicz/AzureFunctionsPersistenceProblem/tree/test

So, my function.json file looks like this:

{
  "scriptFile": "bin\\Bot.dll",
  "entryPoint": "Bot.Runner.Run",
  "bindings": [
    {
      "type": "httpTrigger",
      "direction": "in",
      "webHookType": "genericJson",
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "disabled": false
}

The Bot.Runner.Run function is the same as it was from the template and I do the following on new message:

await Conversation.SendAsync(activity, () => new BasicProactiveEchoDialog());

So it's typical calling of dialogs stack. At this step I am able to chat with the bot (tested both on emulator and on Azure Bot Service) so everything is loaded correctly etc.

The problem is that this dialog is not deserialized properly when it's in the separate class library.
It worked when it was a .csx file in Azure Function, it also works when I move it to BotApplication template project (the one based on ASP.NET MVC), but as soon as it gets into separate class library everything breaks.

I've tried to gather more details, so I've added both parameterless constructor and a method with OnDeserialize attribute, put a breakpoint in each of them and the behaviour is that OnDeserialize method gets invoked, when I lookup in the count variable in this dialog it is correctly deserialized but... after I resume the execution the parameterless constructor gets called.

I've found in documentation or stackoverflow thread (I don't remember excactly where) that the behaviour of bot framework is that if it is not possible to sucessfully deserialize the dialog stack it gets resetted.
So it's probably that case here, but I cannot find a reason of unsucessful deserialization...

Could you take a look at it? Or maybe suggest what to check to provide more details?

Best regards

Łukasz Tomaszkiewicz

@tomaszkiewicz

This comment has been minimized.

Copy link

tomaszkiewicz commented Mar 10, 2017

Hi,

I spent few more hours debugging the problem and here's what I found:

As I supposed the problem relates to error in deserialization. It happens in Store.cs file in the following method:

        bool IStore<T>.TryLoad(out T item)
        {
            try
            {
                return this.store.TryLoad(out item);
            }
            catch (Exception)
            {
                // exception in loading the serialized data
                item = default(T);
                return false;
            }
        }

When you breakpoint in catch part you can find out that the exception is:

Message	"Unable to find assembly 'Microsoft.Bot.Builder, Version=3.5.5.0, Culture=neutral, PublicKeyToken=null'."	
   at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
   at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryHeaderEnum binaryHeaderEnum)
   at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)
   at Microsoft.Bot.Builder.Internals.Fibers.FormatterStore`1.Microsoft.Bot.Builder.Internals.Fibers.IStore<T>.TryLoad(T& item) in D:\Projekty\BotBuilder\CSharp\Library\Microsoft.Bot.Builder\Fibers\Store.cs:line 65
   at Microsoft.Bot.Builder.Internals.Fibers.ErrorResilientStore`1.Microsoft.Bot.Builder.Internals.Fibers.IStore<T>.TryLoad(T& item) in D:\Projekty\BotBuilder\CSharp\Library\Microsoft.Bot.Builder\Fibers\Store.cs:line 108

So BotBuilder cannot deserialize dialog stack because it cannot find... it's own assembly!

I've tried to apply the typical solution for this kind of error by implementing own binder that tries every possible and loaded assembly:

  sealed class SearchAssembliesBinder : SerializationBinder
    {
        private readonly Assembly _currentAssembly;

        public SearchAssembliesBinder(Assembly currentAssembly)
        {
            _currentAssembly = currentAssembly;
        }

        public override Type BindToType(string assemblyName, string typeName)
        {
            var assemblyNames = new List<AssemblyName>();

            assemblyNames.Add(_currentAssembly.GetName());
            assemblyNames.Add(Assembly.GetCallingAssembly().GetName());

            assemblyNames.AddRange(_currentAssembly.GetReferencedAssemblies());
            assemblyNames.AddRange(AppDomain.CurrentDomain.GetAssemblies().Select(s => s.GetName()));

            foreach (AssemblyName an in assemblyNames)
            {
                var typeToDeserialize = Type.GetType($"{typeName}, {an.FullName}");

                if (typeToDeserialize != null)
                    return typeToDeserialize;
            }

            return null;
        }
    }

Unfortunately that doesn't help, but the exception changes to:

Message	"Could not load file or assembly 'Microsoft.Bot.Builder, Version=3.5.5.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified."
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)
   at System.UnitySerializationHolder.GetRealObject(StreamingContext context)
   at System.Runtime.Serialization.ObjectManager.ResolveObjectReference(ObjectHolder holder)
   at System.Runtime.Serialization.ObjectManager.DoFixups()
   at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)
   at Microsoft.Bot.Builder.Internals.Fibers.FormatterStore`1.Microsoft.Bot.Builder.Internals.Fibers.IStore<T>.TryLoad(T& item) in D:\Projekty\BotBuilder\CSharp\Library\Microsoft.Bot.Builder\Fibers\Store.cs:line 101
   at Microsoft.Bot.Builder.Internals.Fibers.ErrorResilientStore`1.Microsoft.Bot.Builder.Internals.Fibers.IStore<T>.TryLoad(T& item) in D:\Projekty\BotBuilder\CSharp\Library\Microsoft.Bot.Builder\Fibers\Store.cs:line 144

There's also FusionLog property of the exception:

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.Bot.Builder, Version=3.5.5.0, Culture=neutral, PublicKeyToken=null
 (Fully-specified)
LOG: Appbase = file:///C:/Users/luktom/AppData/Local/Azure.Functions.Cli/1.0.0-beta.93/
LOG: Initial PrivatePath = NULL
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Users\luktom\AppData\Local\Azure.Functions.Cli\1.0.0-beta.93\func.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Users/luktom/AppData/Local/Azure.Functions.Cli/1.0.0-beta.93/Microsoft.Bot.Builder.DLL.
LOG: Attempting download of new URL file:///C:/Users/luktom/AppData/Local/Azure.Functions.Cli/1.0.0-beta.93/Microsoft.Bot.Builder/Microsoft.Bot.Builder.DLL.
LOG: Attempting download of new URL file:///C:/Users/luktom/AppData/Local/Azure.Functions.Cli/1.0.0-beta.93/Microsoft.Bot.Builder.EXE.
LOG: Attempting download of new URL file:///C:/Users/luktom/AppData/Local/Azure.Functions.Cli/1.0.0-beta.93/Microsoft.Bot.Builder/Microsoft.Bot.Builder.EXE.

So I've tired to put Bot Builder function on the paths specified above. That also doesn't help. This time the exception is not caught, but the following is printed on console:

A ScriptHost error has occurred
Exception while executing function: Functions.messages. Bot: The type initializer for 'Microsoft.Bot.Builder.Dialogs.Conversation' threw an exception. Microsoft.Bot.Builder.Autofac: Could not load file or assembly 'Microsoft.Bot.Connector, Version=3.5.3.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
Exception while executing function: Functions.messages. Bot: The type initializer for 'Microsoft.Bot.Builder.Dialogs.Conversation' threw an exception. Microsoft.Bot.Builder.Autofac: Could not load file or assembly 'Microsoft.Bot.Connector, Version=3.5.3.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

Exception while executing function: Functions.messages
Exception while executing function: Functions.messages. Bot: The type initializer for 'Microsoft.Bot.Builder.Dialogs.Conversation' threw an exception. Microsoft.Bot.Builder.Autofac: Could not load file or assembly 'Microsoft.Bot.Connector, Version=3.5.3.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
Executed 'Functions.messages' (Failed, Id=8e221200-0ad4-48a5-91a9-a876780965a4)
mscorlib: Exception while executing function: Functions.messages. Bot: The type initializer for 'Microsoft.Bot.Builder.Dialogs.Conversation' threw an exception. Microsoft.Bot.Builder.Autofac: Could not load file or assembly 'Microsoft.Bot.Connector, Version=3.5.3.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
  Function had errors. See Azure WebJobs SDK dashboard for details. Instance ID is '8e221200-0ad4-48a5-91a9-a876780965a4'
mscorlib: Exception while executing function: Functions.messages. Bot: The type initializer for 'Microsoft.Bot.Builder.Dialogs.Conversation' threw an exception. Microsoft.Bot.Builder.Autofac: Could not load file or assembly 'Microsoft.Bot.Connector, Version=3.5.3.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

I hope it helps, I have no idea what I can do next in this case.

@willportnoy

This comment has been minimized.

Copy link
Member

willportnoy commented Mar 14, 2017

I see Connector version 3.5.3.0 and Builder version 3.5.5.0 in your logs - any chance of a assembly version mismatch? From what I understand, Azure Bot Service doesn't support bindingRedirects.

@tomaszkiewicz

This comment has been minimized.

Copy link

tomaszkiewicz commented Mar 14, 2017

The 2nd post was based on debug branch in which I updated to newest version. I debugged it also with the newest source code of BotFramework from GitHub.
I don't know if ABS supports bindingRedirects, but in other Azure Functions I never had any problem with any library, so I suppose it could be supported.

@vsadams

This comment has been minimized.

Copy link

vsadams commented Mar 28, 2017

Did you ever figure out a work around for this? I am having a similar issue and have run out of ideas.

@tomaszkiewicz

This comment has been minimized.

Copy link

tomaszkiewicz commented Mar 29, 2017

Unfortunately not, after 2-3 days of dubugging I opened this issue and switched my project to classic asp.net mvc web app.

@BenjaBobs

This comment has been minimized.

Copy link

BenjaBobs commented Apr 18, 2017

When using Azure Bot Service with precompiled .dlls I didn't manage to get nuget dependencies working, but I did find that including all the dependencies (~18 .dll files) in the /bin folder solved my assembly binding exceptions. This is hardly an ideal solution though.

@NicolasHumann

This comment has been minimized.

Copy link

NicolasHumann commented Jun 25, 2017

Hi, any news about this issue ?
I have a workaround, but I think its really bad...

  builder
             .Register((c, p) => new FactoryStore<IFiberLoop<DialogTask>>(new MemoryErrorResilientStore<IFiberLoop<DialogTask>>(new FormatterStore<IFiberLoop<DialogTask>>(p.TypedAs<Stream>(), c.Resolve<IFormatter>(p))), c.Resolve<Func<IFiberLoop<DialogTask>>>(p)))
             .As<IStore<IFiberLoop<DialogTask>>>()
             .InstancePerDependency();

and

public class MemoryErrorResilientStore<T> : IStore<T>
{
    static System.Collections.Concurrent.ConcurrentDictionary<string, object> bag = new System.Collections.Concurrent.ConcurrentDictionary<string, object>();

    private readonly IStore<T> store;
    public MemoryErrorResilientStore(IStore<T> store)
    {
        SetField.NotNull(out this.store, nameof(store), store);
    }

    void IStore<T>.Reset()
    {
        this.store.Reset();
    }

    bool IStore<T>.TryLoad(out T item)
    {
        try
        {
            object obj = null;
            bag.TryGetValue(typeof(T).FullName, out obj);
            if (obj == null)
            {
                item = default(T);
                return false;
            }
            item = (T)obj;
            return true;
        }
        catch (Exception ex)
            {
            // exception in loading the serialized data
            item = default(T);
            return false;
        }
    }

    void IStore<T>.Save(T item)
    {
        bag.TryAdd(typeof(T).FullName, item);
    }

    void IStore<T>.Flush()
    {
        this.store.Flush();
    }
}
@AdamMarczak

This comment has been minimized.

Copy link

AdamMarczak commented Aug 24, 2017

Any updates on this? With new azure functions tooling it seems like compilable functions are way to go and this way we can't take advantage of them,

@NicolasHumann

This comment has been minimized.

Copy link

NicolasHumann commented Aug 25, 2017

Hi, @AdamMarczak
To use the new VS 17 15.3 tooling for Azure function, you have to use https://github.com/Microsoft/BotBuilder-Azure but I found the bug and submit a pull request Microsoft/BotBuilder-Azure#15 to correct the serialization issue

@AdamMarczak

This comment has been minimized.

Copy link

AdamMarczak commented Aug 25, 2017

Great news @NicolasHumann! I did use bot builder for azure but I didn't see there's option for storing state. This means deserialization will no longer be an issue, cool!

@berhir

This comment has been minimized.

Copy link

berhir commented Aug 26, 2017

Thank you @NicolasHumann for your pull request, it pointed me in the right direction. I extended it a little bit to work also for assemblies that are not loaded to the AppDomain yet.

Technically you are not required to use BotBuilder-Azure but it has some features that you would have to implement by yourself. Like the authentication stuff. And I hope it will fix the assembly loading problem in the future.

Until it gets fixed I created my own assembly resolve handler:

public class AzureFunctionsResolveAssembly : IDisposable
{
    public AzureFunctionsResolveAssembly()
    {
        AppDomain.CurrentDomain.AssemblyResolve += CurrentDomain_AssemblyResolve;
    }

    void IDisposable.Dispose()
    {
        AppDomain.CurrentDomain.AssemblyResolve -= CurrentDomain_AssemblyResolve;
    }

    private Assembly CurrentDomain_AssemblyResolve(object sender, ResolveEventArgs arguments)
    {
        var assembly = AppDomain.CurrentDomain.GetAssemblies()
            .FirstOrDefault(a => a.GetName().FullName == arguments.Name);

        if (assembly != null)
        {
            return assembly;
        }

        // try to load assembly from file
        var assemblyDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
        var assemblyName = new AssemblyName(arguments.Name);
        var assemblyFileName = assemblyName.Name + ".dll";
        string assemblyPath;

        if (assemblyName.Name.EndsWith(".resources"))
        {
            var resourceDirectory = Path.Combine(assemblyDirectory, assemblyName.CultureName);
            assemblyPath = Path.Combine(resourceDirectory, assemblyFileName);
        }
        else
        {
            assemblyPath = Path.Combine(assemblyDirectory, assemblyFileName);
        }

        if (File.Exists(assemblyPath))
        {
            return Assembly.LoadFrom(assemblyPath);
        }

        return null;
    }
}

And this is how I use it in my Azure Functions Bot:

[FunctionName("Messages")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "messages")]HttpRequestMessage req, TraceWriter log)
{
    // use custom assembly resolve handler
    using(new AzureFunctionsResolveAssembly())
    using (BotService.Initialize())
    {
        // Deserialize the incoming activity
        string jsonContent = await req.Content.ReadAsStringAsync();
        var activity = JsonConvert.DeserializeObject<Activity>(jsonContent);

        // authenticate incoming request and add activity.ServiceUrl to MicrosoftAppCredentials.TrustedHostNames
        // if request is authenticated
        if (!await BotService.Authenticator.TryAuthenticateAsync(req, new[] { activity }, CancellationToken.None))
        {
            return BotAuthenticator.GenerateUnauthorizedResponse(req);
        }

        if (activity != null)
        {
            switch (activity.GetActivityType())
            {
                case ActivityTypes.Message:
                    await Conversation.SendAsync(activity, () => new RootDialog());
                    break;
                [...]
            }
        }
        return req.CreateResponse(HttpStatusCode.Accepted);
    }
}
@jorupp

This comment has been minimized.

Copy link

jorupp commented Aug 28, 2017

Glad I stumbled across this - I too saw the "Azure Bot Service" thing in the portal was using the csx-style functions and thought I'd be clever by following the same basic pattern but with pre-compiled functions. Basic call-and-response got working quickly, but dialog state just seemed to be disappearing. I even debugged it so far as to confirm that the right state was flowing back and forth to the state server (ie. https://state.botframework.com/v3/botstate/webchat/conversations/XXXX/users/YYYY) - never occurred to me that the deserialization was where the problem was.

Thanks to @NicolasHumann and @berhir though, I have this working now (at least locally in the emulator). Looking forward to continuing to see how this develops.

In addition to fixing it (obviously), is there a good reason exceptions on deserialization should be silently swallowed? Shouldn't we get some way to hook into them? Are there other similar places where things might fail silently? Maybe a custom interface implementation we can provide to the AutoFac config? Or at least have something writing to the console about the possible issue.

@nwhitmont

This comment has been minimized.

Copy link
Contributor

nwhitmont commented Oct 10, 2017

Open a new issue if further questions

@nwhitmont nwhitmont closed this Oct 10, 2017

@DaveWare

This comment has been minimized.

Copy link

DaveWare commented Feb 26, 2018

I was having the same problem using binary serialization through a storage queue. When attempting to deserialize the message with the BinaryFormatter the assembly containing the class would get the "could not find assembly exception". Even though an instance of the class had already been created previously. The fusion loader recorded no bind failures so I was stuck until I found this post. I used the assembly resolver code posted by berhir and now the serialization is working. Thanks!

@rockfordlhotka

This comment has been minimized.

Copy link

rockfordlhotka commented Apr 1, 2018

This sounds like a very old issue with the way some EXEs host the .NET runtime. Way back in time it was often COM+ or IE and then Cassini that had this issue. Long ago (2002 or so?) I wrote a workaround for the issue that you can hook into your AppDomain only once to provide a custom resolver for the assembly resolution failure. It is in this old chunk of code:

https://github.com/MarimerLLC/csla/blob/V1-5-x/cslacs10/NetRun/Launcher.cs#L49

@DaveWare

This comment has been minimized.

Copy link

DaveWare commented Apr 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment