New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.BadImageFormatException: 'Bad IL format.' #18448

Closed
Petermarcu opened this Issue Jun 13, 2018 · 35 comments

Comments

Projects
None yet
@Petermarcu
Member

Petermarcu commented Jun 13, 2018

@atpyk commented on Wed May 30 2018

I've upgraded dotnetcore2.0 to 2.1, and upgraded all packages too.
When I startup my web application, the exception prompts.

StackTrace:
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at GrapeLEAF.AdminService.Program.BuildWebHost(String[] args) in E:\workspace\dotnet\XXX\XXX.AdminService\XXX.AdminService\Program.cs:line 16
at GrapeLEAF.AdminService.Program.Main(String[] args) in E:\workspace\dotnet\XXX\XXX.AdminService\XXX.AdminService\Program.cs:line 12


@atpyk commented on Thu May 31 2018

I uninstall original .net core sdk and runtime; and reinstall VS2017 and 2.1 sdk.
I create a new webapi application by scaffold, it doesn't work.


@VladislavAntonyuk commented on Thu May 31 2018

https://blogs.msdn.microsoft.com/webdev/2018/05/07/asp-net-core-2-1-0-rc1-now-available/


@atpyk commented on Thu May 31 2018

So sad, it doesn't work too.


@VladislavAntonyuk commented on Thu May 31 2018

Make sure you have installed the latest VS and try to reinstall .net core.
Roolback your changes to .net core 2.0 and follow the instruction I have sent you


@atpyk commented on Thu May 31 2018

I debug into Microsoft.AspNetCore.Hosting and find out the error line:
var hostingServices = BuildCommonServices(out var hostingStartupErrors);

It will throw the exception: 'Bad IL format.'

But if I rename this method to BuildCommonServices111 or other, It works.
So weird~
It seems like the namespace "Microsoft.AspNetCore.Hosting" conflicts with others, the .net runtime involves dlls of different version at same time.


@VladislavAntonyuk commented on Fri Jun 01 2018

make sure you have installed version 2.1.300 and try to uninstall all previous versions. Also doublecheck you have finished all steps in the instruction I've sent you


@atpyk commented on Fri Jun 01 2018

I have reinstall my OS, it works.
Thanks.


@mangod3 commented on Sat Jun 02 2018

Did you have NewRelic monitoring your .net app by any chance when you ran into the issue?


@Merurino commented on Sun Jun 03 2018

@mangod3 I was about to comment here. My issue seemed to be Newrelic. Was returning the IL issue. I was building it and running it on docker.


@mangod3 commented on Sun Jun 03 2018

@Merurino Thanks for responding. After debugging the issue it looks like the 2.1 changes related to code versioning (https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/code-versioning-profiler-breaking-changes.md) are causing issues with profiling APIs. @noahfalk who should we follow up with to get this investigated, looks like the ILHeader is not getting initialized correctly after updating a method IL using SetILFunctionBody


@noahfalk commented on Mon Jun 04 2018

Sorry you are running into trouble! Feel free to start with me as the investigator and I can pull in others as needed. To help get us started:

  1. Could you post steps I can follow to try reproducing the problem?
  2. Can you let me know what you observed while debugging that lead you to your conclusions?

@mangod3 commented on Mon Jun 04 2018

Thanks for your response @noahfalk. Will package up a repro and send over email.


@joseclament commented on Wed Jun 06 2018

I have a web job built on netcoreapp2.1 console application, targeting the win10-x64 and getting the "Unhandled Exception: System.BadImageFormatException: Could not load file or assembly exception".
I haven't had any issue with locally. Deployed using the visual studio directly to the app service. Any thoughts?


@noahfalk commented on Wed Jun 06 2018

@joseclament - Are you aware of any .NET profiler or monitoring agent being used in your deployment environment? If not we should probably a open a new issue because there would be a decent chance that the root cause for you will be unrelated to the root cause here.


@joseclament commented on Thu Jun 07 2018

@noahfalk , I am using AppInsight to push the logs, other than no profiling or tooling.
Let me know if I need to open a new issue.


@nil4 commented on Thu Jun 07 2018

@noahfalk I am seeing a similar issue using the AppDynamics .NET Core profiler agent on a netcoreapp2.1 application and .NET Core SDK 2.1.300.

When the application starts, the following exception is thrown:

System.BadImageFormatException
  HResult=0x8007000B
  Message=Bad IL format.
  Source=System.Private.CoreLib
  StackTrace:
   at System.AppDomainManager..ctor()
   at System.Reflection.Assembly.GetEntryAssembly()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at MyApp.Web.Startup.Main(String[] args) 

@joseclament commented on Thu Jun 07 2018

@noahfalk and @nil4 , I got the bottom of the problem. It took couple of days to get the bottom of the issue and made me bit nut. I was able to dig and find this link: https://tutel.me/c/programming/questions/42726350/how+to+change+azure+app+service+to+64bit

Your core should be built on x86 version even if the deploying server(Azure) is based on 64 bit architecture. Because the 64 bit .NET core processes using the .NET core runtime (as opposed to the .NET Framework runtime) are not yet supported on Azure but is planned to be coming in the future. You can verify this by go to the kudu and type dotnet --info. You can see something like this below:

.NET Core SDK (reflecting any global.json):
Version: 2.1.300
Commit: 32f29b6eb9

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
_Base Path: D:\Program Files (x86)\dotnet\sdk\2.1.300_ [You can see only the x86 version here]

Host (useful for support):
Version: 2.1.0
Commit: caa7b7e2ba

.NET Core SDKs installed:
1.1.8 [D:\Program Files (x86)\dotnet\sdk]
2.1.101 [D:\Program Files (x86)\dotnet\sdk]
2.1.300 [D:\Program Files (x86)\dotnet\sdk]

You may be able to debug this locally "Any CPU" option as you might have x86 and x64 based libraries installed on your machine. I could not find any documentation on MSDN about this. I was assumed it would have been automatically support x64 version. Also make sure that the application setting is enabled for the "32" bit.


@nil4 commented on Thu Jun 07 2018

I am not using Azure. The error occurs on my local PC when launching the application, with IIS Express, under the VS debugger, regardless of whether IIS Express 64-bit or 32-bit is selected.

The application starts up correctly if I remove the AppDynamics.Agent.Windows NuGet package, or set CORECLR_ENABLE_PROFILING environment variable to 0 in my Web.config (disabling the profiler).


@mangod3 commented on Thu Jun 07 2018

is it possible for you to capture a full native stack of the exception? That might help with identifying if you are hitting the same issue or something different.


@nil4 commented on Thu Jun 07 2018

@noahfalk Here are the steps to reproduce the issue I see. In a command prompt, run:

mkdir Repro1622 && pushd Repro1622
dotnet new web
dotnet add package AppDynamics.Agent.Windows --version 4.4.3

Open the project in Visual Studio, and build it once.

A number of files brought by the AppDynamics package show up under the project, including Repro1622.AppDynamicsConfig.json. Edit this file and replace its content with:

{
  "controller": {
    "host": "localhost",
    "port": 443,
    "account": "customer1",
    "password": "c025be84-d72d-4f6e-b5b4-b33ce264992b",
    "ssl": true,
    "enable_tls12": true
  },
  "application": {
    "name": "irrelevant",
    "tier": "irrelevant",
    "node": "irrelevant"
  }
}

These values are not important, they just need to be different than the default placeholders for the profiler to activate.

Add a Web.config file to the project, with the following content:

<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%">
      <environmentVariables>
        <environmentVariable name="CORECLR_ENABLE_PROFILING" value="1" />
        <environmentVariable name="CORECLR_PROFILER" value="{39AEABC1-56A5-405F-B8E7-C3668490DB4A}" />

        <environmentVariable name="CORECLR_PROFILER_PATH_32" value="D:\Repro1622\bin\Debug\netcoreapp2.1\AppDynamics.Profiler_x86.dll"/>
        <environmentVariable name="CORECLR_PROFILER_PATH_64" value="D:\Repro1622\bin\Debug\netcoreapp2.1\AppDynamics.Profiler_x64.dll"/>
      </environmentVariables>
    </aspNetCore>
  </system.webServer>
</configuration>

⚠️ Per the AppDynamics docs, the two CORECLR_PROFILER_PATH_* variables must be full paths; adjust as required.

Start debugging (F5) with IIS Express and note BadImageFormatException is thrown at System.AppDomainManager..ctor() when the application starts.

Find the newest .log file under %ProgramData%\AppDynamics\DotNetAgent\Logs\Profiler, which should be similar to:

2018-06-07 18:22:31.086553<info>:Use CoreCLR profiler
2018-06-07 18:22:31.089553<info>:Should instrument standalone profiler process (config found at D:\Repro1622\bin\Debug\netcoreapp2.1\Repro1622.AppDynamicsConfig.json): dotnet as app Repro1622.dll
2018-06-07 18:22:31.092555<info>:Agent Configuration: {"controller":{"account":"customer1","enableTls12":true,"ssl":true,"host":"localhost","password":"***","port":443},"application":{"tier":"irrelevant","node":"irrelevant","name":"irrelevant"}}
2018-06-07 18:22:31.092555<info>:Enabled rejit normally, disabled all NGEN images.
2018-06-07 18:22:31.163553<error>:InjectMEE failed.
2018-06-07 18:22:31.935346<info>:Transforming 57540:clrhost:System.Void System.AppDomainManager..ctor() with [[],[]]

@noahfalk commented on Thu Jun 07 2018

@joseclament - Glad you were able to get the bottom of it and very appreciated that you came back to post your findings in case others hit the same issue in the future! Indeed it looks like your issue had a similar symptom but different root cause than the other ones here. I created #1670 to track any further work there.

@nil4 - Thanks for the repro! Let me try repro your issue so we can figure out if this is the same root cause or something else.


@noahfalk commented on Thu Jun 07 2018

@nil4 - Thanks, I was able to reproduce the problem easily using your excellent instructions! Although the AppDynamics issue is slighly different than the NewRelic one (AppDynamics does not call ICorProfilerInfo::SetILFunctionBody or ICorProfilerFunctionControl::SetILFunctionBody) , the fix I submitted earlier today (#18322) does solve both issues at the same time.

I wasn't sure if you were an employee with AppDynamics or a customer of AppDynamics? For potential short term resolutions to the issue you could either continue to use .Net Core 2.0 or use a daily build of .Net Core that has this fix as soon as the installers are ready for download (I'll write back again when they are). A little further in the future I'm attempting to get this fix into the official servicing releases for 2.1. I can also try to reach out to AppDynamics to see if there are any additional solutions available for the issue.

Let me know if you've got any other questions or concerns?


@nil4 commented on Fri Jun 08 2018

@noahfalk glad I could help, and thank you for the update, much appreciated! I am a customer, and have opened an AppDynamics support ticket about this; will update it shortly with a link to this issue.

@Petermarcu

This comment has been minimized.

Member

Petermarcu commented Jun 13, 2018

@noahfalk is this something we should fix in 2.1 or is this a vNext fix?

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jun 13, 2018

I think we want it in 2.1. I was just attempting to get our ducks in a row testing wise before bringing it up in shiproom.

@mangod3

This comment has been minimized.

mangod3 commented Jun 13, 2018

Yes please hotfix 2.1 with this change, a lot of customers are hitting this already.

@noahfalk noahfalk self-assigned this Jun 15, 2018

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jun 15, 2018

Following up for anyone looking for daily 2.2 preview builds with the fix. The fix is present in Microsoft.NETCore.App version >= 2.2.0-preview1-26610-01. Download runtime daily builds here.

The fix is not yet present in any version of the complete SDK installers. As of the latest SDK, 2.2.100-preview1-009015, it is still bundling the ever so slightly older 2.2.0-preview1-26609-02 runtime.

If you don't want to wait for a complete SDK that bundles the fix you could force it forward yourself:

  1. Install the 2.2.100-preview1-009015 SDK
  2. Remove the 2.2.0-preview1-26609-02 runtime that came with it (delete the folder C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0-preview1-26609-02)
  3. Install the latest runtime build instead

Separately from this I'm working on a hotfix for 2.1.

@sayfulloev

This comment has been minimized.

sayfulloev commented Jun 15, 2018

Looking forward for this fix, as we got hit by it.

@nycdotnet

This comment has been minimized.

nycdotnet commented Jun 21, 2018

Noah we're looking forward to the 2.1 hot fix at Namely. We're on .NET Core 2.0 in prod for many of our services, and we use New Relic. With yesterday's blog post that .NET Core 2.0 will be out of support in ~70 days, we're quite nervous that we will find ourselves in a difficult position soon. Thanks for your work on this issue.

@richlander

This comment has been minimized.

Member

richlander commented Jun 22, 2018

.NET Core 2.0 EOL date has been extended to October 1, 2018 due to this issue.

See: https://twitter.com/dotnet/status/1010269960948465665

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jun 26, 2018

Further follow up for anyone looking for daily 2.2 preview builds. The fix is present in both daily runtime and SDK builds at this point: https://github.com/dotnet/core/blob/master/daily-builds.md.

2.1 servicing is still in progress. Part way through we discovered a 2nd issue (#18602) when using the Visual Studio debugger disassembly window. Although we determined it was pre-existing bug in 2.0 and 2.1, the user visible behavior became worse when combined with the fix for this issue. That 2nd issue now has a fix checked in for daily builds and the 2.1 servicing will include both fixes.

@rhyous

This comment has been minimized.

rhyous commented Jun 29, 2018

I wonder if bug #27907 is related.

@nycdotnet

This comment has been minimized.

nycdotnet commented Jun 29, 2018

We struggled a bit finding the right one, but we're going to try deploying with the .tar.gz for Linux x64 in the "master" column (https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-linux-x64.tar.gz) from here to see if we can get this going now. It has the 3.x banner, which is weird, but we assume this should have the fix. Thanks so much to everyone involved.

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jun 29, 2018

I wonder if bug #27907 is related.

@rhyous - The issue here is in code that is unique to .Net Core and if I understand 27907 correctly it occurs on desktop .NET Framework. I don't believe they are related.

It has the 3.x banner, which is weird

At some point recently I think the 2.2 work went to its own branch and the daily builds are now labeled 3.x. If the issue still persists in that build please let me know and I'll help you get to the bottom of it.

@nycdotnet

This comment has been minimized.

nycdotnet commented Jun 29, 2018

We were indeed able to get Newrelic running on 95c6eff6daa1a69f29c42b2d405400ad44bdec91 3.0.100-preview1-009020. 😄
cc: @abe545

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jun 29, 2018

Thanks for the confirmation!

@abhinavgalodha

This comment has been minimized.

abhinavgalodha commented Jul 5, 2018

We are also having the same issue with AppDynamics Instrumentation. Our application is on asp.net core 2.1 in beta and moving to production next week. Any ideas about the 2.1 Servicing timeline for this issue will help us plan in advance.

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jul 6, 2018

The goal for 2.1.3 servicing release is to ship early to mid August with daily builds potentially available earlier. I reached out to AppDynamics in case they have any additional guidance they want to pass along.

@abhinavgalodha

This comment has been minimized.

abhinavgalodha commented Jul 7, 2018

Thanks @noahfalk for the update.

@Eilon

This comment has been minimized.

Member

Eilon commented Jul 16, 2018

@noahfalk - a customer who was hitting a similar issue to this in aspnet/KestrelHttpServer#2680, and it seems like resolving this issue here will address that. I think the clock is ticking for 2.1.3 - do you know if this is still planned to make it in?

@noahfalk

This comment has been minimized.

Member

noahfalk commented Jul 16, 2018

@Eilon - The changes went into the 2.1.3 release branch a few weeks ago: 1cebf4d

I was hoping to point to an official daily servicing build of 2.1 for developers to try out, but so far I don't think we have those builds being published.

@Eilon

This comment has been minimized.

Member

Eilon commented Jul 16, 2018

@noahfalk got it, thanks for the update!

@mviswana

This comment has been minimized.

mviswana commented Jul 31, 2018

Huge thanks to @noahfalk for proactively reaching out to AppDynamics engineering and partnering with us on these fixes. We had a productive conversation which I believe will help us get robust support for .NET Core 2.1 out the door. Specifically, we are working on uptaking the async/await and ASP.NET Core 2.1 changes. Look for feature updates coming soon. Please reach out again if you have further questions on GitHub or the AppDynamics Community.

Meera Viswanathan
Senior Product Manager, AppDynamics - .NET And Azure

@peacecwz

This comment has been minimized.

peacecwz commented Aug 9, 2018

I couldn't track this issue. This PR is related this issue? #18322 I'm wondering this issue @noahfalk

@mangod3

This comment has been minimized.

mangod3 commented Aug 9, 2018

Is there a more precise ETA for a 2.1 fix to be released?

@martincostello

This comment has been minimized.

martincostello commented Aug 9, 2018

You can use the Early Access Downloads if you can't wait for the final release: https://github.com/aspnet/Home/wiki/.NET-Core-2.1.3-Early-Access-Downloads

@peacecwz

This comment has been minimized.

peacecwz commented Aug 9, 2018

@martincostello Actually we are waiting New Relic for .NET Core 2.1 and they are waiting new version of .NET Core. It will be release soon?

@martincostello

This comment has been minimized.

martincostello commented Aug 9, 2018

@peacecwz I'm under the impression it will ship before the end of next week.

@ghost

This comment has been minimized.

ghost commented Aug 16, 2018

Looks like the August release of the SDK is out (v2.1.4) but it looks like includes coreclr v 2.1.2 still. Does this mean we will have to wait for the September release? Or is there another release coming?

@noahfalk

This comment has been minimized.

Member

noahfalk commented Aug 17, 2018

@rv-calbert - I just looked at our release page and I agree, that is not what I had expected to see. I'm not tight in the loop with the devs that handle the release mechanics but if you hang tight I'm trying to get to the bottom of it. I'll let you know as soon as I hear back from them.

@martincostello

This comment has been minimized.

martincostello commented Aug 17, 2018

I think 2.1.3 is coming with SDK 2.1.401, which hasn’t been formally released yet.

@peacecwz

This comment has been minimized.

peacecwz commented Aug 17, 2018

Why .NET Core 2.1.3 won't be released? If you can't release it before Eid al-Adha 😄

@noahfalk

This comment has been minimized.

Member

noahfalk commented Aug 17, 2018

Release team got back to me and said they are aiming for 2.1.3 next Tuesday. So its still this month, not September. Sorry for the confusion.

@leecow

This comment has been minimized.

Member

leecow commented Aug 19, 2018

Yep, we're priming the pump to get 2.1.3 with the 2.1.401 SDK out to you all on Tuesday.

@peacecwz

This comment has been minimized.

peacecwz commented Aug 21, 2018

.NET Core 2.1.3 is released. This issue was solved? Why is this issue open?

@ZackTech

This comment has been minimized.

ZackTech commented Aug 23, 2018

I am dynamically loading .net standard dll's using MEF on Azure and I am getting Bad IL Format, on my local PC everything works fine. The MEF dlls used to load OK. Also seems to happen on our local IIS server now. Any ideas how I fix the problem? This is an urgent issue for me!

@mikey0000

This comment has been minimized.

mikey0000 commented Aug 23, 2018

@noahfalk

This comment has been minimized.

Member

noahfalk commented Aug 28, 2018

With the release last week of CoreCLR 2.1.3 I'm closing this issue. As best I can tell from my own past testing and reports from the community it does resolve the exception. If there are cases that still reproduce this exception after installing 2.1.3 I'd recommend opening a new issue. Sorry for the trouble this one caused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment