Exception : Binary format of the specified custom attribute was invalid when reading the configuration #330
Comments
|
While it's possible this is a Also, this should be obvious but... can you please format the exception stacktrace in the OP properly - nobody is going to read it if you leave your homework messy like that! Resolving the issue is going to be the normal process of narrowing things down and ruling things out. At the point where you have a failing test that's not "here are 57 interlinked things that together give me a random problem", it becomes a workable issue for this repo. Until then, StackOverflow is going to be the best help (Most people watching this repo also watch the SO |
|
@bartelink I agree that the exception format is not readable, I'm going to fix this. For the rest, well I have two microservices that had the same systematic behavior when upgrading the Serilog* nuget and which "recovered" after downgrading them. My intention is not to put blame on Serilog package which I use for years in production now, but really to understand what happened, maybe it might be a bug or a combination of circumstance (between Elk version/data/settings + Serilog) I don't know, and that's why I require the help of the community. Sorry if I choose the wrong GH issue template or channel for this, I'll try with SO also. |
Best to close this and move it to SO then; feel free to post a link to the SO post as you close - clearly your build+deploy pipeline is consistently leading to a mismatched or mangled set of DLLs. That can be either a matter of turning up warning levels re mismatched component versions, and/or figuring out what's manglign and/or mismatching them after the fact. Wild guess is that some part of your system is still using older versions and they're all getting merged, with the outcome that new stuff is touching old stuff. Given the prevalence of Serilog across all sorts of companies and environments, an actual minimal repro would show up in this repro pretty soon if there was an actual issue (and that'd be turned into a failing regression test first before being fixed) |
|
Thanks for your help, here is the SO link https://stackoverflow.com/questions/73987270/serilog-to-elasticsearch-exception-on-creating-the-builder |
|
Hi @NicolasREY69330 , |
|
StackTrace has link to this change. Not sure what happens here, need to investigate |
I'll try on next week and keep you informed thanks for your help |
|
@skomis-mm <PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" />and when downgrading the package Serilog.Settings.Configuration from 3.4.0 to 3.3.0 it works as expected <PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" /> |
|
Hi all, just to confirm this issue happened to me exactly the same:
The fix for now is to downgrade from 3.4.0 to: |
|
👍 My bet is that you're building for one version of .NET but deploying to a slightly earlier one. How are your apps being published (command lines etc.) and what's the exact Another possibility is that somewhere a method is decorated with an attribute that triggers:
but this one could be a long shot, there are very few reports of this over a very long time frame. In either case, unfortunately, there's a very strong (nearly 100%) probability that your deployment is broken (either because of a misconfiguration, or a tooling bug), and it's only now being revealed because the change linked by @skomis-mm forces slightly more assembly metadata to be loaded than the earlier version did. Definitely still open to helping here, but just letting you know why it's not something we're likely to find/reproduce outside your environments, so a lot more detailed info about those will be needed. If you're super keen to track it down, breaking on all exceptions in a debugger and examining the attribute type that's being inspected on the line linked above should provide some useful clues. Hope this helps! |
|
Hi, thanks for the reply, indeed i also was thinking the problem probably could be the target framework version being lesser than the local installed one. I understand the change introduced and on my case i doubt the reflection methods are picking any attributes causing issues on my minimal app code, yet it can also be because of some code on the framework itself.. |
|
For us:
What is really weird :
Tested with 3 different microservices. We use no custom attributes at all. The only specificity is that the Serilog* lib are bring by a library of our own (they are gathered among other into a single toolbox library) |
|
Thank you @skomis-mm for the workaround. using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Serilog;
using System;
using Serilog.Core;
namespace MyApplication
{
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
// Logger
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(config)
.CreateLogger();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
}
}
}<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<UserSecretsId>02f6ecbb-6e11-43c0-8694-105088e11f24</UserSecretsId>
<DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
<DockerfileContext>..\..</DockerfileContext>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
<NoWarn>1701;1702;1591</NoWarn>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.DependencyModel" Version="6.0.0" />
<PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.15.0" />
<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
</ItemGroup>
</Project> |
|
Hi @NicolasREY69330 , @nunorelvao , |
|
Hi @skomis-mm with the following code, it works public void ConfigureServices(IServiceCollection services)
{
// Logger
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(Configuration)
.Enrich.WithElasticApmCorrelationInfo()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://my-elk-dns:9200"))
{
AutoRegisterTemplate = true,
IndexFormat = "mslogs-{0:yyyy.MM.dd}",
DetectElasticsearchVersion = true,
RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
FailureCallback = e => Console.WriteLine($"Unable to submit event {e?.RenderMessage()} to ElasticSearch. Exception : " + e?.Exception?.ToString()),
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
EmitEventFailureHandling.WriteToFailureSink |
EmitEventFailureHandling.RaiseCallback,
BufferCleanPayload = (failingEvent, statuscode, exception) =>
{
dynamic e = JObject.Parse(failingEvent);
return JsonConvert.SerializeObject(new Dictionary<string, object>()
{
{ "action", "DeniedByElasticSearch"},
{ "@timestamp",e["@timestamp"]},
{ "level","Error"},
{ "message","Error: "+e.message},
{ "messageTemplate",e.messageTemplate},
{ "failingStatusCode", statuscode},
{ "failingException", exception}
});
},
CustomFormatter = new EcsTextFormatter()
})
.CreateLogger();
Log.Information("My logger works");
Console.WriteLine("Two previous step passed");
Log.CloseAndFlush();
System.Threading.Thread.Sleep(5000);
Console.WriteLine("All previous step passed");
} <ItemGroup>
<PackageReference Include="Microsoft.Extensions.DependencyModel" Version="6.0.0" />
<PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.15.0" />
<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.5.0-dev" />
</ItemGroup>The CI logs highlight the exact version applied Logs pushed to Elk Log in the standard output By the way I'm surprised that I don't find my elk log into the standard output as the Serilog setting is (WriteTo Console) "Serilog": {
"Using": [],
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Warning",
"System": "Information",
"Elastic": "Warning",
"Apm": "Warning"
}
},
"WriteTo": [
{
"Name": "Console"
}
],
"Enrich": [
"FromLogContext",
"WithMachineName",
"WithProcessId",
"WithThreadId"
],
"Properties": {
"ApplicationName": "myapplication"
}
} |
|
Hi, |
|
@NicolasREY69330 @alexandrejulien @nunorelvao Do any of you by any chance run Dynatrace in your prod environment? |
|
Yes on our company uses Dynatrace on all environments to monitor all
services and servers. If this is related on my case Dynatrace runs to
monitor all our servers in cloud on my case was Kubernetes on AWS
deployment on Dev environment but is the sane as production just a
different namespace.
Unfortunately i am currently working in other projects and the one with
problems i did not had time yet to retest with new version, hope i can do
it soon, not sure if this info helps for now.
…On Wed, Nov 23, 2022, 21:06 Tor-Erik Hagen ***@***.***> wrote:
@NicolasREY69330 <https://github.com/NicolasREY69330> @alexandrejulien
<https://github.com/alexandrejulien> @nunorelvao
<https://github.com/nunorelvao> Do any of you by any chance run Dynatrace
in your prod environment?
—
Reply to this email directly, view it on GitHub
<#330 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAROQUPNUPIEDJA7HON47QTWJ2BN5ANCNFSM6AAAAAAQ7SNY54>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi yes, we use dynatrace on our prod environment only, typically the only one which throws the exception. Do you think it's linked ? @torerikh |
|
@NicolasREY69330 We're seeing the exact same problem in our prod-environment and are unable to reproduce anywhere else than prod. Stroke me when seeing the mention of APM that it might be related to APM solution and we're also using Dynatrace. I'll attempt to get it verified tomorrow by testing to disable oneagent deep monitoring on the process in question in Dynatrace. I'll also test the 3.5.0-dev version to see as we're able to reproduce this consistently now. |
|
@torerikh we face the exact same situation. Really excited to get your conclusion! |
|
@torerikh did you manage to link the problem to Dynatrace ? |
|
Same situation here. Upgraded Serilog.Settings.Configuration to version 3.4.0 and everything worked fine in development, test and production. Then, a customer using Dynatrace reported the problem. Downgrading Serilog.Settings.Configuration to 3.3.0 fixed the issue. |
|
#333 was not still published |
|
Opened #343 👍 |
Description
Hi,
when upgrading to these packages
I get an exception when creating the logger with
with the assumed configuration:
It throws the following exception :
Downgrading to previous versions solved the problem
Reproduction
The problem occurred only in our Production environment, it was ok in local and in uat environment.
We connect against a containerized Elasticsearch (v 7.17.1)
Expected behavior
No crash on logger creation
Additional context
Kubernetes hosting in AKS
The text was updated successfully, but these errors were encountered: