No Trace logging from App Service

[Xabin]

I'm having trouble getting TRACE logging from Logback to Application Insights when running my application on an Azure App Service.
Java version: 8
Spring Boot version: 1.5.13.RELEASE
Spring Boot starter version: 1.0.0-BETA
Logback version: 1.1.11

I'm using the default configuration of the Spring Boot starter. I have only configured a azure.application-insights.instrumentation-key.
I cannot reproduce this issue locally. Both when running the application from within Eclipse and from a packaged Spring Boot jar, I'm getting TRACE logs to my Application Insights instance.
When running on the Azure App Service I am getting both REQUEST and EXCEPTION logs, it is only TRACE from Logback that is missing.
The problem is actually only visible on one of our Azure environments. We have two staging environments and one for production. One of the staging environments and production is getting TRACE logs as expected. Each environment uses different Application Insights instances.
The Logback logfile produced on the filesystem on Azure contains the expected logging statements. There are no exceptions related to Application Insights.

How can I troubleshoot this issue further? I assume that the issue is related to the Spring Boot starter, but I don't know how to dig deeper.

[dhaval24]

@Xabin let me restate what you referred above:

1. You are using AI SpringBoot starter and ApplicationInsights LogBack adapter to send Logback traces to AI.
2. The issue only happens in App Services environment. You cannot reproduce the issue locally.
3. In App Service environment - issue only happens in one of your staging and prod environment. Other staging and prod environment on app service gets all the logback traces?

If above is correct I will recommend doing the following steps:

1. azure.application-insights.logger.level=TRACE (add this property in your application.properties to enable debug logging for the SDK). This would help us know if something is going wrong on the SDK end.
2. What type of file you are deploying on app service? Is it a war / jar ?
3. Are you using one of the blessed Tomcat image in App Service?
4. Check you have exactly same settings in both the environments you mentioned (i.e one where the traces are flowing and one where traces are not flowing).

[Xabin]

Thank you for the reply!

1. I have added azure.application-insights.logger.level=TRACE to my configuration and attached an excerpt of the logfile produced during and just after application start.
   log.txt
   I can't see anything strange in it. The same AI statements are produced when I run the application locally. When running locally I get the Logback statements as TRACE logging in Application insights ("Starting Application on..." etc), but when running the application on the Azure App Service using the same Spring profile they don't show up.

2. I'm deploying it as a jar.

3. The application is running on Spring Boot's embedded Tomcat server.

4. When running the same Spring profile locally and on the App Service, the settings are exactly the same (even the AI instrumentation key).

[dhaval24]

@Xabin thank you very much for your response. Could you please upload a trimmed down version of your application for further investigation? Please make sure to remove all PIDs including Application Insights instrumentation key.

Further, the way how App Service handles tomcat console logs could be slightly different (i.e locations where it is produced). Upon eliminating the SDK concerns I might need to consult App services team for the same.

[Xabin]

example.zip
Here is a very very trimmed down version. I don't think you'll be able to reproduce the problem with it, because as I said, it works fine locally and on one staging environment, but not the other.

Thank you for trying to help!

[dhaval24]

@Xabin if this is the case, then it would be very hard to reproduce and find the root cause. In many scenarios App Services have their different security configuration and places where the logs are seen. For example recently @littleaj had found something around the same. Arthur, would you mind sharing your observations here if you have any. Further @Xabin have you tried checking out this issue with App Services team?

[Xabin]

No, I didn't try the App Services team. They are not as easy for me to contact as you, since I have to go through my company's infrastructure department to file a support ticket :p
However, I just tried deleting and recreating the Application Insights instance, configured the new instrumentation key, and now it just works. I'm getting all the logging I expect - TRACE, REQUEST and EXCEPTION.
Feel free to close this issue, and thank you for all the help!

[Xabin]

Oh, I guess I can close it myself. Thanks :)