applicationinsights-agent-3.0.2.jar not sending incoming requests logs to Azure Application Insight

[AmudaPalani]

Expected behavior

Setup applicationinsights-agent-3.0.2.jar on my AKS pod. This will continue to send any new incoming requests logs to Azure Application Insight.

Actual behavior

Setup applicationinsights-agent-3.0.2.jar on my AKS pod. This started working and then it stopped. It's not sending any new incoming requests logs to Azure Application Insight.

To Reproduce

Setup this agent as per doc: https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent

Sample Application

If applicable, provide a sample application which reproduces the issue.

System information

Please provide the following information:

• Kubernetes Version: 1.19.6

Logs

2021-03-19 10:58:32.847-05 INFO c.m.a.i.c.c.TransmissionPolicyManager - App throttling is cancelled.

Screenshots

If applicable, add screenshots to help explain your problem.

[trask]

hi @AmudaPalani! can you try with 3.0.3-BETA, and if you still see the issue, post the full applicationinsights.log file?

[AmudaPalani]

@trask we are getting below error.

[AmudaPalani]

2021-03-22 14:28:22.738-05 INFO c.m.applicationinsights.agent - ApplicationInsights Java Agent 3.0.3-BETA started successfully
2021-03-22 14:28:34.284-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441111689-2298398013097503583.trn' does not exist
2021-03-22 14:28:36.285-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441119276-7635388759777416967.trn' does not exist
2021-03-22 14:28:38.285-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441129270-5425245293075875614.trn' does not exist
2021-03-22 14:28:56.409-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441130816-2595047195535987551.trn' does not exist
2021-03-22 14:28:58.410-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441130820-12660904828838094124.trn' does not exist
2021-03-22 14:29:17.433-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441134462-2983047215420502753.trn' does not exist
2021-03-22 14:29:49.777-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441135959-1004901636390337448.trn' does not exist
2021-03-22 14:29:51.778-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441140474-4599192382409736796.trn' does not exist
2021-03-22 14:29:53.778-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441149270-6220368910226370810.trn' does not exist
2021-03-22 14:29:55.779-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441157104-9880024441440588526.trn' does not exist
2021-03-22 14:29:57.779-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441159273-13682851555644349563.trn' does not exist
2021-03-22 14:30:04.386-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441162198-576522069603271398.trn' does not exist
2021-03-22 14:30:06.387-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441166498-1879128208032368390.trn' does not exist
2021-03-22 14:30:08.387-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441172204-12713071665407889679.trn' does not exist
2021-03-22 14:30:14.396-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441178291-5319927742832375435.trn' does not exist
2021-03-22 14:30:16.397-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441179271-15104417645678539531.trn' does not exist
2021-03-22 14:30:18.397-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441183424-12360443826989613950.trn' does not exist
2021-03-22 14:30:24.402-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441188557-12362398487705660652.trn' does not exist
2021-03-22 14:30:26.403-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441189270-15172260893267724308.trn' does not exist
2021-03-22 14:30:28.404-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441193664-7753680347484454687.trn' does not exist
2021-03-22 14:31:34.394-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441198536-3836495600160147920.trn' does not exist
2021-03-22 14:31:36.395-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441209269-8868366947918296986.trn' does not exist
2021-03-22 14:31:38.395-05 ERROR c.m.a.i.c.c.TransmissionFileSystemOutput - Rename To Temporary Name failed, exception: java.io.FileNotFoundException: Source '/tmp/root/transmi
ssions/Transmission-1616441219270-11013841219334941311.trn' does not exist
2021-03-22 14:32:06.926-05 WARN c.m.a.a.i.AppIdSupplier$GetAppIdTask - In the last 5 minutes, the following operation has failed 5 times (out of 5):
unable to retrieve appId

• exception sending request to https://centralus-0.in.applicationinsights.azure.com/api/profiles/fc4bd193-aa2f-470a-9502-001b5d7b1017/appId (5 times)
  2021-03-22 14:37:06.926-05 WARN c.m.a.a.i.AppIdSupplier$GetAppIdTask - In the last 5 minutes, the following operation has failed 5 times (out of 5):
  unable to retrieve appId
• exception sending request to https://centralus-0.in.applicationinsights.azure.com/api/profiles/fc4bd193-aa2f-470a-9502-001b5d7b1017/appId (5 times)
  2021-03-22 14:42:06.926-05 WARN c.m.a.a.i.AppIdSupplier$GetAppIdTask - In the last 5 minutes, the following operation has failed 5 times (out of 5):
  unable to retrieve appId
• exception sending request to https://centralus-0.in.applicationinsights.azure.com/api/profiles/fc4bd193-aa2f-470a-9502-001b5d7b1017/appId (5 times)

[AmudaPalani]

I am seeing the same error. The agent started working and then it stopped. It's not sending any new incoming requests logs to Azure Application Insight.

[AmudaPalani]

2021-04-22 10:15:57.773-05 DEBUG c.m.a.i.c.c.TransmissionFileSystemOutput - Data persisted to file. To be sent when the network is available.
2021-04-22 10:16:02.770-05 DEBUG c.m.a.w.i.c.CdsProfileFetcher - The profile fetch task will not execute for next 240 minutes. Max number of retries reached.
2021-04-22 10:16:02.771-05 DEBUG c.m.a.agent.internal.AppIdSupplier - Application correlation Id could not be retrieved (e.g. task may be pending or failed)
2021-04-22 10:16:02.772-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=86751376beec5164aa3f9c67e04fd22e, spanId=c967ddde1ba56778, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=86751376beec5164aa3f9c67e04fd22e, spanIdAsHexString=ae88a334f85bb371, traceFlags=1, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=OperationHandler.handle, kind=INTERNAL, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=57, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-1}, capacity=1000, totalAddedValues=2}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104562771350503, endEpochNanos=1619104562772308811}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=57, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-1}, capacity=1000, totalAddedValues=2}, totalAttributeCount=2, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=OperationHandler.handle, endEpochNanos=1619104562772308811, internalHasEnded=true}
2021-04-22 10:16:02.772-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=86751376beec5164aa3f9c67e04fd22e, spanId=ae88a334f85bb371, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=/health, kind=SERVER, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=57, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-1, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=56916, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104562771001200, endEpochNanos=1619104562772556013}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=57, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-1, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=56916, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, totalAttributeCount=10, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=/health, endEpochNanos=1619104562772556013, internalHasEnded=true}
2021-04-22 10:16:03.918-05 DEBUG c.m.a.w.i.c.CdsProfileFetcher - The profile fetch task will not execute for next 240 minutes. Max number of retries reached.
2021-04-22 10:16:03.918-05 DEBUG c.m.a.agent.internal.AppIdSupplier - Application correlation Id could not be retrieved (e.g. task may be pending or failed)
2021-04-22 10:16:03.919-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=e83c4eae496c1818a2200124c6439e3c, spanId=22094576fd8da6eb, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=e83c4eae496c1818a2200124c6439e3c, spanIdAsHexString=f769028a41eed44b, traceFlags=1, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=OperationHandler.handle, kind=INTERNAL, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=58, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-2}, capacity=1000, totalAddedValues=2}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104563918273802, endEpochNanos=1619104563919162709}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=58, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-2}, capacity=1000, totalAddedValues=2}, totalAttributeCount=2, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=OperationHandler.handle, endEpochNanos=1619104563919162709, internalHasEnded=true}
2021-04-22 10:16:03.919-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=e83c4eae496c1818a2200124c6439e3c, spanId=f769028a41eed44b, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=/health, kind=SERVER, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=58, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-2, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=56928, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104563918000700, endEpochNanos=1619104563919379011}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=58, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-2, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=56928, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, totalAttributeCount=10, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=/health, endEpochNanos=1619104563919379011, internalHasEnded=true}
2021-04-22 10:16:07.774-05 DEBUG c.m.a.i.c.c.TransmissionFileSystemOutput - Data persisted to file. To be sent when the network is available.
2021-04-22 10:16:12.072-05 DEBUG c.m.a.w.i.c.CdsProfileFetcher - The profile fetch task will not execute for next 240 minutes. Max number of retries reached.
2021-04-22 10:16:12.073-05 DEBUG c.m.a.agent.internal.AppIdSupplier - Application correlation Id could not be retrieved (e.g. task may be pending or failed)
2021-04-22 10:16:12.103-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=84cc9bb514aee43c627bd2466e03b731, spanId=ba5ea4a0f29c32ec, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=HTTP GET, kind=CLIENT, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=53, AttributeKeyImpl{getType=STRING, key=thread.name}=OkHttp https://kubernetes.default.svc/..., AttributeKeyImpl{getType=STRING, key=net.transport}=IP.TCP, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.flavor}=1.1, AttributeKeyImpl{getType=STRING, key=net.peer.name}=kubernetes.default.svc, AttributeKeyImpl{getType=STRING, key=http.url}=https://kubernetes.default.svc/api/v1/namespaces/default/configmaps?watch=true, AttributeKeyImpl{getType=LONG, key=http.status_code}=403}, capacity=1000, totalAddedValues=8}, status=ImmutableStatus{statusCode=ERROR, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104572072000900, endEpochNanos=1619104572102555522}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=53, AttributeKeyImpl{getType=STRING, key=thread.name}=OkHttp https://kubernetes.default.svc/..., AttributeKeyImpl{getType=STRING, key=net.transport}=IP.TCP, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.flavor}=1.1, AttributeKeyImpl{getType=STRING, key=net.peer.name}=kubernetes.default.svc, AttributeKeyImpl{getType=STRING, key=http.url}=https://kubernetes.default.svc/api/v1/namespaces/default/configmaps?watch=true, AttributeKeyImpl{getType=LONG, key=http.status_code}=403}, capacity=1000, totalAddedValues=8}, totalAttributeCount=8, totalRecordedEvents=0, status=ImmutableStatus{statusCode=ERROR, description=null}, name=HTTP GET, endEpochNanos=1619104572102555522, internalHasEnded=true}
2021-04-22 10:16:12.104-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=6d70845c3469a8256c317a4d5873d007, spanId=6ff5aced150d5fdb, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=Exec Failure: HTTP 403, Status: 403 - , kind=INTERNAL, attributes=AttributesMap{data={AttributeKeyImpl{getType=BOOLEAN, key=applicationinsights.internal.log}=true, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_level}=WARN, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.logger_name}=io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager, AttributeKeyImpl{getType=LONG, key=thread.id}=53, AttributeKeyImpl{getType=STRING, key=thread.name}=OkHttp https://kubernetes.default.svc/..., AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_error_stack}=java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess(RealWebSocket.java:224)
at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:195)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
}, capacity=1000, totalAddedValues=6}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104572104000400, endEpochNanos=1619104572104049300}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=BOOLEAN, key=applicationinsights.internal.log}=true, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_level}=WARN, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.logger_name}=io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager, AttributeKeyImpl{getType=LONG, key=thread.id}=53, AttributeKeyImpl{getType=STRING, key=thread.name}=OkHttp https://kubernetes.default.svc/..., AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_error_stack}=java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess(RealWebSocket.java:224)
at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:195)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
}, capacity=1000, totalAddedValues=6}, totalAttributeCount=6, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=Exec Failure: HTTP 403, Status: 403 - , endEpochNanos=1619104572104049300, internalHasEnded=true}
2021-04-22 10:16:12.104-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=63a5b6361aa8cb3987f4283f5f265219, spanId=a250a6a1555fb6b7, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=Current reconnect backoff is 32000 milliseconds (T5), kind=INTERNAL, attributes=AttributesMap{data={AttributeKeyImpl{getType=BOOLEAN, key=applicationinsights.internal.log}=true, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_level}=INFO, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.logger_name}=io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager, AttributeKeyImpl{getType=LONG, key=thread.id}=54, AttributeKeyImpl{getType=STRING, key=thread.name}=scheduleReconnect|Executor for Watch 58638094}, capacity=1000, totalAddedValues=5}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104572104000400, endEpochNanos=1619104572104003000}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=BOOLEAN, key=applicationinsights.internal.log}=true, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.log_level}=INFO, AttributeKeyImpl{getType=STRING, key=applicationinsights.internal.logger_name}=io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager, AttributeKeyImpl{getType=LONG, key=thread.id}=54, AttributeKeyImpl{getType=STRING, key=thread.name}=scheduleReconnect|Executor for Watch 58638094}, capacity=1000, totalAddedValues=5}, totalAttributeCount=5, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=Current reconnect backoff is 32000 milliseconds (T5), endEpochNanos=1619104572104003000, internalHasEnded=true}
2021-04-22 10:16:12.770-05 DEBUG c.m.a.w.i.c.CdsProfileFetcher - The profile fetch task will not execute for next 240 minutes. Max number of retries reached.
2021-04-22 10:16:12.770-05 DEBUG c.m.a.agent.internal.AppIdSupplier - Application correlation Id could not be retrieved (e.g. task may be pending or failed)
2021-04-22 10:16:12.772-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=6145784a81a9e28942742277d325230a, spanId=b4cb85362fbccec9, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=6145784a81a9e28942742277d325230a, spanIdAsHexString=ffbabbd083f91967, traceFlags=1, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=OperationHandler.handle, kind=INTERNAL, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=59, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-3}, capacity=1000, totalAddedValues=2}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104572770343303, endEpochNanos=1619104572771262909}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=59, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-3}, capacity=1000, totalAddedValues=2}, totalAttributeCount=2, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=OperationHandler.handle, endEpochNanos=1619104572771262909, internalHasEnded=true}
2021-04-22 10:16:12.772-05 DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=6145784a81a9e28942742277d325230a, spanId=ffbabbd083f91967, parentSpanContext=ImmutableSpanContext{traceIdAsHexString=00000000000000000000000000000000, spanIdAsHexString=0000000000000000, traceFlags=0, traceState=ArrayBasedTraceState{entries=[]}, remote=false}, name=/health, kind=SERVER, attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=59, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-3, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=57054, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, status=ImmutableStatus{statusCode=UNSET, description=null}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1619104572770000800, endEpochNanos=1619104572771486611}, resolvedLinks=[], resolvedEvents=[], attributes=AttributesMap{data={AttributeKeyImpl{getType=LONG, key=thread.id}=59, AttributeKeyImpl{getType=STRING, key=thread.name}=http-nio-8080-exec-3, AttributeKeyImpl{getType=STRING, key=net.peer.ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=net.peer.port}=57054, AttributeKeyImpl{getType=STRING, key=http.method}=GET, AttributeKeyImpl{getType=STRING, key=http.user_agent}=kube-probe/1.19, AttributeKeyImpl{getType=STRING, key=http.url}=http://10.2.0.69:8080/health, AttributeKeyImpl{getType=STRING, key=http.flavor}=HTTP/1.1, AttributeKeyImpl{getType=STRING, key=http.client_ip}=10.2.0.56, AttributeKeyImpl{getType=LONG, key=http.status_code}=200}, capacity=1000, totalAddedValues=10}, totalAttributeCount=10, totalRecordedEvents=0, status=ImmutableStatus{statusCode=UNSET, description=null}, name=/health, endEpochNanos=1619104572771486611, internalHasEnded=true}

[trask]

hi @AmudaPalani, are you still having issues? I don't see any WARN or ERROR in your most recent log above, so not clear what the current status of your issue is. thx!

[trask]

Closing, but please feel free to post further comment or create a new issue.