New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecated trace headers #467

Closed
adriancole opened this Issue Dec 9, 2016 · 11 comments

Comments

Projects
None yet
3 participants
@adriancole
Contributor

adriancole commented Dec 9, 2016

From @sankarbalu on December 9, 2016 7:4

I have a simple zipkin server with the following pom. This worked well for a while.

From this week, i am seeing the following warning log message in a loop that is filling up my log files that prevents from starting the server. I am not sure why, because to my knowledge no changes were done to the code or config.
'2016-12-09 12:06:41.402 WARN [zipkin-server,b48ab054e4dd157c,c75ad2801a23120c,true] [zipkin-server,,,true] 8924 --- [sleuth.sleuth-1] o.s.c.s.z.s.ConvertToZipkinSpanList : Message tracing cycle detected for: org.springframework.cloud.sleuth.stream.Spans@48848018
2016-12-09 12:06:41.403 WARN [zipkin-server,,,] [zipkin-server,,,] 8924 --- [sleuth.sleuth-1] o.s.c.s.i.m.MessagingSpanExtractor : Deprecated trace headers detected. Please upgrade Sleuth to 1.1 or start sending headers present in the TraceMessageHeaders class'

As per the PR (https://github.com/spring-cloud/spring-cloud-sleuth/pull/335/files), 1.0.4 version of sleuth dependencies should work. I verified the versions that come with Brixton.SR4 and it happened to be '1.0.4.RELEASE' for spring-cloud-sleuth-core and spring-cloud-sleuth-stream. I also tried to update the cloud dependencies version to 'Camden.RELEASE', that updated the sleuth dependency versions to ''1.0.9.RELEASE', but the problem persists.

Any pointers on how to resolve this issue?

Original POM:

org.springframework.boot
spring-boot-starter-parent
1.4.0.RELEASE

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>Brixton.SR4</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
 <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-sleuth-zipkin-stream</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-eureka</artifactId>
	</dependency>
	<dependency>
		<groupId>org.projectlombok</groupId>
		<artifactId>lombok</artifactId>
		<scope>provided</scope>
	</dependency>
</dependencies>

Copied from original issue: openzipkin/zipkin#1430

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Dec 10, 2016

Contributor

The thing is that at some point in time we were using message headers that we're incompatible with jms spec. That's why we have migrated to the new approach while still being backwards compatible. The note over there should be about migrating to 1.2.0 not 1.1.

In other words you have nothing to worry about and just upgrade all apps to latest versions of Sleuth.

Contributor

marcingrzejszczak commented Dec 10, 2016

The thing is that at some point in time we were using message headers that we're incompatible with jms spec. That's why we have migrated to the new approach while still being backwards compatible. The note over there should be about migrating to 1.2.0 not 1.1.

In other words you have nothing to worry about and just upgrade all apps to latest versions of Sleuth.

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Dec 11, 2016

Contributor

@sankarbalu does it make any sense?

Contributor

marcingrzejszczak commented Dec 11, 2016

@sankarbalu does it make any sense?

@sankarbalu

This comment has been minimized.

Show comment
Hide comment
@sankarbalu

sankarbalu Dec 12, 2016

@marcingrzejszczak , The latest version of spring-cloud-sleuth-core i can see at https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-sleuth-core is 1.1.0.RELEASE. That is the version i get along with spring-cloud-sleuth-zipkin-stream-1.1.0.RELEASE as a transitive dependency as well. Where can i find the 1.2.0 version of sleuth dependencies?

@marcingrzejszczak , The latest version of spring-cloud-sleuth-core i can see at https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-sleuth-core is 1.1.0.RELEASE. That is the version i get along with spring-cloud-sleuth-zipkin-stream-1.1.0.RELEASE as a transitive dependency as well. Where can i find the 1.2.0 version of sleuth dependencies?

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Dec 12, 2016

Contributor

1.2.0 is not yet released. What you have to do is start using the new header names. Most likely there is an application in your system that uses an old version of sleuth or there are some messages that have not been taken off a topic or queue that use the old header names

Contributor

marcingrzejszczak commented Dec 12, 2016

1.2.0 is not yet released. What you have to do is start using the new header names. Most likely there is an application in your system that uses an old version of sleuth or there are some messages that have not been taken off a topic or queue that use the old header names

@sankarbalu

This comment has been minimized.

Show comment
Hide comment
@sankarbalu

sankarbalu Dec 21, 2016

@marcingrzejszczak sorry about the delay. I tried clearing up the queues and adding 1.1.0 version of sleuth dependencies to the client applications. But still i get the deprecated header warnings.

How can I ensure that the client does not send the deprecated headers? Can it be configured at the client side? Also, how can i see the sleuth messages generated for zipkin server?

Currently, i have the following log format defined in 'logback-spring.xml', which has the deprecated format. What should be ideal format ?

@marcingrzejszczak sorry about the delay. I tried clearing up the queues and adding 1.1.0 version of sleuth dependencies to the client applications. But still i get the deprecated header warnings.

How can I ensure that the client does not send the deprecated headers? Can it be configured at the client side? Also, how can i see the sleuth messages generated for zipkin server?

Currently, i have the following log format defined in 'logback-spring.xml', which has the deprecated format. What should be ideal format ?

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Dec 21, 2016

Contributor

So in general we're removing the old headers in Dalston (Sleuth 1.2.0). So until then you'll see these entries IMO cause for backwards compatibility we're sending both the new ones and the old ones.

Also, how can i see the sleuth messages generated for zipkin server?

What do you mean? There are no message related spans on the Zipkin side? Are you using Kafka or RabbitMQ as a broker?

Currently, i have the following log format defined in 'logback-spring.xml', which has the deprecated format. What should be ideal format ?

Can you show your file? In general we're translating the message headers into Zipkin related entries that we're putting into MDC. So you shouldn't reference the message related entries in your logback file.

Contributor

marcingrzejszczak commented Dec 21, 2016

So in general we're removing the old headers in Dalston (Sleuth 1.2.0). So until then you'll see these entries IMO cause for backwards compatibility we're sending both the new ones and the old ones.

Also, how can i see the sleuth messages generated for zipkin server?

What do you mean? There are no message related spans on the Zipkin side? Are you using Kafka or RabbitMQ as a broker?

Currently, i have the following log format defined in 'logback-spring.xml', which has the deprecated format. What should be ideal format ?

Can you show your file? In general we're translating the message headers into Zipkin related entries that we're putting into MDC. So you shouldn't reference the message related entries in your logback file.

@sankarbalu

This comment has been minimized.

Show comment
Hide comment
@sankarbalu

sankarbalu Dec 22, 2016

So until sleuth 1.2.0 is released, is there any other option to suppress these deprecated messages? I dont see any problems on the client side, but the zipkin server goes into a loop and ultimately ends going out of memory.

I am using RabbitMQ. Rather I guess my question should be, other than the Zipkin UI, is there a way to see the messages generated from sleuth clients apart from the logs? Like a rest endpoint ?

The log pattern used in logback.xml is

	<property name="SLEUTH_LOG_PATTERN"
			  value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr([${springAppName:-},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]){yellow} %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>

Here are some logs generated

2016-11-01 16:55:53.897  INFO [costing-service,1f1ea35f02a9704e,faf84a64d5ab415c,false] [costing-service,1f1ea35f02a9704e,faf84a64d5ab415c,false] 10252 --- [nio-9002-exec-1] c.a.prodcreation.rest.CostingController  : Received costing hello request
2016-11-01 16:55:56.612  INFO [costing-service,62f45282152088e,58173665de2e32a0,false] [costing-service,62f45282152088e,58173665de2e32a0,false] 10252 --- [nio-9002-exec-2] c.a.prodcreation.rest.CostingController  : Found a CIF object Cif(id=1, cifPercent=0.0, factoryCountry=CHN, materialSupplierCountry=DE, validFrom=null, validTo=null)
2016-11-01 16:55:58.278  INFO [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] 10252 --- [nio-9002-exec-3] c.a.prodcreation.rest.CostingController  : Received cif request for id 1
2016-11-01 16:55:58.298  INFO [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] 10252 --- [nio-9002-exec-3] c.a.prodcreation.rest.CostingController  : Found a CIF object Cif(id=1, cifPercent=0.0, factoryCountry=CHN, materialSupplierCountry=DE, validFrom=null, validTo=null)

sankarbalu commented Dec 22, 2016

So until sleuth 1.2.0 is released, is there any other option to suppress these deprecated messages? I dont see any problems on the client side, but the zipkin server goes into a loop and ultimately ends going out of memory.

I am using RabbitMQ. Rather I guess my question should be, other than the Zipkin UI, is there a way to see the messages generated from sleuth clients apart from the logs? Like a rest endpoint ?

The log pattern used in logback.xml is

	<property name="SLEUTH_LOG_PATTERN"
			  value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr([${springAppName:-},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]){yellow} %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>

Here are some logs generated

2016-11-01 16:55:53.897  INFO [costing-service,1f1ea35f02a9704e,faf84a64d5ab415c,false] [costing-service,1f1ea35f02a9704e,faf84a64d5ab415c,false] 10252 --- [nio-9002-exec-1] c.a.prodcreation.rest.CostingController  : Received costing hello request
2016-11-01 16:55:56.612  INFO [costing-service,62f45282152088e,58173665de2e32a0,false] [costing-service,62f45282152088e,58173665de2e32a0,false] 10252 --- [nio-9002-exec-2] c.a.prodcreation.rest.CostingController  : Found a CIF object Cif(id=1, cifPercent=0.0, factoryCountry=CHN, materialSupplierCountry=DE, validFrom=null, validTo=null)
2016-11-01 16:55:58.278  INFO [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] 10252 --- [nio-9002-exec-3] c.a.prodcreation.rest.CostingController  : Received cif request for id 1
2016-11-01 16:55:58.298  INFO [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] [costing-service,9f2b315dd2beccd9,5c220564c7d3d4ad,false] 10252 --- [nio-9002-exec-3] c.a.prodcreation.rest.CostingController  : Found a CIF object Cif(id=1, cifPercent=0.0, factoryCountry=CHN, materialSupplierCountry=DE, validFrom=null, validTo=null)
@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Dec 22, 2016

Contributor

So until sleuth 1.2.0 is released, is there any other option to suppress these deprecated messages? I dont see any problems on the client side, but the zipkin server goes into a loop and ultimately ends going out of memory.

This is interesting... Why is it going into a loop? What is it trying to endlessly process?

The log pattern used in logback.xml is

Logging pattern doesn't really matter

Contributor

marcingrzejszczak commented Dec 22, 2016

So until sleuth 1.2.0 is released, is there any other option to suppress these deprecated messages? I dont see any problems on the client side, but the zipkin server goes into a loop and ultimately ends going out of memory.

This is interesting... Why is it going into a loop? What is it trying to endlessly process?

The log pattern used in logback.xml is

Logging pattern doesn't really matter

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
Contributor

marcingrzejszczak commented Dec 29, 2016

@sankarbalu

This comment has been minimized.

Show comment
Hide comment
@sankarbalu

sankarbalu Jan 1, 2017

Sorry about the delay. I simply updated the Zipkin log levels to ERROR to avoid the infinite WARN logs that were causing the OOM error on zipkin server. As sleuth 1.2.0 is going to send only the new header versions, I guess this quick fix should keep us going in the short run.

Sorry about the delay. I simply updated the Zipkin log levels to ERROR to avoid the infinite WARN logs that were causing the OOM error on zipkin server. As sleuth 1.2.0 is going to send only the new header versions, I guess this quick fix should keep us going in the short run.

@marcingrzejszczak

This comment has been minimized.

Show comment
Hide comment
@marcingrzejszczak

marcingrzejszczak Jan 1, 2017

Contributor

Yeah, makes sense. So let's close it for now :) Happy new year!!!

Contributor

marcingrzejszczak commented Jan 1, 2017

Yeah, makes sense. So let's close it for now :) Happy new year!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment