Release 1.13.1

Backend Changes

Breaking Changes

New Features

Bug fixes, Minor Improvements

Change default for bearer-token-propagation to false (#1642, @wsoula)

UI Changes

Release 1.13.0

Backend Changes

Breaking Changes

• The traces related metrics on collector now have a new tag sampler_type (#1576, @guanw)

  This might break some existing metrics dashboard (if so, users need to update query to aggregate over this new tag).

  The list of metrics affected: traces.received, traces.rejected, traces.saved-by-svc.

• Remove deprecated index prefix separator : from Elastic (#1620, @pavolloffay)

  In Jaeger 1.9.0 release the Elasticsearch index separator was changed from : to -. To keep backwards
  compatibility the query service kept querying indices with : separator, however the new indices
  were created only with -. This release of Jaeger removes the query capability for indices containing :,
  therefore it's recommended to keep using older version until indices containing old separator are
  not queried anymore.

New Features

• Passthrough OAuth bearer token supplied to Query service through to ES storage (#1599, @rubenvp8510)
• Kafka kerberos authentication support for collector/ingester (#1589, @rubenvp8510)
• Allow Cassandra schema builder to use credentials (#1635, @PS-EGHornbostel)
• Add docs generation command (#1572, @pavolloffay)

Bug fixes, Minor Improvements

• Fix data race between Agent.Run() and Agent.Stop() (#1625, @tigrannajaryan)
• Use json number when unmarshalling data from ES (#1618, @pavolloffay)
• Define logs as nested data type (#1622, @pavolloffay)
• Fix archive storage not querying old spans older than maxSpanAge (#1617, @pavolloffay)
• Query service: fix logging errors on SIGINT (#1601, @jan25)
• Direct grpc logs to Zap logger (#1606, @yurishkuro)
• Fix sending status to health check channel in Query service (#1598, @jan25)
• Add tmp-volume to all-in-one image to fix badger storage (#1571, @burmanm)
• Do not fail es-cleaner if there are no jaeger indices (#1569, @pavolloffay)
• Automatically set GOMAXPROCS (#1560, @rubenvp8510)
• Add CA certs to all-in-one image (#1554, @chandresh-pancholi)

UI Changes

• UI pinned to version 1.3.0. The changelog is available here v1.3.0

Release 1.12.0

Backend Changes

Breaking Changes

• The kafka flags were removed in favor of kafka.producer and kafka.consumer flags (#1424, @ledor473)

  The following flags have been removed in the Collector and the Ingester:

  --kafka.brokers
  --kafka.encoding
  --kafka.topic
  --ingester.brokers
  --ingester.encoding
  --ingester.topic
  --ingester.group-id
  

  In the Collector, they are replaced by:

  --kafka.producer.brokers
  --kafka.producer.encoding
  --kafka.producer.topic
  

  In the Ingester, they are replaced by:

  --kafka.consumer.brokers
  --kafka.consumer.encoding
  --kafka.consumer.topic
  --kafka.consumer.group-id
  

• Add Admin port and group all ports in one file (#1442, @yurishkuro)

  This change fixes issues #1428, #1332 and moves all metrics endpoints from API ports to admin ports. It requires re-configuring Prometheus scraping rules. Each Jaeger binary has its own admin port that can be found under --admin-http-port command line flag by running the ${binary} help command.

New Features

• Add gRPC resolver using external discovery service (#1498, @guanw)
• gRPC storage plugin framework (#1461, @chvck)
• Supports customized kafka client id (#1507, @newly12)
• Support gRPC for query service (#1307, @annanay25)
• Expose tls.InsecureSkipVerify to es.tls.* CLI flags (#1473, @stefanvassilev)
• Return info msg for /health endpoint (#1465, @stefanvassilev)
• Add pprof endpoint to admin endpoint (#1375, @konradgaluszka)
• Add inbound transport as label to collector metrics #1446 (guanw)
• Sorted key/value store badger backed storage plugin (#760, @burmanm)
• Add Admin port and group all ports in one file (#1442, @yurishkuro)
• Adds support for agent level tag (#1396, @annanay25)
• Add a Downsampling writer that drop a percentage of spans (#1353, @guanw)

Bug fixes, Minor Improvements

• Sort traces in memory store to return most recent traces (#1394, @jacobmarble)
• Add span format tag for jaeger-collector (#1493, @guo0693)
• Upgrade gRPC to 1.20.1 (#1492, @guanw)
• Switch from counter to a gauge for partitions held (#1485, @bobrik)
• Add CORS handling for Zipkin collector service (#1463, @JonasVerhofste)
• Check elasticsearch nil response (#1467, @YEXINGZHE54)
• Disable sampling in logger - zap(#1460, @psinghal20)
• New layout for proto definitions and generated files (#1427, @annanay25)
• Upgrade Go to 1.12.1 (#1437 ,@yurishkuro)

UI Changes

• UI pinned to version 1.2.0. The changelog is available here v1.2.0

Release 1.11.0

Backend Changes

Breaking Changes

• Introduce kafka.producer and kafka.consumer flags to replace kafka flags (1360, @ledor473)

  The following flags have been deprecated in the Collector and the Ingester:

  --kafka.brokers
  --kafka.encoding
  --kafka.topic
  

  In the Collector, they are replaced by:

  --kafka.producer.brokers
  --kafka.producer.encoding
  --kafka.producer.topic
  

  In the Ingester, they are replaced by:

  --kafka.consumer.brokers
  --kafka.consumer.encoding
  --kafka.consumer.group-id
  

New Features

• Support secure gRPC channel between agent and collector (#1391, @ghouscht, @yurishkuro)
• Allow to use TLS with ES basic auth (#1388, @pavolloffay)

Bug fixes, Minor Improvements

• Make esRollover.py init idempotent (#1407 and #1408, @pavolloffay)
• Allow thrift reporter if grpc hosts are not provided (#1400, @pavolloffay)
• Deprecate colon in index prefix in ES dependency store (#1386, @pavolloffay)
• Make grpc reporter default and add retry (#1384, @pavolloffay)
• Use CQLSH_HOST in final call to cqlsh (#1372, @funny-falcon)

UI Changes

• UI pinned to version 1.1.0. The changelog is available here v1.1.0

Release 1.10.1

Backend Changes

• Discover dependencies table version automatically (#1364, @black-adder)

Breaking Changes

New Features

Bug fixes, Minor Improvements

• Separate query-service functionality from http handler (#1312, @annanay25)

UI Changes

Release 1.10.0

Backend Changes

Breaking Changes

• Remove cassandra SASI indices (#1328, @black-adder)

Migration Path:

1. Run plugin/storage/cassandra/schema/migration/v001tov002part1.sh which will copy dependencies into a csv, update the dependency UDT, create a new dependencies_v2 table, and write dependencies from the csv into the dependencies_v2 table.
2. Run the collector and query services with the cassandra flag cassandra.enable-dependencies-v2=true which will instruct jaeger to write and read to and from the new dependencies_v2 table.
3. Update spark job to write to the new dependencies_v2 table. The feature will be done in #58.
4. Run plugin/storage/cassandra/schema/migration/v001tov002part2.sh which will DELETE the old dependency table and the SASI index.

Users who wish to continue to use the v1 table don't have to do anything as the cassandra flag cassandra.enable-dependencies-v2 will default to false. Users may migrate on their own timeline however new features will be built solely on the dependencies_v2 table. In the future, we will remove support for v1 completely.

• Remove ErrorBusy metric, it essentially duplicates SpansDropped (#1091, @cstyan)

New Features

• Support certificates in elasticsearch scripts (#1339, @pavolloffay)
• Add ES Rollover support to main indices (#1309, @pavolloffay)
• Load ES auth token from file (#1319, @pavolloffay)
• Add username/password authentication to ES index cleaner (#1318, @gregoryfranklin)
• Add implementation of FindTraceIDs function for Elasticsearch reader (#1280, @vlamug)
• Support archive traces for ES storage (#1197, @pavolloffay)

Bug fixes, Minor Improvements

• Use Zipkin annotations if the timestamp is zero (#1341, @geobeau)
• Use GRPC round robin balancing even if only one hostname (#1329, @benley)
• Tolerate whitespaces in ES servers and kafka brokers (#1305, @verma-varsha)
• Let cassandra servers contain whitespace in config (#1301, @karlpokus)

UI Changes

Release 1.9.0

Backend Changes

Breaking Changes

• Change Elasticsearch index prefix from : to - (#1284, @pavolloffay)

Changed index prefix separator from : to - because Elasticsearch 7 does not allow : in index name.
Jaeger query still reads from old indices containing - as a separator, therefore no configuration or migration changes are required.

• Add CLI configurable es.max-num-spans while retrieving spans from ES (#1283, @annanay25)

The default value is set to 10000. Before no limit was applied.

• Update to jaeger-lib 2 and latest sha for jaeger-client-go, to pick up refactored metric names (#1282, @objectiser)

Update to latest version of jaeger-lib, which includes a change to the naming of counters exported to
prometheus, to follow the convention of using a _total suffix, e.g. jaeger_query_requests is now
jaeger_query_requests_total.

Jaeger go client metrics, previously under the namespace jaeger_client_jaeger_ are now under
jaeger_tracer_.

• Add gRPC metrics to agent (#1180, @pavolloffay)

The following metrics:

jaeger_agent_tchannel_reporter_batch_size{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batch_size{format="zipkin"} 0
jaeger_agent_tchannel_reporter_batches_failures{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batches_failures{format="zipkin"} 0
jaeger_agent_tchannel_reporter_batches_submitted{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batches_submitted{format="zipkin"} 0
jaeger_agent_tchannel_reporter_spans_failures{format="jaeger"} 0
jaeger_agent_tchannel_reporter_spans_failures{format="zipkin"} 0
jaeger_agent_tchannel_reporter_spans_submitted{format="jaeger"} 0
jaeger_agent_tchannel_reporter_spans_submitted{format="zipkin"} 0

jaeger_agent_collector_proxy{endpoint="baggage",result="err"} 0
jaeger_agent_collector_proxy{endpoint="baggage",result="ok"} 0
jaeger_agent_collector_proxy{endpoint="sampling",result="err"} 0
jaeger_agent_collector_proxy{endpoint="sampling",result="ok"} 0

have been renamed to:

jaeger_agent_reporter_batch_size{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batch_size{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_batches_failures{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batches_failures{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_batches_submitted{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batches_submitted{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_spans_failures{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_spans_failures{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_spans_submitted{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_spans_submitted{format="zipkin",protocol="tchannel"} 0

jaeger_agent_collector_proxy{endpoint="baggage",protocol="tchannel",result="err"} 0
jaeger_agent_collector_proxy{endpoint="baggage",protocol="tchannel",result="ok"} 0
jaeger_agent_collector_proxy{endpoint="sampling",protocol="tchannel",result="err"} 0
jaeger_agent_collector_proxy{endpoint="sampling",protocol="tchannel",result="ok"} 0

• Rename tcollector proxy metric in agent (#1182, @pavolloffay)

The following metric:

jaeger_http_server_errors{source="tcollector-proxy",status="5xx"}

has been renamed to:

jaeger_http_server_errors{source="collector-proxy",status="5xx"}

New Features

• Add tracegen utility for generating traces (#1245, @yurishkuro)
• Use DCAwareRoundRobinPolicy as fallback for TokenAwarePolicy (#1285, @vprithvi)
• Add Zipkin Thrift as kafka ingestion format (#1256, @geobeau)
• Add FindTraceID to the spanstore interface (#1246, @vprithvi)
• Migrate from glide to dep (#1240, @isaachier)
• Make tchannel timeout for reporting in agent configurable (#1034, @gouthamve)
• Add archive traces to all-in-one (#1189, @pavolloffay)
• Start moving components of adaptive sampling to OSS (#973, @black-adder)
• Add gRPC communication between agent and collector (#1165, #1187, #1181 and #1180, @pavolloffay)

Bug fixes, Minor Improvements

• Update exposed ports in ingester dockerfile (#1289, @objectiser)
• Upgrade Shopify/Sarama for proper handling newest kafka servers 2.x by ingester (#1248, @vprithvi)
• Fix sampling strategies overwriting service entry when no sampling type is specified (#1244, @objectiser)
• Fix dot replacement for int (#1272, @pavolloffay)
• Add C* query to error logs (#1250, @vprithvi)
• Add locking around partitionIDToState map accesses (#1239, @vprithvi)
• Reorganize config manager packages in agent (#1198, @pavolloffay)

UI Changes

• UI pinned to version 1.0.0. The changelog is available here v1.0.0

Release 1.8.2

UI Changes

New Features

• Embedded components (SearchTraces and Tracepage) (#263, @aljesusg)

Bug fixes, Minor Improvements

• Fix link in scatter plot when embed mode (#283, @aljesusg)
• Fix rendering X axis in TraceResultsScatterPlot - pass milliseconds to moment.js (#274, @istrel)

Release 1.8.1

Backend Changes

Bug fixes, Minor Improvements

• Make agent timeout for reporting configurable and fix flags overriding (#1034, @gouthamve)
• Fix metrics handler registration in agent (#1178, @pavolloffay)

Release 1.8.0

Backend Changes

Breaking Changes

• Refactor agent configuration (#1092, @pavolloffay)

The following agent flags has has been deprecated in order to support multiple reporters:

--collector.host-port
--discovery.conn-check-timeout
--discovery.min-peers

New flags:

--reporter.tchannel.host-port
--reporter.tchannel.discovery.conn-check-timeout
--reporter.tchannel.discovery.min-peers

• Various changes around metrics produced by jaeger-query: Names scoped to the query component, generated for all span readers (not just ES), consolidate query metrics and include result tag (#1074, #1075 and #1096, @objectiser)

For example, sample of metrics produced for find_traces operation before:

jaeger_find_traces_attempts 1
jaeger_find_traces_errLatency_bucket{le="0.005"} 0
jaeger_find_traces_errors 0
jaeger_find_traces_okLatency_bucket{le="0.005"} 0
jaeger_find_traces_responses_bucket{le="0.005"} 1
jaeger_find_traces_successes 1

And now:

jaeger_query_latency_bucket{operation="find_traces",result="err",le="0.005"} 0
jaeger_query_latency_bucket{operation="find_traces",result="ok",le="0.005"} 2
jaeger_query_requests{operation="find_traces",result="err"} 0
jaeger_query_requests{operation="find_traces",result="ok"} 2
jaeger_query_responses_bucket{operation="find_traces",le="0.005"} 2

New Features

• Configurable deadlock detector interval for ingester (#1134, @marqc)
• Emit spans for elastic storage backend (#1128, @annanay25)
• Allow to use TLS certificates for Elasticsearch authentication (#1139, @clyang82)
• Add ingester metrics, healthcheck and rename Kafka cli flags (#1094, @ledor473)
• Add a metric for number of partitions held (#1154, @vprithvi)
• Log jaeger-collector tchannel port (#1136, @mindaugasrukas)
• Support tracer env based initialization in hotrod (#1115, @eundoosong)
• Publish ingester as binaries and docker image (#1086, @ledor473)
• Use Go 1.11 (#1104, @isaachier)
• Tag images with commit SHA and publish to -snapshot repository (#1082, @pavolloffay)

Bug fixes, Minor Improvements

• Fix child span context while tracing cassandra queries (#1131, @annanay25)
• Deadlock detector hack for Kafka driver instability (#1087, @vprithvi)
• Fix processor overriding data in a buffer (#1099, @pavolloffay)

UI Changes

New Features

• Span Search - Highlight search results (#238), @davit-y
• Span Search - Improve search logic (#237), @davit-y
• Span Search - Add result count, navigation and clear buttons (#234), @davit-y

Bug Fixes, Minor Improvements

• Use correct duration format for scatter plot (#266), @tiffon)
• Fix collapse all issues (#264), @tiffon)
• Use a moderately sized canvas for the span graph (#257), @tiffon)