jaegertracing / jaeger

Backend Changes

Breaking Changes

• Upgrade Badger from v1.6.2 to v3.2103.0 (#3096, @Ashmita152):
  • Deprecated --badger.truncate flag.
  • All badger related expvar prefix has changed from badger to badger_v3.

New Features

• Add docker images for linux/arm64 (#3124, @GaruGaru)
• Add s390x support on multiarch docker images (#2948, @kun-lu20)
• Add TLS support for Prometheus reader (#3096, @albertteoh)

Monitor tab for service health metrics

• Add HTTP handler for metrics querying #3095, @albertteoh)
• Add MetricsQueryService grcp handler #3091, @albertteoh)
• Hook up MetricsQueryService to main funcs (#3079, @albertteoh)
• Add metrics query capability to query service (#3061, @albertteoh)

Bug fixes, Minor Improvements

• Add build info metrics to Jaeger components (#3087, @Ashmita152)
• Upgrade gRPC to 1.38.x (#3096, @pavolloffay)

Backend Changes

Breaking Changes

• Remove unused --es-archive.max-span-age flag (#2865, @albertteoh):
  • max-span-age is only relevant when searching for unarchived traces. Archived traces are searched with no look-back limit.

New Features

• Inject trace context to grpc metadata (#2870, @lujiajing1126)
• Passing default sampling strategies file as environment variable (#3027, @Ashmita152)
• [es] Add index rollover mode that can choose day and hour (#2965, @WalkerWang731)
• Add a TIMEOUT environment variable for es rollover (#2938, @ediezh)
• Allow the ILM policy name to be configurable (#2971, @jrRibeiro)
• [es] Add remote read clusters option for cross-cluster querying (#2874, @dgrizzanti)
• Enable logging in ES client (#2862, @albertteoh)

Bug fixes, Minor Improvements

• Fix jaeger-agent reproducible memory leak (#3050, @jpkrohling)
• Changed Range Query to use startTimeMillis date field instead of startTime field (#2980, @Sreevani871)
• Verify FindTraces() received a query (#2979, @esnible)
• Set Content-Type in healthcheck's http response (#2926, @logeable)
• Add jaeger-query HTTP handler diagnostic logging (#2906, @albertteoh)
• Fix es-archive namespace default values (#2865, @albertteoh)

UI Changes

Enhancements

• Make search panel more compact, button more visible (@meenal06 in #724)

Fixes

• Add null check for span.logs in search/filter-spans (@achesin in #734)
• Several updates to dependencies.

Backend Changes

Breaking Changes

• Remove deprecated TLS flags (#2790, @albertteoh):

  • --cassandra.tls is replaced by --cassandra.tls.enabled
  • --cassandra-archive.tls is replaced by --cassandra-archive.tls.enabled
  • --collector.grpc.tls is replaced by --collector.grpc.tls.enabled
  • --collector.grpc.tls.client.ca is replaced by --collector.grpc.tls.client-ca
  • --es.tls is replaced by --es.tls.enabled
  • --es-archive.tls is replaced by --es-archive.tls.enabled
  • --kafka.consumer.tls is replaced by --kafka.consumer.tls.enabled
  • --kafka.producer.tls is replaced by --kafka.producer.tls.enabled
  • --reporter.grpc.tls is replaced by --reporter.grpc.tls.enabled

• Remove deprecated flags of Query Server --query.port and --query.host-port, please use dedicated HTTP --query.http-server.host-port (defaults to :16686) and gRPC --query.grpc-server.host-port (defaults to :16685) host-ports flags instead (#2772, @rjs211)

  • By default, if no flags are set, the query server starts on the dedicated ports. To use common port for gRPC and HTTP endpoints, the host-port flags have to be explicitly set

• Remove deprecated CLI flags (#2751, @LostLaser):

  • --collector.http-port is replaced by --collector.http-server.host-port
  • --collector.grpc-port is replaced by --collector.grpc-server.host-port
  • --collector.zipkin.http-port is replaced by --collector.zipkin.host-port

• Remove deprecated flags --health-check-http-port & --admin-http-port, please use --admin.http.host-port (#2752, @pradeepnnv)

• Remove deprecated flag --es.max-num-spans, please use --es.max-doc-count (#2482, @BernardTolosajr)

• Remove deprecated flag --jaeger.tags, please use --agent.tags (#2753, @yurishkuro)

• Remove deprecated Cassandra flags (#2789, @albertteoh):

  • --cassandra.enable-dependencies-v2 - Jaeger will automatically detect the version of the dependencies table
  • --cassandra.tls.verify-host - please use --cassandra.tls.skip-host-verify instead

• Remove incorrectly scoped downsample flags from the query service (#2782, @joe-elliott)

  • --downsampling.hashsalt removed from jaeger-query
  • --downsampling.ratio removed from jaeger-query

New Features

• Add TLS Support for gRPC and HTTP endpoints of the Query and Collector servers (#2337, #2772, #2798, @rjs211)

  • If TLS in enabled on either or both of gRPC or HTTP endpoints, the gRPC host-port and the HTTP host-port have to be different
  • If TLS is disabled on both endpoints, common HTTP and gRPC host-port can be explicitly set using the following host-port flags respectively:
    • Query: --query.http-server.host-port and --query.grpc-server.host-port
    • Collector: --collector.http-server.host-port and --collector.grpc-server.host-port

• Add support for Kafka SASL/PLAIN authentication via SCRAM-SHA-256 or SCRAM-SHA-512 mechanism (#2724, @WalkerWang731)

• [agent] Add metrics to show connections status between agent and collectors (#2657, @WalkerWang731)

• Add plaintext as a supported kafka auth option (#2721, @pdepaepe)

• Add ability to use JS file for UI configuration (#123 from jaeger-ui) (#2707, @th3M1ke)

• Support Elasticsearch ILM for managing jaeger indices (#2796, @bhiravabhatla)

• Push official images to quay.io, in addition to Docker Hub (#2783, @Ashmita152)

• Add status command (#2684, @sniperking1234)

  • Usage:

    $ ./cmd/collector/collector-darwin-amd64 status
    {"status":"Server available","upSince":"2021-02-19T17:57:12.671902+11:00","uptime":"25.241233383s"}

• Support configurable date separator for Elasticsearch index names (#2637, @sniperking1234)

Bug fixes, Minor Improvements

• Use workaround for windows x509.SystemCertPool() (#2756, @Ashmita152)
• Guard against mal-formed payloads received by the agent, potentially causing high memory utilization (#2780, @jpkrohling)
• Expose cache TTL for ES span writer index+service (#2737, @necrolyte2)
• Copy spans from memory store (#2720, @bobrik)
• [pkg/queue] Add StartConsumersWithFactory function (#2714, @mx-psi)
• Fix potential cross-site scripting issue (#2697, @yurishkuro)
• Updated gRPC Storage Plugin README with example (#2687, @js8080)
• Deduplicate collector tags (#2658, @Betula-L)
• Add latency metrics on collector HTTP endpoints (#2664, @dimitarvdimitrov)
• Fix collector panic due to sarama sdk (#2654, @Betula-L)
• Handle collector Start error (#2647, @albertteoh)
• [anonymizer] Save trace in UI format (#2629, @yurishkuro)

UI Changes

• UI pinned to version 1.13.0. The changelog is available here v1.13.0

Backend Changes

New Features

• New debug Docker images for Jaeger backends are included in the distribution (#2545, @Ashmita152)
• A utility to anonymize a trace for sharing publicly (#2621, #2585, @Ashmita152)
• Sampling strategies file can be loaded from a URL, not just a file path (#2519, @goku321)
• Configuration parameters can be inspected at runtime via /debug/vars admin endpoint (#2496, @dstdfx)
• OTLP-proto encoding for Kafka supported in the OTEL Ingester (#2580, @XSAM)
• Display backend & UI versions in Jaeger UI
  • Inject version info into index.html (#2547, @yurishkuro)
  • Added jaeger ui version to about menu (#606, @alanisaac)

Bug fixes, Minor Improvements

• Update x/text to v0.3.4 (#2625, @objectiser)
• Update CodeQL to latest best practices (#2615, @jhutchings1)
• Bump opentelemetry-collector to v0.14.0 (#2617, @Vemmy124)
• Bump Badger to v1.6.2 (#2613, @Ackar)
• Fix sarama consumer deadlock (#2587, @albertteoh)
• Avoid deadlock if Stop is called before Serve (#2608, @chlunde)
• Return buffers to pool on network errors or queue overflow (#2609, @chlunde)
• Clarify deadlock panic message (#2605, @yurishkuro)
• fix: don't create tags w/ empty name for internal zipkin spans (#2596, @mzahor)
• TBufferedServer: Avoid channel close/send race on Stop (#2583, @chlunde)
• Bumped OpenTelemetry Collector to v0.12.0 (#2562, @jpkrohling)
• Disable Zipkin server if port/address is not configured (#2554, @yurishkuro)
• [hotrod] Add links to traces (#2536, @yurishkuro)
• OTel Cassandra/Elasticsearch Exporter queue defaults (#2533, @joe-elliott)
• [otel] Update jaeger-lib to v2.4.0 (#2538, @dstdfx)
• Remove unnecessary ServiceName index seek if tags query is available (#2535, @burmanm)
• Update static UI assets path in contrib doc (#2548, @albertteoh)

UI Changes

• UI pinned to version 1.12.0. The changelog is available here v1.12.0

Backend Changes

Breaking Changes

• Configurable ES doc count (#2453, @albertteoh)

  The --es.max-num-spans flag has been deprecated in favour of --es.max-doc-count.
  --es.max-num-spans is marked for removal in v1.21.0 as indicated in the flag description.

  If both --es.max-num-spans and --es.max-doc-count are set, the lesser of the two will be used.

  The use of --es.max-doc-count (which defaults to 10,000) will limit the results from all Elasticsearch
  queries by the configured value, limiting counts for Jaeger UI:

  • Services
  • Operations
  • Dependencies (edges in a dependency graph)
  • Span fetch size for a trace

• The default value for the flag query.max-clock-skew-adjustment has changed to 0s, meaning that the clock skew adjustment is now disabled by default. See #1459.

New Features

• Grpc plugin archive storage support (#2317, @m8rge)
• Separate Ports for GRPC and HTTP requests in Query Server (#2387, @rjs211)
• Configurable ES doc count (#2453, @albertteoh)
• Add storage metrics to OTEL, metrics by span service name (#2431, @pavolloffay)

Bug fixes, Minor Improvements

• Increase coverage on otel/app/defaultconfig and otel/app/defaultcomponents (#2515, @joe-elliott)
• Use OTEL Kafka Exporter/Receiver Instead of Jaeger Core (#2494, @joe-elliott)
• Fix OTEL kafka receiver/ingester panic (#2512, @pavolloffay)
• Disable clock-skew-adjustment by default. (#2513, @jpkrohling)
• Fix ES OTEL status code (#2501, @pavolloffay)
• OTel: Factored out Config Factory (#2495, @joe-elliott)
• Fix failing ServerInUseHostPort test on MacOS (#2477, @albertteoh)
• Fix unmarshalling in OTEL badger (#2488, @pavolloffay)
• Improve UI placeholder message (#2487, @yurishkuro)
• Translate OTEL instrumentation library to ES DB model (#2484, @pavolloffay)
• Add partial retry capability to OTEL ES exporter. (#2456, @pavolloffay)
• Log deprecation warning only when deprecated flags are set (#2479, @pavolloffay)
• Clean-up Badger's trace-not-found check (#2481, @yurishkuro)
• Run the jaeger-agent as a non-root user by default (#2466, @chgl)
• Regenerate certificates to use SANs instead of Common Name (#2461, @albertteoh)
• Support custom port in cassandra schema creation (#2472, @MarianZoll)
• Consolidated OTel ES IndexNameProviders (#2458, @joe-elliott)
• Add positive confirmation that Agent made a connection to Collector (… (#2423, @BernardTolosajr)
• Propagate TraceNotFound error from grpc storage plugins (#2455, @joe-elliott)
• Use new ES reader implementation in OTEL (#2441, @pavolloffay)
• Updated grpc-go to v1.29.1 (#2445, @jpkrohling)
• Remove olivere elastic client from OTEL (#2448, @pavolloffay)
• Use queue retry per exporter (#2444, @pavolloffay)
• Add context.Context to WriteSpan (#2436, @yurishkuro)
• Fix mutex unlock in storage exporters (#2442, @pavolloffay)
• Add Grafana integration example (#2408, @fktkrt)
• Fix TLS flags settings in jaeger OTEL receiver (#2438, @pavolloffay)
• Add context to dependencies endpoint (#2434, @yoave23)
• Fix error equals (#2429, @albertteoh)

UI Changes

• UI pinned to version 1.11.0. The changelog is available here v1.11.0

Upgrade to a working UI version before React refactoring.

Revert UI back to 1.9 due to a bug jaegertracing/jaeger-ui#628

Known Issues

The pull request #2297 aimed to add TLS support for the gRPC Query server but the flag registration is missing, so that this feature can't be used at the moment. A fix is planned for the next Jaeger version (1.20).

Backend Changes

New Features

• Reload TLS certificates on change (#2389, @pavolloffay)
• Add --kafka.producer.batch-min-messages collector flag (#2371, @prymitive)
• Make UDP socket buffer size configurable (#2336, @kbarukhov)
• Enable batch and queued retry processors by default (#2330, @pavolloffay)
• Add trace anonymizer prototype (#2328, @yurishkuro)
• Add native OTEL ES exporter (#2295, @pavolloffay)
• Define busy error type in processor (#2314, @pavolloffay)
• Use gRPC instead of tchannel in hotrod (#2307, @pavolloffay)
• TLS support for gRPC Query server (#2297, @jan25)

Bug fixes, Minor Improvements

• Check missing server URL in ES OTEL client (#2411, @pavolloffay)
• Fix Elasticsearch version in ES OTEL writer (#2409, @pavolloffay)
• Fix go vet warnings on Go 1.15 (#2401, @prymitive)
• Add new Elasticsearch reader implementation (#2364, @pavolloffay)
• Only add the collector port if it was not explicitly set (#2396, @joe-elliott)
• Fix panic in collector when Zipkin server is shutdown (#2392, @Sreevani871)
• Update validity of TLS certificates to 3650 days (#2390, @rjs211)
• Added span and trace id to hotrod logs (#2384, @joe-elliott)
• Jaeger agent logs "0" whenever sampling strategies are requested (#2382, @jpkrohling)
• Fix shutdown order for collector components (#2381, @Sreevani871)
• Serve jquery-3.1.1.min.js locally (#2378, @chaseSpace)
• Use a single shared set of CA, client & server keys/certs for testing (#2343, @rjs211)
• fix null pointer in jaeger-spark-dependencies (#2327, @moolen)
• Rename ARCH to TARGETARCH for multi platform build by docker buildx (#2320, @morlay)
• Mask passwords when written as json (#2302, @objectiser)

UI Changes

• UI pinned to version 1.10.0. The changelog is available here v1.10.0

Backend Changes

Security Fixes

• CVE-2020-10750: jaegertracing/jaeger: credentials leaked to container logs (@chlunde)

New Features

• Add ppc64le support (#2293, @Siddhesh-Ghadi)
• Expose option to enable TLS when sniffing an Elasticsearch Cluster (#2263, @jennynilsen)
• Enable OTEL receiver by default (#2279, @pavolloffay)
• Add Badger OTEL exporter (#2269, @pavolloffay)
• Add all-in-one OTEL component (#2262, @pavolloffay)
• Add arm64 support for binaries and docker images (#2176, @MrXinWang)
• Add Zipkin OTEL receiver (#2247, @pavolloffay)

Bug fixes, Minor Improvements

• Remove experimental flag from rollover (#2290, @pavolloffay)
• Move ES dependencies index mapping to JSON template file (#2285, @pavolloffay)
• Bump go-plugin to 1.3 (#2261, @yurishkuro)
• Ignore chmod events on UI config watcher. (#2254, @rubenvp8510)
• Normalize CLI flags to use host:port addresses (#2212, @ohdearaugustin)
• Add kafka receiver flags to ingester (#2251, @pavolloffay)
• Configure Jaeger receiver and exporter by flags (#2241, @pavolloffay)