Why Metric values reset when application restarts

[manu-balakrishnan]

Hi,

I am exporting application custom metrics to Prometheus TSDB with the help of OTLP exporter and Otel collector. Application is exporting metric values to Prometheus successfully and able to view same value in Prometheus UI. However, when application restarts, metric values are resetting to its default value and start updating based on default value instead of last value which got saved in Prometheus TSDB.

Is there any way to update the metric as per last value that saved in TSDB even after application restarts?

here is my configurations:

docker-compose.yaml:

version: '3.9'

    
services:

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

      
  # OpenTelemetry Collector
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./output:/etc/output:rw

    ports:
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver
      - "55681:55681"   # OTLP http receiver

    depends_on:
      - jaeger-collector
 
   
  jaeger-collector:
    image: jaegertracing/jaeger-collector
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200", "--collector.zipkin.host-port=:9411"]
    ports:
      - "14269"
      - "14268:14268"
      - "14250"
      - "6831:6831/udp"
      
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch

    
  jaeger-query:
    image: jaegertracing/jaeger-query
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200"]
    ports:
      - "16686:16686"
      - "16687"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch


  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3
    environment:
      - discovery.type=single-node
    ports:
    - "9200:9200/tcp"

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:      
      http:
      
exporters:
  jaeger:
    endpoint: jaeger-collector:14250
    insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
  logging:
  # Export to a file
  file:
    path: /etc/output/logs.json

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]
      processors: [batch]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
      processors: [batch]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [logging, file]

Prometheus.yaml

scrape_configs:
- job_name: 'otel-collector'
  scrape_interval: 1s
  static_configs:
  - targets: ['otel-collector:8889']
  - targets: ['otel-collector:8888']

OTLP Exporter (.NET):

builder.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(this.Configuration.GetValue<string>("Otlp:ServiceName")))
            .AddOtlpExporter((otlpOptions, readOption) =>
             {
                 otlpOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
                 otlpOptions.Endpoint = new Uri(this.Configuration.GetValue<string>("Otlp:Endpoint")
                     + "/v1/metrics"
                     );
             });
        break;