Skip to content

Latest commit

 

History

History
1000 lines (770 loc) · 33.6 KB

upgrading.md

File metadata and controls

1000 lines (770 loc) · 33.6 KB
Raw

Upgrading

Upgrading to v0.96.0-sumo-0

sumologic exporter: remove json_logs

json_logs has been removed in favor of transform processor.

Please follow the migration process

sumologic exporter: remove clear_logs_timestamp

clear_logs_timestamp has been removed in favor of transform processor.

Please follow the migration process

Upgrading to v0.94.0-sumo-0

servicegraph processor: removed in favor of servicegraph connector

The deprecated Service Graph processor has been removed. Use the Service Graph connector instead.

Upgrading to v0.92.0-sumo-0

Exporters: changed retry logic when using persistent queue

In previous versions, when an exporter (e.g. Sumo Logic exporter) was configured to use retries and persistent queue, the data would be retried indefinitely as long as the queue wasn't full. This was because after reaching the retry limit configured in retry_on_failure.max_elapsed_time, the data would be put back in the sending queue, if the queue wasn't full. Starting in v0.92.0-sumo-0, this behavior is changed. Now the data is only retried for retry_on_failure.max_elapsed_time (which currently defaults to five minutes) and dropped after that. To prevent the exporter form ever dropping data that was successfully queued, set retry_on_failure.max_elapsed_time to 0.

For example, change this:

exporters:
  sumologic:
    endpoint: ...
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      storage: file_storage

to this:

exporters:
  sumologic:
    endpoint: ...
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0
    sending_queue:
      enabled: true
      storage: file_storage

See the change for details: open-telemetry/opentelemetry-collector#9090.

Upgrading to v0.91.0-sumo-0

Sumo Logic Schema processor replaced with Sumo Logic processor

The Sumo Logic Schema processor has been deprecated in favor of the Sumo Logic processor. To ensure you are using the latest version change sumologic_schema to sumologic in your configuration.

For example, change this:

processors:
  sumologic_schema:
    # ...
service:
  pipelines:
    logs:
      processors:
        - sumologic_schema

to this:

processors:
  sumologic:
    # ...
service:
  pipelines:
    logs:
      processors:
        - sumologic

k8s_tagger processor: default name of podID attribute has changed

By a mistake, in the k8s_tagger, the default name for podID was set to k8s.pod.id. It has been changed to k8s.pod.uid. If you want to still use the old name, add the following option to the config of the k8s_tagger:

processors:
  k8s_tagger:
    tags:
      podID: k8s.pod.id

Upgrading to v0.90.1-sumo-0

Change configuration for syslogexporter

To migrate, rename the following keys in configuration for syslogexporter:

  • rename protocol property to network
  • rename format property to protocol

For example, given the following configuration:

  syslog:
    protocol: tcp
    port: 514
    endpoint: 127.0.0.1
    format: rfc5424
    tls:
      ca_file: ca.pem
      cert_file: cert.pem
      key_file: key.pem

change it to:

  syslog:
    network: tcp
    port: 514
    endpoint: 127.0.0.1
    protocol: rfc5424
    tls:
      ca_file: ca.pem
      cert_file: cert.pem
      key_file:  key.pem

sumologic exporter: deprecate clear_logs_timestamp

clear_logs_timestamp has been deprecated in favor of transform processor. It is going to be removed in v0.95.0-sumo-0. To migrate:

  • set clear_logs_timestamp to false

  • add the following processor:

    processors:
  transform/clear_logs_timestamp:
    log_statements:
      - context: log
        statements:
          - set(time_unix_nano, 0)

For example, given the following configuration:

exporters:
  sumologic:

change it to:

exporters:
  sumologic:
    clear_logs_timestamp: false
processors:
  transform/clear_logs_timestamp:
    log_statements:
      - context: log
        statements:
          - set(time_unix_nano, 0)
service:
  pipelines:
    logs:
      processors:
        # - ...
        - transform/clear_logs_timestamp

sumologic exporter: remove routing_attributes_to_drop

routing_attributes_to_drop has been removed from sumologic exporter in favor of routing processor's drop_resource_routing_attribute.

To migrate, perform the following steps:

  • remove routing_attributes_to_drop from sumologic exporter
  • add drop_resource_routing_attribute to routing processor

For example, given the following configuration:

processors:
  routing:
    from_attribute: X-Tenant
    default_exporters:
    - jaeger
    table:
    - value: acme
      exporters: [jaeger/acme]
exporters:
  sumologic:
    routing_attributes_to_drop: X-Tenant

change it to:

processors:
  routing:
    drop_resource_routing_attribute: true
    from_attribute: X-Tenant
    default_exporters:
    - jaeger
    table:
    - value: acme
      exporters: [jaeger/acme]
exporters:
  sumologic:

sumologic exporter: deprecate json_logs

json_logs has been deprecated in favor of transform processor. It is going to be removed in v0.95.0-sumo-0.

To migrate perform the following steps:

  • use transform processor in replace of json_logs.add_timestamp and json_logs.timestamp_key:

    processors:
  transform/add_timestamp:
    log_statements:
      - context: log
        statements:
          - set(time, Now()) where time_unix_nano == 0
          - set(attributes["timestamp_key"], Int(time_unix_nano / 1000000))

  • use transform processor in replace of json_logs.flatten_body:

    processors:
  transform/flatten:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, body, "insert") where IsMap(body)
          - set(body, "") where IsMap(body)

  • use transform processor in replace of json_logs.log_key:

    processors:
  transform/set_log_key:
    log_statements:
      - context: log
        statements:
          - set(attributes["log"], body)
          - set(body, "")

Migration example for add_timestamp and timestamp_key

Given the following configuration:

exporters:
  sumologic:
    log_format: json
    json_logs:
      timestamp_key: timestamp_key
      add_timestamp: true

change it to:

exporters:
  sumologic:
    log_format: json
    json_logs:
      add_timestamp: false
processors:
  transform/add_timestamp:
    log_statements:
      - context: log
        statements:
          - set(time, Now()) where time_unix_nano == 0
          - set(attributes["timestamp_key"], Int(time_unix_nano / 1000000))
service:
  pipelines:
    logs:
      processors:
        # ...
        - transform/add_timestamp

Migration example for flatten_body

Given the following configuration:

exporters:
  sumologic:
    log_format: json
    json_logs:
      flatten_body: true

change it to:

exporters:
  sumologic:
    log_format: json
    json_logs:
      flatten_body: false
processors:
  transform/flatten:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, body, "insert") where IsMap(body)
          - set(body, "") where IsMap(body)
service:
  pipelines:
    logs:
      processors:
        # ...
        - transform/flatten

Migration example for log_key

Given the following configuration:

exporters:
  sumologic:
    log_format: json
    json_logs:
      log_key: my_log

change it to:

exporters:
  sumologic:
    log_format: json
    json_logs:
processors:
  transform/set_log_key:
    log_statements:
      - context: log
        statements:
          - set(attributes["my_log"], body)
          - set(body, "")
service:
  pipelines:
    logs:
      processors:
        # ...
        - transform/set_log_key

Upgrading to v0.89.0-sumo-0

remoteobserver processor: renamed to remotetap processor

To migrate, change the processor name remoteobserver to remotetap in your configuration files.

For example, given the following configuration:

processors:
  remoteobserver:
    port: 1234
  remoteobserver/another-one:
    limit: 2

pipelines:
  logs:
    exporters: ["..."]
    processors:
    - remoteobserver
    receivers: ["..."]
  metrics:
    exporters: ["..."]
    processors:
    - remoteobserver/another-one
    receivers: ["..."]

change it to:

processors:
  remotetap:
    port: 1234
  remotetap/another-one:
    limit: 2

pipelines:
  logs:
    exporters: ["..."]
    processors:
    - remotetap
    receivers: ["..."]
  metrics:
    exporters: ["..."]
    processors:
    - remotetap/another-one
    receivers: ["..."]

sumologic exporter: changed default timeout from 5s to 30s

We believe 30 seconds is a better default timeout for the Sumo Logic exporter. The bigger the payload is, the longer it takes for the Sumo Logic backend to process it.

If you want to revert to the previous behavior, set the timeout property to 5s or another value. Example:

exporters:
  sumologic:
    timeout: 5s

sumologic extension: changed default discover_collector_tags from false to true

If you want to revert to the previous behavior, set the discover_collector_tags property to false. Example:

extensions:
  sumologic:
    discover_collector_tags: false

Upgrading to v0.84.0-sumo-0

sumologic extension: removed install_token in favor of installation_token

The install_token configuration property was deprecated in v0.72.0-sumo-0 in February 2023 in favor of installation_token. It is now being removed in v0.84.0-sumo-0. To upgrade, replace install_token property with installation_token in your Sumo Logic extension configuration.

For example, change this:

extensions:
  sumologic:
    install_token: xyz

to this:

extensions:
  sumologic:
    installation_token: xyz

Upgrading to v0.77.0-sumo-0

Full Prometheus metric name normalization is now disabled by default

Prometheus and otel metric naming conventions aren't entirely compatible. Prometheus metric name normalization is a feature intended to convert metrics between the two conventions. See the upstream documentation for details.

In the 0.76.1 release, the feature flag for this normalization was enabled by default.

The feature primarily affects the prometheus and prometheusremotewrite exporters. However, it also modifies some metrics collected by the prometheus receiver. More specifically, it trims certain suffixes from metric names. Unfortunately, this affects a lot of widely used metrics. For example, the standard container CPU usage metric:

container_cpu_usage_seconds_total -> container_cpu_usage_seconds

This change breaks a lot of content built using existing metric names and prevents the Prometheus receiver from being used as a drop-in replacement for Prometheus. Therefore, we've decided to default to having this flag disabled.

The current behaviour can be re-enabled by passing the --feature-gates=+pkg.translator.prometheus.NormalizeName flag to the collector at startup.

There is an ongoing discussion about making this behaviour configurable at runtime. Please follow this issue if you'd like to learn more.

Upgrading to v0.73.0-sumo-1

The default collector name for sumologic extension is now the host FQDN

In an effort to make defaults consistent between resource detection processor and the Sumo extension, we've changed the default collector name and the host name it reports in its metadata to be the host FQDN instead of the hostname reported by the OS. This makes the value consistent with the value of the host.name attribute resource detection processor sets by default.

This will only affect newly registered collectors. If you have local credentials on your host, the existing collector will be used, but if those credentials are cleared, a new collector will be created with a different name. If you'd like to keep using the old name, set CollectorName explicitly in the extension settings.

Upgrading to v0.66.0-sumo-0

filelog receiver: has been removed from sub-parsers

preserve_to has been removed from sub-parsers (#9331).

Now sub-parsers behaves like preserve_to would be equal to parse_from.

sending_queue: require explicit storage set

persistent_storage_enabled configuration option is no longer available (#5784). It is replaced by storage which takes component name as value:

exporters:
  sumologic:
    sending_queue:
      enabled: true
      storage: file_storage

extensions:
  file_storage:
    directory: .
  sumologic:
    collector_name: sumologic-demo
    installation_token: ${SUMOLOGIC_INSTALLATION_TOKEN}

apache receiver: turn on feature gates for resource attributes

The metrics in this receiver are now being sent with two resource attributes: apache.server.name and apache.server.port. Additionally, apache.server.name replaces server_name metric-level attribute.

Both features are hidden behind feature gates, but because they are important for Sumo Logic apps, they have been enabled by default ahead of the normal deprecation timeline.

To disable the new features, disable the feature gates in otelcol's arguments:

otelcol-sumo --config=file:config.yaml --feature-gates=-receiver.apache.emitServerNameAsResourceAttribute,-receiver.apache.emitPortAsResourceAttribute

More information about the feature gates can be found here. The target release for the removal of feature gates is v0.68.

elasticsearch receiver: turn on more datapoints

The metrics elasticsearch.index.operation.count, elasticsearch.index.operation.time and elasticsearch.cluster.shards emit more data points now.

These features are hidden behind feature gates, but because they are important for Sumo Logic apps, they have been enabled by default ahead of the normal deprecation timeline.

To disable the new features, disable the feature gates in otelcol's arguments:

otelcol-sumo --config=file:config.yaml --feature-gates=-receiver.elasticsearch.emitClusterHealthDetailedShardMetrics,-receiver.elasticsearch.emitAllIndexOperationMetrics

More information about the feature gates can be found here. The target release for the removal of feature gates is v0.71.

Upgrading to v0.57.2-sumo-0

sumologic exporter: drop support for source headers

The following properties of the Sumo Logic exporter are deprecated in v0.57.2-sumo-0:

  • source_category
  • source_host
  • source_name

To upgrade, move these properties from the Sumo Logic exporter configuration to the Source processor.

For example, the following configuration:

exporters:
  sumologic:
    source_category: my-source-category
    source_host: my-source-host
    source_name: my-source-name

should be changed to the following:

processors:
  source:
    source_category: my-source-category
    source_category_prefix: ""
    source_category_replace_dash: "-"
    source_host: my-source-host
    source_name: my-source-name

The reason for the additional source_category_prefix and source_category_replace_dash is that the Source processor has more features and these properties must be set to make it behave like the Sumo Logic exporter.

See the Source processor documentation for more details.

Upgrading to v0.56.0-sumo-0

sumologic exporter: drop support for translating attributes

Translating the attributes is harmless, but the exporters should not modify the data. This functionality has been moved to the sumologicschema processor. Due to that, this functionality is now deprecated in the exporter and will be removed soon.

However, if the attributes are not translated, some Sumo Logic apps might not work correctly. To migrate, add a sumologicschema processor to your pipelines that use the sumologic exporter and disable that functionality in the exporter:

processors:
  # ...
  sumologic_schema:
    translate_attributes: true

exporters:
  sumologic:
    # ...
    translate_attributes: false

# ...

service:
  pipelines:
    logs:
      processors:
        # - ...
        - sumologic_schema

Note: by default, the sumologicschema processor also performs other actions, like adding cloud.namespace attribute to the data. If you don't want this to happen, you should explicitly disable these functionalities:

processors:
  sumologic_schema:
    add_cloud_namespace: false
    # ...

Full list of configuration settings can be found in the readme of the processor.

sumologic exporter: drop support for translating Telegraf metric names

Similar as above, the translation should not happen in the exporter and has been moved to the sumologicschema processor. The functionality is now deprecated and will be removed soon.

However, if the attributes are not translated, some Sumo Logic apps might not work correctly. To migrate, add a sumologicschema processor to your pipelines that use the sumologic exporter and disable that functionality in the exporter:

processors:
  # ...
  sumologic_schema:
    translate_telegraf_attributes: true

exporters:
  sumologic:
    # ...
    translate_telegraf_attributes: false

# ...

service:
  pipelines:
    logs:
      processors:
        # - ...
        - sumologic_schema

Note: By default, the sumologicschema processor also performs other actions. Please see a corresponding warning in paragraph sumologic exporter: drop support for translating attributes for more information.

Upgrading to v0.55.0-sumo-0

filter processor: drop support for expr language

Expr language is supported by logstransform processor, so there is no need to have this functionality in filter processor.

The following configuration of filter processor:

processors:
  filter:
    logs:
      include:
        match_type: expr
        expressions:
          - Body matches "log to include"
      exclude:
        match_type: expr
        expressions:
          - Body matches "log to exclude"

is equivalent of the following configuration of logstransform processor:

processors:
  logstransform:
    operators:
      - type: filter
        expr: 'body matches "log to include"'
      - type: filter
        expr: 'not(body matches "log to exclude")'

Upgrading to v0.52.0-sumo-0

sumologic exporter: Removed carbon2 and graphite metric formats

These metric formats don't offer any advantages over Prometheus or OTLP formats. To migrate, simply switch the format to prometheus.

exporters:
  sumologic:
    metric_format: prometheus

Upgrading to v0.51.0-sumo-0

k8s_tagger processor: removed clusterName metadata extraction option

Before v0.51.0-sumo-0, you could specify clusterName as one of the options for metadata extraction:

processors:
  k8s_tagger:
    extract:
      metadata:
      - clusterName

Starting with v0.51.0-sumo-0, the clusterName option is removed. This is a result of an upstream change that removes the k8s.cluster.name metadata (link), which in turn is a result of Kubernetes libraries removing support for this deprecated and non-functional field (link).

How to upgrade

Check if you are using the extract.metadata option in your k8s_tagger processor configuration and if yes, check if it includes the clusterName entry. If it does, remove it.

For example, change the following configuration:

processors:
  k8s_tagger:
    extract:
      metadata:
      - clusterName
      - deploymentName
      - podName
      - serviceName
      - statefulSetName

to the following, removing the clusterName entry from the list:

processors:
  k8s_tagger:
    extract:
      metadata:
      - deploymentName
      - podName
      - serviceName
      - statefulSetName

sumologic exporter: metadata translation: changed the attribute that is translated to _sourceName from file.path.resolved to log.file.path_resolved

This change should have landed already in previous version v0.49.0-sumo-0. It is a result of a change in Filelog receiver, which starting with v0.49.0, changed the names of the attributes it creates when one of the configuration properties is set to true:

  • include_file_name: changed attribute name from file.name to log.file.name
  • include_file_name_resolved: changed attribute name from file.name.resolved to log.file.name_resolved
  • include_file_path: changed attribute name from file.path to log.file.path
  • include_file_path_resolved: changed attribute name from file.path.resolved to log.file.path_resolved

See documentation that describes this change.

How to upgrade

This change is technically a breaking change, but it should be transparent and require no changes in your configuration.

If you have a pipeline that includes Filelog receiver (configured with include_file_path_resolved: true) and Sumo Logic exporter, the Filelog receiver will create the log.file.path_resolved attribute and Sumo Logic exporter will translate this attribute to _sourceName.

exporters:
  sumologic:
    endpoint: ...

receivers:
  filelog:
    include:
    - ...
    include_file_path_resolved: true

service:
  pipelines:
    logs:
      exporters:
      - sumologic
      receivers:
      - filelog

In the unlikely scenario that you have a component other than Filelog receiver that creates the file.path.resolved attribute that you relied on Sumo Logic exporter to be translated to _sourceName attribute, you can perform the translation with Resource processor like the following:

processors:
  resource:
    attributes:
    - action: insert
      from_attribute: file.path.resolved
      key: _sourceName
    - action: delete
      key: file.path.resolved

Upgrading to 0.49.0-sumo-0

Several changes to receivers using opentelemetry-log-collection

The affected receivers are: filelog, syslog, tcplog, udplog, and journald.

opentelemetry-log-collection is a shared library used by the aforementioned receivers. This release contains several breaking changes to its configuration syntax. Please refer to the official upgrade guide for more information.

Sumo Logic exporter metadata handling

The OpenTelemetry data format makes a distinction between record-level attributes and resource-level attributes. The metadata_attributes configuration option in the sumologicexporter allowed setting metadata for records sent to the Sumo Logic backend based on both record and resource-level attributes. Only attributes matching the supplied regular expressions were sent.

However, this is conceptually incompatible with OpenTelemetry. Our intent with the exporter is to use OpenTelemetry conventions as much as we can, to the point where it should eventually be possible to export data to Sumo using the upstream OTLP exporter. This is why we are changing the behaviour. From now on:

  1. metadata_attributes no longer exists.
  2. Metadata for sent records is based on resource-level attributes.

In order to retain current behaviour, processors should be used to transform the data before it is exported. This potentially involves two transformations:

Removing unnecessary metadata using the resourceprocessor

metadata_attributes allowed filtering based on regular expressions. An equivalent processor doesn't yet exist, but resource-level attributed can be dropped using the resourceprocessor. For example:

processors:
  resource:
    attributes:
      - pattern: ^k8s\.pod\..*
        action: delete

will delete all attributes starting with k8s.pod..

NOTE: The ability to delete attributes based on a regular expression is currently unique to our fork of the resourceprocessor, and isn't available in upstream.

Moving record-level attributes used for metadata to the resource level

This can be done using the Group by Attributes processor. If you were using the Sumo Logic exporter to export data with a host record-level attribute:

exporters:
  sumologicexporter:
    ...
    metadata_attributes:
      - host

You can achieve the same effect with the following processor configuration:

processors:
  groupbyattrsprocessor:
    keys:
      - host

Keep in mind that your attribute may already be resource-level, in which case no changes are necessary.