Getting error while updating Promitor from 2.4.0 to 2.9.1 app version using helm

[ShivangKumarSingh]

We are trying to upgrade promitor from 2.4.0 to 2.9.0 (Helm chart is 2.5.1 to 2.10.1) but getting the below error :

BackgroundService failed
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[16:24:57 FTL] The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[16:24:57 WRN] Cancelled scraping metrics for job Scraper-14Metrics-Every_5_minutes-fdc30d1751f6433b9b5007f4f0b6b545.

• I also tried updating it manually using helm but receiving "Error: INSTALLATION FAILED: failed to download "promitor/promitor-agent-scraper" in ubuntu.

Providing the Promitor-agent-scrapper.yaml user values

`# Default values for promitor-agent-scraper.

This is a YAML-formatted file.

replicaCount: 1

nameOverride: ""
fullnameOverride: ""

image:
repository: containers.promitor.io/tomkerkhove/promitor-agent-scraper
pullPolicy: Always
tag: 2.9.1

azureAuthentication:
mode: "ServicePrincipal"
identity:
id: ""
key: ""

metricSinks:
atlassianStatuspage:
enabled: false
pageId: ""
apiKey: ""
systemMetricMapping: []
# - id:
# promitorMetricName:
prometheusScrapingEndpoint:
enabled: true
baseUriPath: /metrics
enableMetricTimestamps: true
metricUnavailableValue: NaN
enableServiceDiscovery: false
statsd:
enabled: false
host: ""
port:
metricPrefix: ""
resourceDiscovery:
enabled: false
host: ""
port:
telemetry:
defaultLogLevel: "Error"
applicationInsights:
enabled: false
key: ""
logLevel: ""
containerLogs:
enabled: false
logLevel: ""

Metric Declaration YAML

azureMetadata:
tenantId: ""
subscriptionId: ""
resourceGroupName: ""
metricDefaults:
aggregation:
interval: 00:05:00
scraping:
schedule: "*/5 * * * *"
metrics:[]
secrets:

To use your own secret, set createSecret to false and define the name/keys that your secret uses

createSecret: true

service:
port:
targetPort:
loadBalancer:
enabled: false
azure:
dnsPrefix:
exposeInternally: false

health:
readiness:
enabled: false
verifyDependencies: false
delay: 5
interval: 5
liveness:
enabled: false
verifyDependencies: false
delay: 5
interval: 30

resources:
limits:
cpu: 200m
memory: 1.5Gi
requests:
cpu: 100m
memory: 1Gi

Role-based access control

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

rbac:

If true, create & use RBAC resources

create: true

If true, create & use Pod Security Policy resources

ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

podSecurityPolicyEnabled: false

Service Account for pods

ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

serviceAccount:
## Specifies whether a service account should be created
create: true

## The name of the service account to use if create is false
## If create is true, a name is generated using the fullname template
name: promitor-scraper
annotations: {}

---

Please let us know if it can be fixed by updating this promitor-agent-scrapper.yaml or do we need to update anything?

Ubuntu: 20.04.3
Helm: 3.9.3

[tomkerkhove]

Sorry, can you elaborate what is wrong? If you mean the task cancelation then that is likely just because of stopping an instance to use a new one instead

[ShivangKumarSingh]

Promitor YAML file :

_# Default values for promitor-agent-scraper.

This is a YAML-formatted file.

replicaCount: 1

nameOverride: ""
fullnameOverride: ""

image:
repository: containers.promitor.io/tomkerkhove/promitor-agent-scraper
pullPolicy: Always
tag: 2.9.1

azureAuthentication:
mode: "ServicePrincipal"
identity:
id: ""
key: ""

metricSinks:
atlassianStatuspage:
enabled: false
pageId: ""
apiKey: ""
systemMetricMapping: []
# - id:
# promitorMetricName:
prometheusScrapingEndpoint:
enabled: true
baseUriPath: /metrics
enableMetricTimestamps: true
metricUnavailableValue: NaN
enableServiceDiscovery: false
statsd:
enabled: false
host: ""
port:
metricPrefix: ""
resourceDiscovery:
enabled: false
host: ""
port:
telemetry:
defaultLogLevel: "Error"
applicationInsights:
enabled: false
key: ""
logLevel: ""
containerLogs:
enabled: false
logLevel: ""

Metric Declaration YAML

azureMetadata:
tenantId: ""
subscriptionId: ""
resourceGroupName: ""
metricDefaults:
aggregation:
interval: 00:05:00
scraping:
schedule: "*/5 * * * *"
metrics:[]
secrets:

To use your own secret, set createSecret to false and define the name/keys that your secret uses

createSecret: true

service:
port:
targetPort:
loadBalancer:
enabled: false
azure:
dnsPrefix:
exposeInternally: false

health:
readiness:
enabled: false
verifyDependencies: false
delay: 5
interval: 5
liveness:
enabled: false
verifyDependencies: false
delay: 5
interval: 30

resources:
limits:
cpu: 200m
memory: 1.5Gi
requests:
cpu: 100m
memory: 1Gi

Role-based access control

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

rbac:

If true, create & use RBAC resources

create: true

If true, create & use Pod Security Policy resources

ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

podSecurityPolicyEnabled: false

Service Account for pods

ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

serviceAccount:
## Specifies whether a service account should be created
create: true

## The name of the service account to use if create is false
## If create is true, a name is generated using the fullname template
name: promitor-scraper
annotations: {}

_

[tomkerkhove]

I think this is just because of the upgrade, helm implicitly does the same so I'd just continue unless it happens again in a day or so

[ShivangKumarSingh]

After recieving the error pod is getting terminated and once restarted the same cycle follows. Is there any specific reason for that?

[ShivangKumarSingh]

Hi, I am able to fix this issue by updating the image tag to 2.9.1 and helm chart to 2.6.1 (One upgraded version from the previous one) and it seems to work. But whenever I am updating the helm chart to a newer version than 2.6.1 like 2.7.0 or 2.10.1, I was facing the above-mentioned error only. I wanted to know if anything changed after the 2.6.1 upgrade?

[ShivangKumarSingh]

Any update on what possibly changed after 2.6.1 upgrade that may have raised this issue?

[tomkerkhove]

Frankly, I didn't hear any other reports so I'm not sure what is the cause

[Charlotte-br560]

It seems there might be compatibility or configuration issues during the Promitor upgrade. Here's a quick checklist:

Compatibility: Ensure Promitor 2.9.1 is compatible with your environment, including Kubernetes and Helm versions.

Configuration: Review any changes in configuration between versions and update promitor-agent-scrapper.yaml accordingly.

Error Handling: Investigate any background service failures or scheduled task issues mentioned in the error logs.

Helm Installation: Check permissions and accessibility for pulling Docker images during manual Helm installation.

Troubleshooting: Look for detailed logs to pinpoint the exact cause of the failure.

If problems persist, consider reaching out to the Promitor community for assistance. Also, Crawlbase could be an alternative solution if it fits your needs.