Enterprise features permanently blocked by "usage reporting failure" on fresh v0.90.3 + chart 0.80.3 (startup race in start_license_check)

[StringKe]

Summary

On a fresh distributed deploy of openobserve-enterprise:v0.90.3 via the official helm chart openobserve 0.80.3, every enterprise feature (SSO button, Custom Logo Text, OpenFGA RBAC, Service Graph, Custom Hide Self Logo) is silently blocked by block_feature_for_report_failure(). The block persists across restarts unless the router is rollout-restarted after ingester/querier are already online.

Environment

Item	Value
chart	openobserve/openobserve 0.80.3 (latest in repo)
image	o2cr.ai/openobserve/openobserve-enterprise:v0.90.3
deployment mode	distributed (router + ingester + querier + alertmanager + compactor + nats x3 + cnpg PG + openfga + dex + o2ai)
license	v2 JWT, active=true, expires_at 2027-06, no_usage_report_allowed_duration: 604800000000μs (7 days), validator_url: https://license-validator.openobserve.ai
platform	Alibaba Cloud ACK 1.35, NetworkPolicy disabled, pods can reach license-validator.openobserve.ai (HTTP 307 from in-cluster curl)
SSO upstream	Cloudflare Access OIDC via Dex
OpenFGA	enterprise.openfga.enabled=true

Symptoms

1. GET /config returns "sso_enabled": false, "custom_logo_text" shows chart default instead of O2_CUSTOM_LOGO_TEXT value
2. GET /config/dex_login returns {"code":500,"message":"feature blocked due to usage reporting failure"}
3. License is valid: GET /api/license returns active=true, expired=false, version=2, ingestion_used=46.3%
4. _meta.usage stream contains 6.5M+ documents (self-reporting writes work fine)
5. All enterprise pods Ready, ingester/querier registered in NATS cluster
6. Dex pod healthy, /dex/.well-known/openid-configuration 200 from public + cluster-internal
7. Network egress to license-validator.openobserve.ai confirmed working (307 redirect from in-cluster curl)

Reproduce

helm install o2 openobserve/openobserve --version 0.80.3 \
  --set image.enterprise.tag=v0.90.3 \
  --set enterprise.enabled=true \
  --set enterprise.openfga.enabled=true \
  --set enterprise.dex.enabled=true \
  --set auth.O2_ENTERPRISE_LICENSE_KEY=<valid-v2-jwt>
# Wait ~3-5 min for all pods Ready
curl https://<o2-host>/config | jq .sso_enabled   # => false
curl https://<o2-host>/config/dex_login           # => 500 "feature blocked..."

Manual workaround that proves it's a startup race (not a license / network / config issue):

kubectl rollout restart deploy/o2-openobserve-router -n openobserve
# Wait until rollout completes AFTER ingester+querier are confirmed Ready
sleep 90
curl https://<o2-host>/config | jq .sso_enabled   # => true (unblocked)

The block returns whenever router is rolled out again while ingester is restarting (node drain, helm upgrade, image change).

What we observe in code

src/handler/http/request/status/mod.rs:335:

let sso_enabled = enterprise_value!(false, dex_cfg.dex_enabled, block_features);

block_features = block_feature_for_report_failure().await from the closed-source o2_enterprise::enterprise::license crate (introduced PR #11192).

src/job/mod.rs init_deferred():

o2_enterprise::enterprise::license::start_license_check(
    crate::service::self_reporting::search::get_usage,
    get_nats_lock,
    crate::service::self_reporting::search::get_license_usage_data_from_node,
    LOCAL_NODE.is_router() || LOCAL_NODE.is_single_role(),
).await;

The router's earliest log lines on every fresh start:

ERROR o2_enterprise::enterprise::license::limits: no online ingester node found for syncing license info
ERROR o2_enterprise::enterprise::license: error in checking search limits : no online ingester node found

The helm chart deployment ordering doesn't guarantee ingester is online before router calls init_deferred(). The first start_license_check invocation hits an empty ingester pool, and from then on /config/dex_login returns "feature blocked due to usage reporting failure" until the router is restarted while ingester is already up.

Chart defaults that compound the issue

charts/openobserve/values.yaml 0.80.3:

config:
  ZO_USAGE_REPORTING_ENABLED: "false"
  ZO_USAGE_REPORTING_CREDS: ""
  ZO_USAGE_REPORTING_URL: "http://localhost:5080/api/_meta/usage/_json"

templates/router-deployment.yaml only renders a hard-coded check-nats initContainer with no exposed extraInitContainers field, so adding a wait-for-ingester initContainer via values requires forking the chart.

Our workaround

1. Inject via component extraEnv (router + ingester + querier + alertmanager + compactor):

   - { name: ZO_USAGE_REPORTING_ENABLED, value: "true" }
   - { name: ZO_USAGE_REPORTING_CREDS, value: "Basic <base64(root_email:root_password)>" }
   - { name: ZO_USAGE_REPORTING_MODE, value: "local" }

2. After every helm install/helm upgrade, run a K8s Job (with RBAC) that:

   • kubectl rollout status sts/o2-openobserve-ingester --timeout=600s
   • kubectl rollout status sts/o2-openobserve-querier --timeout=600s
   • sleep 20 (NATS cluster register settle)
   • kubectl rollout restart deploy/o2-openobserve-router
   • kubectl rollout status deploy/o2-openobserve-router

This unblocks reliably but adds ~5 min to every helm operation, and any subsequent router restart (node drain, image pull retry) re-races.

Additional observation

Once a v0.90 binary boots, it writes new rows to seaql_migrations (m20260415_000001_create_model_pricing_table, m20260415_000002_add_source_to_model_pricing, m20260504_000001_add_anomaly_detection_config_folder_fk). After this, every enterprise pod started with v0.80.2 panics on boot with Migration file of version '<...>' is missing, this migration has been applied but its file is missing. So a v0.90 → v0.80 image downgrade is not possible without DB surgery.

[simranquirky]

Thanks for the detailed analysis and reproduction steps. We'll look into this and share an update soon.