Skip to content

Add independent log level fields for llama-stack and lightspeed-service-api#100

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-lightspeed:lcore-migrationfrom
k-pavlo:lcore-migration
May 29, 2026
Merged

Add independent log level fields for llama-stack and lightspeed-service-api#100
openshift-merge-bot[bot] merged 1 commit into
openstack-lightspeed:lcore-migrationfrom
k-pavlo:lcore-migration

Conversation

@k-pavlo
Copy link
Copy Markdown

@k-pavlo k-pavlo commented May 5, 2026

Summary

Adds a logging subsection to the OpenStackLightspeed CRD with three configurable log level fields for independent control of logging in OGX/llama-stack, lightspeed-service-api, and dataverse exporter containers.

Changes

  • Added LoggingConfig struct with three independent log level fields:
    • ogxLogLevel: Controls OGX/llama-stack container logging
      • Supports standard levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) OR fine-grained component syntax ("core=debug,providers=info")
      • No enum restriction - unrestricted string for flexibility
      • Defaults to "all=info" if empty
    • lightspeedStackLogLevel: Controls lightspeed-service-api container logging
      • Enum with all Python log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
      • Defaults to INFO
    • dataverseExporterLogLevel: Controls dataverse exporter sidecar container logging
      • Enum with all Python log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
      • Defaults to INFO
    • All fields are optional
  • Embedded LoggingConfig as logging subsection in OpenStackLightspeedCore spec
  • Updated OGX/llama-stack container to set both LLAMA_STACK_LOGGING and OGX_LOGGING env vars for compatibility
  • Updated lightspeed-service-api container to use LIGHTSPEED_STACK_LOG_LEVEL env var
  • Updated dataverse exporter sidecar to use --log-level flag with configurable value (previously hardcoded to INFO)
  • Added helper function getOGXLogLevel() with validation using slices.Contains()
  • Updated controller to read from instance.Spec.Logging.* fields
  • Regenerated CRD OpenAPI schema and bundle manifests
  • Updated KUTTL tests to verify all environment variables and container args

Structure

  • The logging configuration is organized in a dedicated "logging" subsection to make its purpose clear and allow to easily add logging options & new services.
  • Each service has an independent field because most troubleshooting scenarios involve issues isolated to one container. Independent fields allow targeted debugging without flooding disk with logs from all containers.
  • ogxLogLevel has fine-grained control as the OGX/llama-stack runtime supports component-level logging control (e.g., "core=debug,providers=info,agents=warning"), which is invaluable for debugging specific subsystems without flooding logs. The unrestricted string field supports both simple usage ("DEBUG") and advanced troubleshooting scenarios.
  • lightspeedStackLogLevel and dataverseExporterLogLevel use enums to ensure only valid Python log levels are specified, with kubebuilder default of "INFO".

Testing

KUTTL Tests

Updated KUTTL tests to verify logging configuration:

  • Tests standard log levels (ogxLogLevel: DEBUG, lightspeedStackLogLevel: WARNING, dataverseExporterLogLevel: DEBUG)
  • Validates environment variables in deployments (LLAMA_STACK_LOGGING=all=debug, OGX_LOGGING=all=debug, LIGHTSPEED_STACK_LOG_LEVEL=WARNING)
  • Validates dataverse exporter container args (--log-level DEBUG)
  • Tests fine-grained component logging (ogxLogLevel: "core=debug,providers=info")
  • Tests dynamic configuration updates with different log levels
  • Validates all Python log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)

Manual Testing

Manually deployed operator and tested all scenarios:

  1. Standard log levels - ogxLogLevel: DEBUG, lightspeedStackLogLevel: WARNING, dataverseExporterLogLevel: DEBUG:

    • OGX/llama-stack: Shows DEBUG logs (API registration, inference steps, provider details)
    • lightspeed-service-api: Shows WARNING and ERROR logs only
    • dataverse-exporter: Shows DEBUG logs (file monitoring, upload attempts)
  2. Independent control - ogxLogLevel: INFO, lightspeedStackLogLevel: DEBUG, dataverseExporterLogLevel: ERROR:

    • OGX/llama-stack: Shows INFO logs only (no DEBUG)
    • lightspeed-service-api: Shows DEBUG logs (SQL queries, A2A tracing, telemetry)
    • dataverse-exporter: Shows ERROR logs only
  3. Fine-grained component logging - ogxLogLevel: "core=debug,providers=info":

    • OGX/llama-stack: Debug logs for core subsystem, info logs for providers
    • Allows surgical debugging of specific components
  4. Dynamic updates: Patching the CR triggers operator reconciliation and updates all container log levels correctly

Example Usage

Basic configuration:

apiVersion: lightspeed.openstack.org/v1beta1
kind: OpenStackLightspeed
metadata:
  name: openstack-lightspeed
  namespace: openshift-lightspeed
spec:
  # ... other fields ...
  logging:
    ogxLogLevel: DEBUG                    # Debug OGX/llama-stack only
    lightspeedStackLogLevel: WARNING      # Warning level for service
    dataverseExporterLogLevel: INFO       # Info level for exporter

Advanced troubleshooting with component-level logging:

spec:
  logging:
    ogxLogLevel: "core=debug,providers=info,agents=warning"
    lightspeedStackLogLevel: DEBUG
    dataverseExporterLogLevel: DEBUG

@openshift-ci openshift-ci Bot requested review from Akrog and umago May 5, 2026 09:46
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 5, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 44e4eea2-e124-4e81-87ae-7bc6de0c304a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented May 5, 2026

Hi @k-pavlo. Thanks for your PR.

I'm waiting for a openstack-lightspeed member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@lpiwowar
Copy link
Copy Markdown
Contributor

lpiwowar commented May 5, 2026

/ok-to-test

Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread internal/controller/lcore_deployment.go
@k-pavlo k-pavlo force-pushed the lcore-migration branch 3 times, most recently from dcf2b68 to 1d21c4f Compare May 5, 2026 14:00
Comment thread bundle/manifests/openstack-lightspeed-operator.clusterserviceversion.yaml Outdated
@lpiwowar
Copy link
Copy Markdown
Contributor

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 13, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor

@lpiwowar lpiwowar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of comments but overall I like the new structure for configuring the log level!!:) ➕

Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread api/v1beta1/openstacklightspeed_types.go Outdated
Comment thread internal/controller/lcore_deployment.go Outdated
Comment thread internal/controller/lcore_deployment.go Outdated
Comment thread api/v1beta1/openstacklightspeed_types.go
@k-pavlo k-pavlo force-pushed the lcore-migration branch 2 times, most recently from f430f24 to 98e7be2 Compare May 13, 2026 13:30
@k-pavlo
Copy link
Copy Markdown
Author

k-pavlo commented May 13, 2026

/test openstack-lightspeed-kuttl-4-18

@k-pavlo k-pavlo force-pushed the lcore-migration branch from 98e7be2 to e70ade8 Compare May 14, 2026 09:47
@k-pavlo
Copy link
Copy Markdown
Author

k-pavlo commented May 14, 2026

The happy path is clean. What happens if the third-party API returns 429 here? Might be worth handling that case explicitly.

Sorry @Sulhan12, I didn't quite get what you mean. Could you please expand this.

@umago
Copy link
Copy Markdown
Contributor

umago commented May 15, 2026

The happy path is clean. What happens if the third-party API returns 429 here? Might be worth handling that case explicitly.

Sorry @Sulhan12, I didn't quite get what you mean. Could you please expand this.

It looks like a bot. Ignore it

umago
umago previously approved these changes May 15, 2026
Copy link
Copy Markdown
Contributor

@umago umago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just looked into it. Thanks Pavlo

@k-pavlo k-pavlo force-pushed the lcore-migration branch 2 times, most recently from 7196b57 to 58db421 Compare May 27, 2026 12:39
@k-pavlo k-pavlo force-pushed the lcore-migration branch from 58db421 to fe1d6ff Compare May 28, 2026 10:46
@k-pavlo k-pavlo force-pushed the lcore-migration branch from fe1d6ff to 7aae2c1 Compare May 28, 2026 11:42
umago
umago previously approved these changes May 28, 2026
Copy link
Copy Markdown
Contributor

@umago umago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Pavlo!

Copy link
Copy Markdown
Contributor

@umago umago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, can you squash the commits before we merge this ?

@umago umago dismissed their stale review May 28, 2026 13:14

Commits should be squashed before

@openshift-ci openshift-ci Bot removed the approved label May 28, 2026
lpiwowar
lpiwowar previously approved these changes May 28, 2026
Copy link
Copy Markdown
Contributor

@lpiwowar lpiwowar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Nice PR! Thank you 🎉

@lpiwowar
Copy link
Copy Markdown
Contributor

Missed the comment from @umago . Overall LGTM. But waiting for the commit squash.

Copy link
Copy Markdown
Contributor

@lpiwowar lpiwowar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are adding the dataverse exporter log config as part of this PR one small change probably needed on the side of the KUTTL tests (reason for their failure) but then it should be good :) 🤞

@k-pavlo k-pavlo force-pushed the lcore-migration branch from 1f2e720 to ad0c77e Compare May 29, 2026 09:11
Add a logging subsection to the OpenStackLightspeed CRD with three
configurable log level fields for independent control of each container.

Changes:
- Add LoggingConfig struct with three fields:
  - ogxLogLevel: Controls OGX/llama-stack container
    - Supports standard levels (DEBUG, INFO) OR fine-grained syntax
      ("core=debug,providers=info")
    - Defaults to "all=info" if empty
  - lightspeedStackLogLevel: Controls lightspeed-service-api container
    - Enum: DEBUG, INFO, WARNING, ERROR, CRITICAL
    - Defaults to INFO
  - dataverseExporterLogLevel: Controls dataverse exporter sidecar
    - Enum: DEBUG, INFO, WARNING, ERROR, CRITICAL
    - Defaults to INFO
- Embed LoggingConfig under 'logging:' subsection in CRD spec
- Update KUTTL tests to verify environment variables and container args

Independent fields enable targeted debugging without flooding logs from
all containers. Fine-grained OGX logging supports component-level control
for troubleshooting specific subsystems.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@k-pavlo k-pavlo force-pushed the lcore-migration branch from ad0c77e to 9f5e97b Compare May 29, 2026 09:57
@k-pavlo
Copy link
Copy Markdown
Author

k-pavlo commented May 29, 2026

/retest

// +kubebuilder:default="INFO"
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Dataverse Exporter Log Level"
// Log level for the dataverse exporter sidecar container. Supports standard Python log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.
DataverseExporterLogLevel string `json:"dataverseExporterLogLevel,omitempty"`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this!

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented May 29, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k-pavlo, lpiwowar, umago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lpiwowar
Copy link
Copy Markdown
Contributor

/lgtm

Thank you!:) 🎉

Nice we have all the app containers covered!

@openshift-ci openshift-ci Bot added the lgtm label May 29, 2026
@openshift-merge-bot openshift-merge-bot Bot merged commit dbe8c44 into openstack-lightspeed:lcore-migration May 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants