mgr/cephadm: Use a persistent volume to store Loki DB

[aaSharma14]

Current Behavior: The Loki configuration directs all data to /tmp/loki, which is not mounted as a persistent volume. This results in data being stored in a temporary location that gets cleared upon container restart.

Expected Behavior: Loki data should be stored in /loki, which is backed by the persistent volume mapped to /var/lib/ceph//loki./data, ensuring data persistence across container restarts.

Fixes: https://tracker.ceph.com/issues/69029

Signed-off-by: Aashish Sharma aasharma@redhat.com

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows
• jenkins test rook e2e

[aaSharma14]

jenkins test make check arm64

[phlogistonjohn]

This should probably be scoped as mgr/cephadm not mgr/dashboard as dashboard is not touched by this change.

Second question: this volume mapping will tie this instance of loki to the particular host is it running on. Is this ok, if so I'd like to see that explained in comments (and commit message). Only resources that are meant to be tied to a single host or only semi-persistent (caches) ought to be volume mapped into the containers.

[phlogistonjohn]

Setting 'Request changes' in order to ensure we discuss the above comment before this is considered mergable.

[adk3798]

https://pulpito.ceph.com/adking-2025-01-10_07:24:29-orch:cephadm-wip-adk-testing-2025-01-09-1901-distro-default-smithi/

Failures:

• 5x mds_upgrade_sequence failures, known issue
• 2x staggered upgrade with agent failures, known issue
• https://pulpito.ceph.com/adking-2025-01-10_07:24:29-orch:cephadm-wip-adk-testing-2025-01-09-1901-distro-default-smithi/8070280 failed with Invalid command: Duplicate argument 'define_user_pass' found., which was found to be caused by a PR in the run. Luckily no actual SMB module PRs were in the run, so this should only block merging the culprit PR.
• 2x rados_api_tests tests failing during the cleanup phase of its test script, known issue
• https://pulpito.ceph.com/adking-2025-01-10_07:24:29-orch:cephadm-wip-adk-testing-2025-01-09-1901-distro-default-smithi/8070270 is a new test being added by a PR in the run that isn't working yet
• https://pulpito.ceph.com/adking-2025-01-10_07:24:29-orch:cephadm-wip-adk-testing-2025-01-09-1901-distro-default-smithi/8070230 is a relatively new failure, but I confirmed it's happening on main (https://pulpito.ceph.com/adking-2025-01-08_19:05:19-orch:cephadm-main-distro-default-smithi/8068445) so it's unrelated to PRs in this run
• https://pulpito.ceph.com/adking-2025-01-10_07:24:29-orch:cephadm-wip-adk-testing-2025-01-09-1901-distro-default-smithi/8070229/ is the one unknown failure. I'm still root causing it, and it will likely block merging of nfs module related PRs in the run

Overall, can't cause the two PRs that caused the direct failures mentioned or nfs module PRs. Other changes should be okay.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[aaSharma14]

@phlogistonjohn ,

1. Yes, this volume mapping does tie the Loki instance to a particular host since it stores data in a host-mounted volume. This is intentional because Loki's logs need to persist across container restarts and avoid loss due to /tmp/loki being ephemeral.
2. Ceph related services already leverage persistent storage under /var/lib/ceph, so placing Loki data here maintains consistency.
3. While this ties Loki to a specific node, it's a necessary trade-off to avoid losing logs. If required, we can explore using an external object store backend in the future for a more distributed setup.
4. Also, we can probably add a warning in the documentation that the /var/lib/ceph//loki./data can grow very quickly and should be recommend it being configured in a separate FS than /

[aaSharma14]

jenkins test dashboard cephadm

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

[adk3798]

https://pulpito.ceph.com/adking-2025-09-17_13:26:44-orch:cephadm-wip-adk-testing-2025-09-16-1923-distro-default-smithi/

okay run for merging outside of anything SMB related

[adk3798]

don't know much about this, but the PR looks like it's doing what's described

[phlogistonjohn]

.

.